我正在尝试通过遵循TensorFlow指南网站的教程中的步骤来训练Alexnet CNN模型。/p>
mnist = tf.contrib.learn.datasets.load_dataset("mnist")
train_data = mnist.train.images # Returns np.array
train_labels = np.asarray(mnist.train.labels, dtype=np.int32)
eval_data = mnist.test.images # Returns np.array
eval_labels = np.asarray(mnist.test.labels, dtype=np.int32)
对我来说,我编写了一个脚本将数据集示例写入Tfrecord文件,然后在培训期间,尝试回读这些记录并将其送入Alexnet网络。请参阅下面的代码:
#FUNCTION TO GET ALL DATASET DATA
def _read_multiple_images(filenames, perform_shuffle=False, repeat_count=1,
batch_size=1, available_record=39209, num_of_epochs=1):
def _read_one_image(serialized):
#Specify the fatures you want to extract
features = {'image/shape': tf.FixedLenFeature([], tf.string),
'image/class/label': tf.FixedLenFeature([], tf.int64),
'image/class/text': tf.FixedLenFeature([], tf.string),
'image/filename': tf.FixedLenFeature([], tf.string),
'image/encoded': tf.FixedLenFeature([], tf.string)}
parsed_example = tf.parse_single_example(serialized,
features=features)
#Finese extracted data
image_raw = tf.decode_raw(parsed_example['image/encoded'], tf.uint8)
shape = tf.decode_raw(parsed_example['image/shape'], tf.int32)
label = tf.cast(parsed_example['image/class/label'], dtype=tf.int32)
reshaped_img = tf.reshape(image_raw, shape)
casted_img = tf.cast(reshaped_img, tf.float32)
label_tensor= [label]
image_tensor = [casted_img]
return label_tensor, image_tensor
complete_labels = np.array([])
complete_images = np.array([])
dataset = tf.data.TFRecordDataset(filenames=filenames)
dataset = dataset.map(_read_one_image)
dataset = dataset.repeat(repeat_count) #Repeats dataset this # times
dataset = dataset.batch(batch_size) #Batch size to use
iterator = dataset.make_initializable_iterator()
labels_tensor, images_tensor = iterator.get_next() #Get batch data
no_of_rounds = int(math.ceil(available_record/batch_size));
#Create tf session, get nest set of batches, and evelauate them in batches
sess = tf.Session()
count=1
for _ in range(num_of_epochs):
sess.run(iterator.initializer)
while True:
try:
evaluated_label, evaluated_image = sess.run([labels_tensor,
images_tensor])
#convert evaluated tensors to np array
label_np_array = np.asarray(evaluated_label, dtype=np.uint8)
image_np_array = np.asarray(evaluated_image, dtype=np.uint8)
#squeeze np array to make dimesnsions appropriate
squeezed_label_np_array = label_np_array.squeeze()
squeezed_image_np_array = image_np_array.squeeze()
#add current batch to total
complete_labels = np.append(complete_labels, squeezed_label_np_array)
complete_images = np.append(complete_images, squeezed_image_np_array)
except tf.errors.OutOfRangeError:
print("End of Dataset Reached")
break
count=count+1
sess.close()
return complete_labels, complete_images
我的主要问题是,在将数据集(227x227x3)中的所有39209图像转向恢复器时,作为NP数组,以便我可以喂给我的TF估算器。我的计算机用尽了内存。
train_input_fn = tf.estimator.inputs.numpy_input_fn(x={"x":
complete_images},y=complete_labels,batch_size=100,num_epochs=1,
shuffle=True)
dataset_classifier.train(input_fn=train_input_fn,num_epochs=1,hooks=
[logging_hook])
有没有办法将图像和标签从TF记录中取出,然后将其批量地送到我的tf.tf.simator,而不是必须将其全部加载到NP数组中,此教程
如果您可以以 tf.data.Dataset
访问数据,则无需在将其传递给Estimator
之前将其转换为numpy数组。您可以直接在输入函数中构建Dataset
,并具有以下内容:
def train_input_fn():
dataset = tf.data.TFRecordDataset(filenames=filenames)
dataset = dataset.map(_read_one_image)
dataset = dataset.repeat(1) # Because `num_epochs=1`.
dataset = dataset.batch(100) # Because `batch_size=1`.
dataset = dataset.prefetch(1) # To improve performance by overlapping execution.
iterator = dataset.make_one_shot_iterator() # NOTE: Use a "one-shot" iterator.
labels_tensor, images_tensor = iterator.get_next()
return {"x": images_tensor}, labels_tensor
dataset_classifier.train(
input_fn=train_input_fn, num_epochs=1, hooks=[logging_hook])
这应该比构建Numpy阵列要高得多,因为它避免了一次在内存中实现整个数据集。您还可以使用Dataset.prefetch()
等性能增强功能和Dataset.map()
的并行版本来提高训练速度。