我在Xavier Jetson:上有一个非常特定的代码序列的分段错误
import os
import requests
import tensorflow as tf
# 1
print('SET MEMORY GROWTH')
physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)
# 2
print(f'REQUESTS GET')
requests.get('https://speed.hetzner.de/100MB.bin')
# 3
command = 'ls'
print(f'SYSTEM CALL ({command})')
os.system(command)
# 4
print('MODEL LOAD')
model = tf.keras.models.load_model('mnv2_xavier.h5')
如果我删除其中一个步骤,代码将毫无问题地运行。我不知道其他一些代码序列是否会导致同样的行为,但我很确定它们是否存在。
我正试图弄清楚是什么原因导致这里出现分割错误,但直到现在,我都没有运气。
我认为这可能与tensorflow内存增长策略以及Xavier Jetson在CPU和GPU之间共享内存的事实有关。
我想知道是否有任何方法可以解决这个问题或解决方法,以及是否有人对这种行为有解释。
注意:
创建此模型的代码:
from tensorflow.keras.applications import MobileNetV2
from tensorflow.keras.models import Model
from tensorflow.keras import Input
x = Input((224,244,3))
y = MobileNetV2()(x)
model = Model(x,y)
model.save('mnv2_xavier.h5')
版本:
Jetpack 4.4
tensorflow 2.3.0
keras 2.4.0
python 3.6.9
输出:
2021-04-15 16:51:22.031610: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
SET MEMORY GROWTH
2021-04-15 16:51:25.349940: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-04-15 16:51:25.374098: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:949] ARM64 does not support NUMA - returning NUMA node zero
2021-04-15 16:51:25.374309: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.377GHz coreCount: 8 deviceMemorySize: 31.18GiB deviceMemoryBandwidth: 82.08GiB/s
2021-04-15 16:51:25.374437: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-04-15 16:51:25.377470: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-04-15 16:51:25.379874: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-04-15 16:51:25.380541: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-04-15 16:51:25.383268: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-04-15 16:51:25.385455: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-04-15 16:51:25.385918: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-04-15 16:51:25.386201: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:949] ARM64 does not support NUMA - returning NUMA node zero
2021-04-15 16:51:25.386633: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:949] ARM64 does not support NUMA - returning NUMA node zero
2021-04-15 16:51:25.386723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
REQUESTS GET
SYSTEM CALL (ls)
code logs logs2
bashc.sh main-log.log tests
Desktop Documents mnv2_xavier.h5
Downloads model.py Music
Videos Pictures go
Public segfault.py
MODEL LOAD
2021-04-15 16:51:29.542399: W tensorflow/core/platform/profile_utils/cpu_utils.cc:108] Failed to find bogomips or clock in /proc/cpuinfo; cannot determine CPU frequency
2021-04-15 16:51:29.543521: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xcbba840 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-04-15 16:51:29.543595: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
Segmentation fault (core dumped)
发生此错误是因为系统试图使用超出应有的内存。当系统不允许这样做时,会出现"分段故障"错误。首先,按如下方式检查错误文件。
$gdb python3
(gdb) run pythonfile.py
如果错误为libapt-pkg5.0,请为您的操作系统安装相应的软件包适用于基于unix的操作系统(Xaiver、Nano、TX2(;
$sudo dpkg --purge --force-depends apt apt-utils libapt-inst2.0:arm64 libapt-pkg5.0:arm64
如果错误仍未解决;
$gedit ~/.bashrc
添加;
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1