Nvidia Xavier Jetson在尝试加载启用内存增长的模型时出现tensorflow分段错误



我在Xavier Jetson:上有一个非常特定的代码序列的分段错误

import os
import requests
import tensorflow as tf

# 1    
print('SET MEMORY GROWTH')
physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)  
# 2
print(f'REQUESTS GET')
requests.get('https://speed.hetzner.de/100MB.bin')
# 3
command = 'ls'
print(f'SYSTEM CALL ({command})')
os.system(command)
# 4 
print('MODEL LOAD') 
model = tf.keras.models.load_model('mnv2_xavier.h5')

如果我删除其中一个步骤,代码将毫无问题地运行。我不知道其他一些代码序列是否会导致同样的行为,但我很确定它们是否存在。

我正试图弄清楚是什么原因导致这里出现分割错误,但直到现在,我都没有运气。

我认为这可能与tensorflow内存增长策略以及Xavier Jetson在CPU和GPU之间共享内存的事实有关。

我想知道是否有任何方法可以解决这个问题或解决方法,以及是否有人对这种行为有解释。

注意:

创建此模型的代码:

from tensorflow.keras.applications import MobileNetV2
from tensorflow.keras.models import Model
from tensorflow.keras import Input
x = Input((224,244,3))
y = MobileNetV2()(x)
model = Model(x,y)
model.save('mnv2_xavier.h5')

版本:

Jetpack 4.4
tensorflow 2.3.0
keras 2.4.0
python 3.6.9

输出:

2021-04-15 16:51:22.031610: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
SET MEMORY GROWTH
2021-04-15 16:51:25.349940: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-04-15 16:51:25.374098: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:949] ARM64 does not support NUMA - returning NUMA node zero
2021-04-15 16:51:25.374309: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.377GHz coreCount: 8 deviceMemorySize: 31.18GiB deviceMemoryBandwidth: 82.08GiB/s
2021-04-15 16:51:25.374437: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-04-15 16:51:25.377470: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-04-15 16:51:25.379874: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-04-15 16:51:25.380541: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-04-15 16:51:25.383268: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-04-15 16:51:25.385455: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-04-15 16:51:25.385918: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-04-15 16:51:25.386201: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:949] ARM64 does not support NUMA - returning NUMA node zero
2021-04-15 16:51:25.386633: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:949] ARM64 does not support NUMA - returning NUMA node zero
2021-04-15 16:51:25.386723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
REQUESTS GET
SYSTEM CALL (ls)
code          logs          logs2
bashc.sh      main-log.log  tests
Desktop       Documents     mnv2_xavier.h5
Downloads     model.py      Music
Videos        Pictures      go  
Public        segfault.py 
MODEL LOAD
2021-04-15 16:51:29.542399: W tensorflow/core/platform/profile_utils/cpu_utils.cc:108] Failed to find bogomips or clock in /proc/cpuinfo; cannot determine CPU frequency
2021-04-15 16:51:29.543521: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xcbba840 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-04-15 16:51:29.543595: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
Segmentation fault (core dumped)

发生此错误是因为系统试图使用超出应有的内存。当系统不允许这样做时,会出现"分段故障"错误。首先,按如下方式检查错误文件。

$gdb python3
(gdb) run pythonfile.py

如果错误为libapt-pkg5.0,请为您的操作系统安装相应的软件包适用于基于unix的操作系统(Xaiver、Nano、TX2(;

$sudo dpkg --purge --force-depends apt apt-utils libapt-inst2.0:arm64 libapt-pkg5.0:arm64

如果错误仍未解决;

$gedit ~/.bashrc

添加;

export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1

相关内容

  • 没有找到相关文章

最新更新