我使用mpi4py库用Python编写MPI程序。Python生成并调用低级C代码。
我只是有一个seg错误，需要追踪它。
如果我是用C编写的，我会在gdb中编译和运行我的程序，找到有问题的代码行，然后从那里开始调试。
如果我是用Python编写的，我仍然会使用gdb(你可以调用gdb python filename.py，gdb会把你指向C的错误行。)
但是遗憾的是，我使用的是MPI，它使用的是Python，它使用C
mrocklin@baconost:~/workspace/ape$ mpiexec -np 3 -hostfile tmp/hostfile -rankfile tmp/rankfile python ape/codegen/run.py tmp/
[baconost:27370] *** Process received signal ***
[baconost:27370] Signal: Segmentation fault (11)
[baconost:27370] Signal code: Address not mapped (1)
[baconost:27370] Failing at address: 0x8
[baconost:27370] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) [0x7fcd529c5c60]
[baconost:27370] [ 1] /home/mrocklin/Software/openmpi/lib/libmpi.so.0(MPI_Comm_get_errhandler+0xa0) [0x7fcd46fae240]
[baconost:27370] [ 2] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/python2.7/site-packages/mpi4py/MPI.so(+0x2ddbc) [0x7fcd47234dbc]
[baconost:27370] [ 3] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/python2.7/site-packages/mpi4py/MPI.so(initMPI+0x1f2e) [0x7fcd4724f95e]
[baconost:27370] [ 4] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(_PyImport_LoadDynamicModule+0xc2) [0x7fcd52cc2e02]
[baconost:27370] [ 5] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xecd90) [0x7fcd52cc0d90]
[baconost:27370] [ 6] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xed031) [0x7fcd52cc1031]
[baconost:27370] [ 7] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyImport_ImportModuleLevel+0x2be) [0x7fcd52cc206e]
[baconost:27370] [ 8] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xd3fed) [0x7fcd52ca7fed]
[baconost:27370] [ 9] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyObject_Call+0x68) [0x7fcd52c19d18]
[baconost:27370] [10] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x56) [0x7fcd52ca8516]
[baconost:27370] [11] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x28b8) [0x7fcd52caba78]
[baconost:27370] [12] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8d2) [0x7fcd52cb0722]
[baconost:27370] [13] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32) [0x7fcd52cb0772]
[baconost:27370] [14] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0xc2) [0x7fcd52cbf6e2]
[baconost:27370] [15] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xebcae) [0x7fcd52cbfcae]
[baconost:27370] [16] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xecd90) [0x7fcd52cc0d90]
[baconost:27370] [17] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xed264) [0x7fcd52cc1264]
[baconost:27370] [18] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyImport_ImportModuleLevel+0x118) [0x7fcd52cc1ec8]
[baconost:27370] [19] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xd3fed) [0x7fcd52ca7fed]
[baconost:27370] [20] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyObject_Call+0x68) [0x7fcd52c19d18]
[baconost:27370] [21] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x56) [0x7fcd52ca8516]
[baconost:27370] [22] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x28b8) [0x7fcd52caba78]
[baconost:27370] [23] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8d2) [0x7fcd52cb0722]
[baconost:27370] [24] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32) [0x7fcd52cb0772]
[baconost:27370] [25] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0xc2) [0x7fcd52cbf6e2]
[baconost:27370] [26] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xebcae) [0x7fcd52cbfcae]
[baconost:27370] [27] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xed58d) [0x7fcd52cc158d]
[baconost:27370] [28] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xecd90) [0x7fcd52cc0d90]
[baconost:27370] [29] /home/mrocklin/Software/epd-7.2-1-rh5-x86_64/lib/libpython2.7.so.1.0(+0xed264) [0x7fcd52cc1264]
[baconost:27370] *** End of error message ***
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 27370 on node baconost.cs.uchicago.edu exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
两个问题：
在一般情况下，在检测到错误后，用这个软件堆栈调试程序的最佳方法是什么(我想避免"你应该用测试来防止错误，而不是诊断它们"的评论。我同意这一点，但有时尽管我们尽了最大努力，还是会发生segfaults)
在这种特定的情况下，这是否指向MPI/
mpi4py问题？最后几行表明，也许情况就是这样。
这个特定错误的问题是旧版本的OpenMPI不能很好地处理rankfile。此错误已报告，并且自1.6.2起似乎已修复。显然，这部分代码很少被使用，也没有经过测试。
首先，当您使用分布式模块/产品时，可能会出现版本不匹配的情况。检查您是否使用了Python/MPI/mpi4y的正确版本组合。
检查了所有这些之后，你所能做的就是：
- 假设是您的代码导致了问题，这通常是一个很好的起点
- 试着将这个案例归结为几个导致SEGV的简单语句。我知道，这是最难的部分
- 如果代码中的错误不明显，那么请询问有关代码的特定问题
- 如果其他一切都失败了，那么可能是其他地方的错误