仅 Scrapyd - 没有模块名称 django_project.settings

Question

此错误仅发生在命令scrapyd-deploy上。

Packing version r240-master
Deploying to project "foo_bot" in http://localhost:6802/addversion.json
Server response (200):
{
"status": "error", 
"message": "Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/lib/python2.7/dist-packages/scrapyd/runner.py", line 40, in <module>
main()
File "/usr/local/lib/python2.7/dist-packages/scrapyd/runner.py", line 37, in main
execute()
File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 109, in execute
settings = get_project_settings()
File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/project.py", line 68, in get_project_settings
settings.setmodule(settings_module_path, priority='project')
File "/usr/local/lib/python2.7/dist-packages/scrapy/settings/__init__.py", line 292, in setmodule
module = import_module(module)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "build/bdist.linux-x86_64/egg/foo_bot/settings.py", line 21, in <module>
File "/usr/local/lib/python2.7/dist-packages/django/__init__.py", line 22, in setup
configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 56, in __getattr__
self._setup(name)
File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 41, in _setup
self._wrapped = Settings(settings_module)
File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 110, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named foo_project.settings
", 
"node_name": "ubuntu"
}

项目结构如下：

/home/dev/workspace/fufu
|--dev_venv/bin/activate
|
|--foo_bot
|  |--scrapy.cfg
|  |
|  |--foo_bot
|     |--settings.py
|     |--settings_dev.py
|     |--settings_deploy.py
|     |--items.py and blablabla
|     |--spiders
|        |--blablablaspider.py
|
|--foo_project
|--manage.py
|
|--app
|  |--models.py and blablabla
|
|--foo_project
|--settings.py
|--settings_dev.py
|--settings_deploy.py
|--wsgi.py and blablabla

scrapy.cfg

[settings]
default = foo_bot.settings
[deploy:local]
url = http://localhost:6801/
project = foo_bot
[deploy:development]
url = http://localhost:6802/
project = foo_bot
version = GIT
[deploy:deployment]
url = http://localhost:6800/
project = foo_bot
version = GIT

foo_bot/foo_bot/settings.py

import os
import sys
import django
########## PATH CONFIGURATION
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FALSE_BASE_DIR = os.path.dirname(BASE_DIR)
#WORKSPACE_DIR = os.path.dirname(FALSE_BASE_DIR) // fail with just scrapy crawl bar
WORKSPACE_DIR = '/'.join([os.path.expanduser('~'), 'workspace'])
BOT_NAME = 'foo_bot'
########## END PATH CONFIGURATION
########## DJANGO CONFIGURATION
DJANGO_PROJECT_PATH = '/'.join([FALSE_BASE_DIR, 'foo_project'])
DJANGO_SETTINGS_MODULE = 'foo_project.settings'
sys.path.append(DJANGO_PROJECT_PATH)
sys.path.insert(0, DJANGO_PROJECT_PATH)
os.environ['DJANGO_SETTINGS_MODULE'] = DJANGO_SETTINGS_MODULE
django.setup()
########## END DJANGO CONFIGURATION
#blablabla settings
try:
from settings_dev import *
except ImportError:
pass

foo_project/foo_project/settings.py

import os
########## PATH CONFIGURATION
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
FALSE_BASE_DIR = os.path.dirname(BASE_DIR)
WORKSPACE_DIR = os.path.dirname(FALSE_BASE_DIR)
PROJECT_NAME = 'foo_project'
########## END PATH CONFIGURATION

foo_project/foo_project/wsgi.py

import os
from django import conf
from django.core.wsgi import get_wsgi_application
conf.ENVIRONMENT_VARIABLE = 'DJANGO_SETTINGS_MODULE'
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'foo_project.settings')
application = get_wsgi_application()

我使用virtualenv在脚本末尾设置环境变量PYTHONPATH和DJANGO_SETTINGS_MODULEactivate如下所示：

export PYTHONPATH="/home/dev/workspace/fufu/foo_project"
export DJANGO_SETTINGS_MODULE="foo_project.settings"

我在fufu/foo_bot运行scrapy crawl bar，它运行成功(并通过djangoitem保存到数据库)。

我在fufu/foo_project运行django-admin.py，它成功运行，没有任何警告。

我在fufu/foo_project运行python manage.py，它运行成功，没有警告。

我用用户dev和 virutalenvdev_venv运行scrapyd-deploy development -p foo_bot。

它返回如上所述的错误。

scrapyd在/home/dev/workspace/调用，用户dev和端口6802

但是，我有另一个用户deploy，/home/deploy/workspace/fufu/有一个项目副本，只有from settings_devfrom settings_deploy和那些绝对路径不同。

我用用户deploy和 virutalenvdeploy_venv运行scrapyd-deploy deployment -p foo_bot。

它运行完美。

scrapyd在/home/deploy/workspace/调用，用户deploy和端口6800。

我该如何解决这个问题？我已经看到了所有的线程，我不知道为什么dev用户失败deploy但用户工作。

Answer 1

好的，我明白了。 正是因为os.path.dirname()会让我走空路。

foo_bot/foo_bot/settings.py

########## PATH CONFIGURATION
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
#FALSE_BASE_DIR = os.path.dirname(BASE_DIR)
#WORKSPACE_DIR = os.path.dirname(FALSE_BASE_DIR) // fail with just scrapy crawl bar
WORKSPACE_DIR = '/'.join([os.path.expanduser('~'), 'workspace'])
FALSE_BASE_DIR = '/'.join([WORKSPACE_DIR, 'fufu'])
BOT_NAME = 'foo_bot'
########## END PATH CONFIGURATION