'Thread'对象没有属性'_children' - Django + scikit-learn

Question

我在使用随机森林分类器 （http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html） 对项目进行分类的 django 应用程序时遇到了问题。我收到的错误说：

AttributeError at /items/
'Thread' object has no attribute '_children'
Request Method:     POST
Request URL:    http://localhost:8000/items/
Django Version:     1.7.6
Exception Type:     AttributeError
Exception Value:    
'Thread' object has no attribute '_children'
Exception Location:     /usr/lib/python2.7/multiprocessing/dummy/__init__.py in start, line 73
Python Executable:  /home/cristian/env/bin/python
Python Version:     2.7.3
Python Path:    
['/home/cristian/filters',
 '/home/cristian/env/lib/python2.7',
 '/home/cristian/env/lib/python2.7/plat-linux2',
 '/home/cristian/env/lib/python2.7/lib-tk',
 '/home/cristian/env/lib/python2.7/lib-old',
 '/home/cristian/env/lib/python2.7/lib-dynload',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-linux2',
 '/usr/lib/python2.7/lib-tk',
 '/home/cristian/env/local/lib/python2.7/site-packages']
Server time:    Fri, 24 Apr 2015 16:08:20 +0000

问题是我根本没有使用线程。这是代码：

def item_to_dict(item):
    item_dict = {}
    for key in item:
            value = item[key]
            # fix encoding 
            if isinstance(value, unicode):
                    value = value.encode('utf-8')
            item_dict[key] = [value]
    return item_dict
def load_classifier(filter_name):
        clf = joblib.load(os.path.join(CLASSIFIERS_PATH, filter_name, 'random_forest.100k.' + filter_name.lower() +  '.pkl'))
        return clf
@api_view(['POST'])
 def classify_item(request):
    """
    Classify item
    """
    if request.method == 'POST':
            serializer = ItemSerializer(data=request.data['item'])
            if serializer.is_valid():
                    # get item and filter_name
                    item = serializer.data
                    filter_name = request.data['filter']
                    item_dict = item_to_dict(item)
                    clf = load_classifier(filter_name)
                    # score item
                    y_pred = clf.predict_proba(pd.DataFrame(item_dict))
                    item_score = y_pred[0][1]
                    # create and save classification
                    classification = Classification(classifier_name=filter_name,score=item_score,item_id=item['_id'])
                    classification_serializer = ClassificationSerializer(classification)
                    return Response(classification_serializer.data, status=status.HTTP_201_CREATED)
            else:
                    return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

我能够打印出"clf"和"item_dict"变量，一切似乎都很好。当我调用分类器的方法"predict_proba"时，会引发错误。要补充的一件重要事情是，当我运行服务器并首次发送 post 方法时，我没有收到错误。

以下是完整的回溯：

File "/home/cristian/env/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
line 111.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/cristian/env/local/lib/python2.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
line 57.         return view_func(*args, **kwargs)
File "/home/cristian/env/local/lib/python2.7/site-packages/django/views/generic/base.py" in view
line 69.             return self.dispatch(request, *args, **kwargs)
File "/home/cristian/env/local/lib/python2.7/site-packages/rest_framework/views.py" in dispatch
line 452.             response = self.handle_exception(exc)
File "/home/cristian/env/local/lib/python2.7/site-packages/rest_framework/views.py" in dispatch
line 449.             response = handler(request, *args, **kwargs)
File "/home/cristian/env/local/lib/python2.7/site-packages/rest_framework/decorators.py" in handler
line 50.             return func(*args, **kwargs)
File "/home/cristian/filters/classifiers/views.py" in classify_item
line 70.             y_pred = clf.predict_proba(pd.DataFrame(item_dict))
File "/home/cristian/env/local/lib/python2.7/site-packages/sklearn/pipeline.py" in predict_proba
line 159.         return self.steps[-1][-1].predict_proba(Xt)
File "/home/cristian/env/local/lib/python2.7/site-packages/sklearn/ensemble/forest.py" in predict_proba
line 468.             for i in range(n_jobs))
File "/home/cristian/env/local/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py" in __call__
line 568.             self._pool = ThreadPool(n_jobs)
File "/usr/lib/python2.7/multiprocessing/pool.py" in __init__
line 685.         Pool.__init__(self, processes, initializer, initargs)
File "/usr/lib/python2.7/multiprocessing/pool.py" in __init__
line 136.         self._repopulate_pool()
File "/usr/lib/python2.7/multiprocessing/pool.py" in _repopulate_pool
line 199.             w.start()
File "/usr/lib/python2.7/multiprocessing/dummy/__init__.py" in start
line 73.         self._parent._children[self] = None
Exception Type: AttributeError at /items/
Exception Value: 'Thread' object has no attribute '_children'

Answer 1

作为一种解决方法，您可以使用以下命令在预测时禁用线程：

clf = load_classifier(filter_name)
clf.set_params(n_jobs=1)
y_pred = clf.predict_proba(pd.DataFrame(item_dict))

另请注意，在每个请求时调用load_classifier可能会很昂贵，因为它实际上从磁盘加载模型。

您可以将mmap_mode='r'传递到内存joblib.load将磁盘中的数据映射。即使您有并发请求同时访问相同的模型参数（如果您使用像 gunicorn 之类的东西，则使用不同的线程和不同的 Python 进程），也可以只加载一次模型。

Answer 2

看起来这个问题是从Python 2.7.5修复的。这基本上是multiprocessing中的一个错误。