从 Sagemaker 笔记本访问 S3 中的 bz2 文件

Question

我能够从 Sagemaker 笔记本从 S3 存储桶读取和写入 csv 文件，但是当尝试使用 csv 文件中使用的路径方法读取 bz2 文件时，我收到没有文件或目录的错误

IOErrorTraceback (most recent call last)
<ipython-input-19-d14d47a702e1> in <module>()
      2 # Create corpus
      3 #%time wiki = WikiCorpus("resources/articles1.xml.bz2", tokenizer_func=spacy_tokenize)
----> 4 wiki = WikiCorpus("s3://sagemakerq/enwiki.xml.bz2", tokenizer_func=spacy_tokenize)
/home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages/gensim/corpora/wikicorpus.pyc in __init__(self, fname, processes, lemmatize, dictionary, filter_namespaces, tokenizer_func, article_min_tokens, token_min_len, token_max_len, lower, filter_articles)
    634 
    635         if dictionary is None:
--> 636             self.dictionary = Dictionary(self.get_texts())
    637         else:
    638             self.dictionary = dictionary
/home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages/gensim/corpora/dictionary.pyc in __init__(self, documents, prune_at)
     82 
     83         if documents is not None:
---> 84             self.add_documents(documents, prune_at=prune_at)
     85 
     86     def __getitem__(self, tokenid):
/home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages/gensim/corpora/dictionary.pyc in add_documents(self, documents, prune_at)
    195 
    196         """
--> 197         for docno, document in enumerate(documents):
    198             # log progress & run a regular check for pruning, once every 10k docs
    199             if docno % 10000 == 0:
/home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages/gensim/corpora/wikicorpus.pyc in get_texts(self)
    676             ((text, self.lemmatize, title, pageid, tokenization_params)
    677              for title, text, pageid
--> 678              in extract_pages(bz2.BZ2File(self.fname), self.filter_namespaces, self.filter_articles))
    679         pool = multiprocessing.Pool(self.processes, init_to_ignore_interrupt)
    680 
IOError: [Errno 2] No such file or directory: 's3://sagemakerq/enwiki.xml.bz2'

Answer 1

看起来您正在使用 Python gensim 包从 S3 中基于 wiki 的数据库转储构建语料库。该软件包不支持直接从 S3 读取。相反，您可以下载文件并使用它。

import boto3
from gensim.corpora.wikicorpus import WikiCorpus
s3 = boto3.client('s3')
s3.download_file('BUCKET_NAME', 'OBJECT_NAME', 'FILE_NAME')
wiki = WikiCorpus('FILE_NAME')