通过papermill[gcs]
papermill gs://my-bucket/test.ipynb gs://my-bucket/output/test.ipynb
我遇到了错误：http 429超过速率限制。
如果输出笔记本是本地写的，则有效：
papermill gs://my-bucket/test.ipynb /tmp/test.ipynb
本地文件大小为：57K
ls -alh /tmp/test.ipynb
-rw-r--r-- 1 user wheel 57K Feb 14 10:37 /tmp/test.ipynb
Papermill代码：https：//github.com/nteract/papermill/papermill/blob/master/papermill/iorw.py#l258
class GCSHandler(object):
def __init__(self):
self._client = None
def _get_client(self):
if self._client is None:
self._client = GCSFileSystem()
return self._client
def read(self, path):
with self._get_client().open(path) as f:
return f.read()
def listdir(self, path):
return self._get_client().ls(path)
def write(self, buf, path):
with self._get_client().open(path, 'w') as f:
return f.write(buf)
def pretty_path(self, path):
return path
如何复制？
pip install papermill[gcs]
papermill gs://cloud-samples-data/papermill/samples/test.ipynb gs://<your bucket>/test.ipynb
日志：
10
Ending Cell 6------------------------------------------
Exception gcsfs.utils.HtmlError: HtmlError(u'The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.',) in <bound method GCSFile.__del__ of <GCSFile d
pe-sandbox/test.ipynb>> ignored
Traceback (most recent call last):
File "/usr/local/bin/papermill", line 11, in <module>
sys.exit(papermill())
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/papermill/cli.py", line 165, in papermill
cwd=cwd,
File "/usr/local/lib/python2.7/dist-packages/papermill/execute.py", line 90, in execute_notebook
start_timeout=start_timeout,
File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 56, in execute_notebook_with_engine
return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 296, in execute_notebook
nb = cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 352, in execute_managed_notebook
preprocessor.preprocess(nb_man, kwargs)
File "/usr/local/lib/python2.7/dist-packages/papermill/preprocess.py", line 27, in preprocess
nb, resources = self.papermill_process(nb_man, resources)
File "/usr/local/lib/python2.7/dist-packages/papermill/preprocess.py", line 81, in papermill_process
nb_man.cell_complete(nb.cells[index])
File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 76, in wrapper
return func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 219, in cell_complete
self.save()
File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 76, in wrapper
return func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/papermill/engines.py", line 138, in save
write_ipynb(self.nb, self.output_path)
File "/usr/local/lib/python2.7/dist-packages/papermill/iorw.py", line 280, in write_ipynb
papermill_io.write(nbformat.writes(nb), path)
File "/usr/local/lib/python2.7/dist-packages/papermill/iorw.py", line 82, in write
return self.get_handler(path).write(buf, path)
File "/usr/local/lib/python2.7/dist-packages/papermill/iorw.py", line 251, in write
return f.write(buf)
File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-150>", line 2, in close
File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 1548, in close
self.flush(force=True)
File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-145>", line 2, in flush
File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 1367, in flush
self._simple_upload()
File "</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-148>", line 2, in _simple_upload
File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 1465, in _simple_upload
validate_response(r, path)
File "/usr/local/lib/python2.7/dist-packages/gcsfs/core.py", line 162, in validate_response
raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
+ err 'Processing notebook failed'
++ date +%Y-%m-%dT%H:%M:%S%z
+ echo '[2019-02-14T18:32:58+0000]: Processing notebook failed'
[2019-02-14T18:32:58+0000]: Processing notebook failed
+ exit 1
我已经定义：
export GOOGLE_APPLICATION_CREDENTIALS=/keys/my-project.json
gcloud config set account XXXXXXX-compute@developer.gserviceaccount.com
gcloud auth activate-service-account --key-file=/keys/my-project.json
在MacOS环境中，我得到类似的错误：（添加调试）
papermill gs://cloud-samples-data/papermill/samples/test.ipynb gs://dpe-sandbox/test.ipynb
Input Notebook: gs://cloud-samples-data/papermill/samples/test.ipynb
Output Notebook: gs://dpe-sandbox/test.ipynb
('dpe-cloud-mle', 'full_control', None, None, 'none', None)
0%| | 0/28 [00:00<?, ?it/s]uploading
11317
uploading
11342
uploading
11372
4%|███▎ | 1/28 [00:01<00:50, 1.88s/it]uploading
11397
uploading
11427
7%|██████▌ | 2/28 [00:02<00:41, 1.59s/it]uploading
11452
uploading
11482
11%|█████████▊ | 3/28 [00:03<00:33, 1.32s/it]uploading
11507
uploading
11537
14%|█████████████ | 4/28 [00:04<00:26, 1.10s/it]uploading
11562
429
429
Traceback (most recent call last):
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 296, in execute_notebook
nb = cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 352, in execute_managed_notebook
preprocessor.preprocess(nb_man, kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/preprocess.py", line 27, in preprocess
nb, resources = self.papermill_process(nb_man, resources)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/preprocess.py", line 81, in papermill_process
nb_man.cell_complete(nb.cells[index])
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
return func(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 219, in cell_complete
self.save()
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
return func(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 138, in save
write_ipynb(self.nb, self.output_path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 280, in write_ipynb
papermill_io.write(nbformat.writes(nb), path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 82, in write
return self.get_handler(path).write(buf, path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 251, in write
return f.write(buf)
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
self.flush(force=True)
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
self._simple_upload()
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
validate_response(r, path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/user/Documents/Development/dpe/venv/papermill/bin/papermill", line 10, in <module>
sys.exit(papermill())
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/cli.py", line 165, in papermill
cwd=cwd,
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/execute.py", line 90, in execute_notebook
start_timeout=start_timeout,
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 56, in execute_notebook_with_engine
return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 302, in execute_notebook
nb_man.notebook_complete()
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
return func(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 249, in notebook_complete
self.save()
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 76, in wrapper
return func(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/engines.py", line 138, in save
write_ipynb(self.nb, self.output_path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 280, in write_ipynb
papermill_io.write(nbformat.writes(nb), path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 82, in write
return self.get_handler(path).write(buf, path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/papermill/iorw.py", line 251, in write
return f.write(buf)
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
self.flush(force=True)
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
self._simple_upload()
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
validate_response(r, path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
429
429
Exception ignored in: <bound method GCSFile.__del__ of <GCSFile dpe-sandbox/test.ipynb>>
Traceback (most recent call last):
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-153>", line 2, in __del__
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1575, in __del__
self.close()
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
self.flush(force=True)
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
self._simple_upload()
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
validate_response(r, path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
429
429
Exception ignored in: <bound method GCSFile.__del__ of <GCSFile dpe-sandbox/test.ipynb>>
Traceback (most recent call last):
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-153>", line 2, in __del__
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1575, in __del__
self.close()
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-152>", line 2, in close
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1552, in close
self.flush(force=True)
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-147>", line 2, in flush
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1369, in flush
self._simple_upload()
File "</Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/decorator.py:decorator-gen-150>", line 2, in _simple_upload
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 51, in _tracemethod
return f(self, *args, **kwargs)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 1467, in _simple_upload
validate_response(r, path)
File "/Users/user/Documents/Development/dpe/venv/papermill/lib/python3.6/site-packages/gcsfs/core.py", line 163, in validate_response
raise HtmlError(error)
gcsfs.utils.HtmlError: The total number of changes to the object dpe-sandbox/test.ipynb exceeds the rate limit. Please reduce the rate of create, update, and delete requests.
当您尝试在短时间内过于频繁或太快地创建或更新对象时，会发生此错误。如Google Cloud Storage中的配额和限制中所述，只能更新或覆盖单个特定对象每秒一次。在这种情况下，我建议修改您更新或创建对象的方式。
在papermill 0.18.1中添加了将保存到Google Cloud Storage的速率限制，因此此问题不应直接由Papermill引起。