Gramex 服务 Redis 缓存有时会引发错误"TypeError: can't pickle _thread.RLock objects"

Question

我正在使用Gramex Redis Cache服务，但有时会遇到以下错误，这是随机发生的，它会在Gramex控制台中抛出错误，你能帮忙吗？

这是来自gramex控制台的错误日志

错误日志-

E 19-Jul 07:23:59 gramex:cache 8000 gramex.cache.open: <class 'gramex.services.rediscache.RedisCache'> cannot cache <tornado.template.Template object at 0x7f5cc43c1290>
Traceback (most recent call last):
  File "/home/star/conda/lib/python3.7/site-packages/tornado/web.py", line 1704, in _execute
    result = await result
  File "/home/star/conda/lib/python3.7/site-packages/tornado/gen.py", line 769, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/home/star/conda/lib/python3.7/site-packages/gramex/handlers/filehandler.py", line 188, in get
    yield self._get_path(self.root)
  File "/home/star/conda/lib/python3.7/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/home/star/conda/lib/python3.7/site-packages/tornado/gen.py", line 234, in wrapper
    yielded = ctx_run(next, result)
  File "/home/star/conda/lib/python3.7/site-packages/gramex/handlers/filehandler.py", line 244, in _get_path
    raise HTTPError(FORBIDDEN, f'{self.file} not allowed')
tornado.web.HTTPError: HTTP 403: Forbidden (/home/star/conda/lib/python3.7/site-packages/gramex/favicon.ico not allowed)
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/home/star/conda/lib/python3.7/site-packages/gramex/cache.py", line 168, in open
    _cache[key] = cached
  File "/home/star/conda/lib/python3.7/site-packages/gramex/services/rediscache.py", line 58, in __setitem__
    value = pickle.dumps(value, pickle.HIGHEST_PROTOCOL)
TypeError: can't pickle _thread.RLock objects

这是我的gramex.yaml配置集：

gramex.yaml配置-

cache:
    memory:
      default: false
    redis:
      type: redis
      # path: $REDIS_CACHE_HOST:$REDIS_CACHE_PORT:$REDIS_CACHE_DB
      path: $REDIS_CACHE_HOST:$REDIS_CACHE_PORT:$REDIS_CACHE_DB:password=$REDIS_CACHE_PASSWORD
      size: 0 #GB cache
      default: true

app:
  session:
    type: redis         # Persistent multi-instance data store
    path: $REDIS_HOST:$REDIS_PORT:0:password=$REDIS_PASSWORD     # Redis server
    expiry: 10          # Session cookies expiry in days
    purge: 86400        # Delete old sessions periodically (in seconds)
    domain: .mydomain.com

url:
  show_daypart_data:
    pattern: /$YAMLURL/show_daypart_data
    handler: FormHandler
    kwargs:
      cors: true
      methods: $http_methods
      headers:
        $request_headers
      frm_30:
        url: $BigQ_CONN
        credentials_path: $CREDENTIAL_PATH
        state: my_utilities.cache_query(handler, '30 mi')
        queryfunction: star.get_show_daypart_query(handler, 'frm_period')
      modify: star.modify_show_daypart_data(handler, data)
      default:
        _limit: 450000
      error: *API_ERROR

这是"；状态"；python函数-

def cache_query(handler, table):
    '''Runs a cache validation query'''
    args = handler.argparse(
        src={'default': 'any'},
        view={'default': 'net'})
    state = ''
    for tbl in table.split('+'):
        table_name = rds_tables[args.src][tbl]
        query = f"""SELECT CONCAT(year, '-', week) as week FROM {table_name}
                        ORDER BY year DESC, week DESC limit 1"""
        try:
            val = REDIS_CACHE[str(query)]
            if not val:
                df = gramex.cache.query(query, db_engine)
                val = df.week.iloc[0]
                REDIS_CACHE[str(query)] = val
            state = '-' + val
        except Exception as error:
            app_log.error(f'Cache query failed ==> {error}')
            pass
    return state

Answer 1

这是Gramex自2022年7月(1.81版(以来的一个错误。发生这种情况是因为

• Gramex BaseHandler加载带有gramex.cache.open的错误模板
• 但是Redis缓存不支持缓存这样的对象

因此不会显示自定义错误，而是报告上面的TypeError。

不幸的是，除了避免Redis缓存之外，目前还没有解决方法。我们计划在Gramex解决这个问题。