环境详细信息
- Python 3.7.12
- 谷歌api核心1.23.0
- 谷歌认证1.35.0
- bigquery 2.3.1
- 如果我能提供任何其他库版本，请告诉我
我们在airflow中使用python查询bigquery中的一些数据，并将结果转换为数据帧。查看以下代码块：
from google.cloud import bigquery
bq = bigquery.Client()
query_result = bq.query(f"select count(*) as num_rows from our_project.ourdataset.our_table")
our_df = query_result.to_dataframe()
query_result是<google.cloud.bigquery.job.query.QueryJob object at
当我们运行最后一行our_df = query_result.to_dataframe()时，我们得到错误TypeError: secure_channel() got an unexpected keyword argument 'default_scopes'。整个错误消息是：
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/job/query.py", line 1313, in to_dataframe
date_as_object=date_as_object,
File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/table.py", line 1695, in to_dataframe
create_bqstorage_client=create_bqstorage_client,
File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/table.py", line 1510, in to_arrow
bqstorage_client = self.client._create_bqstorage_client()
File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/client.py", line 446, in _create_bqstorage_client
return bigquery_storage.BigQueryReadClient(credentials=self._credentials)
File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery_storage_v1/services/big_query_read/client.py", line 386, in __init__
or Transport == type(self).get_transport_class("grpc_asyncio")
File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery_storage_v1/services/big_query_read/transports/grpc.py", line 170, in __init__
("grpc.max_receive_message_length", -1),
File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery_storage_v1/services/big_query_read/transports/grpc.py", line 221, in create_channel
**kwargs,
File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 286, in create_channel
return grpc_gcp.secure_channel(target, composite_credentials, **kwargs)
TypeError: secure_channel() got an unexpected keyword argument 'default_scopes'
这行代码以前为我们工作。我记得自上次代码工作以来，我们所做的唯一更改是：
- 已安装
dbt==0.19.0
to.data_frame()似乎是最基本的python函数，但它在这里不起作用，这让我们很沮丧。我们该怎么办才能解决这个问题？
我发现了这个问题，但我不知道如何解决。dbt-bigquery 0.19.0 depends on google-api-core<1.24 and >=1.16.0当我们安装dbt时，它一定将google-api-core版本从更高的版本更改为1.23.0。然而，低至1.23的google-api-core现在导致了to_dataframe()的另一个问题。我知道这一点是因为当我手动将google-api-core升级到1.26时，.to_dataframe()又开始工作了。
EDIT:将dbt升级到0.20.0允许使用googleapicore1.3+，这解决了我们的问题！
对我来说，升级google-auth、google-auth-httplib2和google-api-core成功了。