正如标题所示,我正在尝试解析数据并插入到postgresql数据库中。
下面两个函数是我用来完成这项任务的。正如您所看到的,第一个函数接受输入并将其转换为列表列表。
for循环遍历每个列表,并将它们分配给一个对象。你可以看到";insertToDatabase((";从for循环中调用,并将对象作为字符串向下传递。
连接到数据库后,我首先运行一个查询以返回数据库中的所有post_id,并将其存储到列表中。然后我试着做";如果id不在结果中";,继续插入。
然而,这不起作用,每次程序运行时都会添加我的条目,使多个条目相同。我尝试了类似的方法,试图只获取最后一个发布的条目,按时间戳降序排列,然后执行";if id!=last_posted";,但这也没有奏效。
必须有更好的方法来做到这一点。我在这里做错了什么?如果数据库中已经存在某个项的"id"(例如"7229511362"(,我希望跳过将其重新插入数据库的操作,继续循环以检查所有结果。
代码:
def initialParse(results):
rList = [list(r.values()) for r in results]
print(rList)
for l in rList:
r_id = str(l[0])
r_name = str(l[2])
r_url = str(l[3])
r_datetime = str(l[4])
r_updated = str(l[5])
r_price = str(l[6])
r_where = str(l[7])
insertToDatabase(r_id, r_name, r_url, r_datetime, r_updated, r_price, r_where)
def insertToDatabase(id, name, url, date, updated, price, where):
global last_insert
cnxn = connectDb()
cursor = cnxn.cursor()
cursor.execute('select post_id from listings order by tstmp desc')
results = cursor.fetchall()
print(results)
try:
if id not in results:
Logger.writeAndPrintLine('Adding ' + id + ' to database...', 0)
cursor.execute("insert into listings (post_id, timestamp, url, subject, price, location, tstmp) values ("+id+", '"+date+"', '"+url+"', '"+name+"', '"+price+"', '"+where+"', (current_timestamp));")
print('inserted')
cursor.commit()
time.sleep(1)
except:
pass
cursor.close()
disconnectDb(cnxn)
转换为列表列表后的输入数据示例:
[['7230609794', None, '2004 Nissan Sentra sedan automatic runs excellent', 'https://monterey.craigslist.org/cto/d/salinas-2004-nissan-sentra-sedan/7230609794.html', '2020-11-13 17:35', '2020-11-13 17:35', '$1,850', 'Salinas', True, None, False], ['7230559009', None, '2006 mini cooper', 'https://monterey.craigslist.org/cto/d/king-city-2006-mini-cooper/7230559009.html', '2020-11-13 15:38', '2020-11-13 15:38', '$3,000', 'King city', True, None, False]]
cursor.fetchall((返回的示例:
[('7229511362', ), ('7229470879', ), ('7229511362', ), ('7229697890', ), ('7229839309', ), ('7229957054', ), ('7230191646', ), ('7230491972', ), ('7230558061', ), ('7230559009', ), ('7230609794', ), ('7229470879', ), ('7229511362', ), ('7229697890', ), ('7229839309', ), ('7229957054', ), ('7230191646', ), ('7230491972', ), ('7230558061', ), ('7230559009', ), ('7230609794', ), ('7229470879', ), ('7229511362', ), ('7229697890', ), ('7229839309', ), ('7229957054', ), ('7230191646', ), ('7230491972', ), ('7230558061', ), ('7230559009', ), ('7230609794', ), ('7229470879', ), ('7229697890', ), ('7229839309', ), ('7229957054', ), ('7230191646', ), ('7230491972', ), ('7230558061', ), ('7230559009', ), ('7230609794', ), ('7229470879', ), ('7229697890', ), ('7229839309', ), ('7229957054', ), ('7230191646', ), ('7230491972', ), ('7230558061', ), ('7230559009', ), ('7230609794', )]
.fetchall()
返回pyodbc.Row
对象的列表。如果您想使用in
来测试是否返回了特定的id值,那么您首先需要将Row
对象列表转换为标量值列表:
crsr = cnxn.cursor()
rows = crsr.execute("""
SELECT 'foo' AS col1
UNION ALL
SELECT 'bar' AS col1
""").fetchall()
print(rows) # [('foo', ), ('bar', )]
print("foo" in rows) # False
ids = [row[0] for row in rows]
print(ids) # ['foo', 'bar']
print("foo" in ids) # True