python在磁盘上存储一个活动变量

Question

我使用的是python 3，内存中有大量变量，我想无缝地将它们写入磁盘，并在实际需要时加载它们，而无需在需要时明确地从磁盘中进行酸洗和读取。这可能吗？怎么可能？

我试过RDFlip，但似乎不起作用这是一家你需要明确处理的商店，我正在努力让它无缝衔接。

Answer 1

您可能想研究对象关系映射(ORM)库之类的东西，它允许您将对象存储到数据库中，并通过使用Python方法/函数调用(而不是SQL语句)来检索它们。SQLAlchemy是python最受欢迎的ORM之一，它有大量的在线文档和社区支持。当为SQLAlchemy定义数据库表和配置数据库连接等时，你只需要做一次你所说的"显式"工作，但在那之后，你可以只使用一个方法调用将变量写入磁盘(在数据库中)，并使用另一个方法来检索它们。与pickle不同的是，您可以在数据库中存储任何二进制对象，因此您不受可以序列化/存储哪种数据的限制。

Answer 2

您尝试过HDF5吗。我想这将是你想要的东西。HDF5

Answer 3

就是这样，我需要它像一个常规的python变量一样灵活，I=4，这就是

您似乎希望变量i存储在磁盘中，而不是存储在内存中，并且您希望像i=4语法那样灵活地存储i。还需要任何数据类型的变量。

请注意，使用赋值运算符(=)总是导致程序将变量存储在内存中，因此您需要其他方法，例如从后面的逻辑是将其存储到磁盘的类中调用对象的方法vardisk.set('i', 4)，您可以使用此语法访问变量vardisk.get('i')

您可以通过首先定义类来做到这一点：

#@title VariableOnDisk
import pickle
import os
class VariableOnDisk():
'''
Save and load variable on disk.
'''
def __init__(self, storage_path="./var_disk/"):
try:
os.mkdir(storage_path)
except:
print('Storage path already exist, here is available variables:', os.listdir(storage_path))
# We only need storage path
self.storage_path = storage_path

def set(self, variable_name, value):
with open(os.path.join(self.storage_path, variable_name), 'wb') as f:
pickle.dump(value, f)

def get(self, variable_name):
if os.path.exists(os.path.join(self.storage_path, variable_name)):
with open(os.path.join(self.storage_path, variable_name), 'rb') as f:
return pickle.load(f)
else:
raise NameError(f"name '{variable_name}' is not defined") # Same error when you try access variable that never defined.

我使用pickle将变量的任何对象存储并加载到文件中。

这是如何使用该类的示例：

# Create instance of VariableOnDisk
vardisk = VariableOnDisk(storage_path='./var_disk/')
# Example to define variable 'i' to disk
vardisk.set('i', 4)
# Example to use variable 'i' from disk
print(vardisk.get('i'), type(vardisk.get('i')))

输出：

4 <class 'int'>

就是这样，上面的代码是这样的：

i = 4
print(i, type(i))

这是另一个具有缓存机制的高级类

class VariableOnDisk():
'''
Save and load variable on disk.
'''
def __init__(self, storage_path='./var_disk/'):
# Make exception for this assignment of __setattr__
self.___storage_path = storage_path 
self.___cached_value = None
self.___cached_varname = None
try:
os.mkdir(storage_path)
except:
print('Storage path already exists, here are available variables:', self)
def __repr__(self):
return str(set(os.listdir(self.___storage_path)))
def __setattr__(self, varname, value):
if '___' in varname:  # Call superclass's __setattr__ for constructor assignment
super().__setattr__(varname, value)
else:
if self.___cached_value == value:
print('Write was cached, skipped!')
return
else:
with open(os.path.join(self.___storage_path, varname), 'wb') as f:
self.___cached_value = value
self.___cached_varname = None
pickle.dump(value, f)
def __getattr__(self, varname):
variable_path = os.path.join(self.___storage_path, varname)
if os.path.exists(variable_path):
if self.___cached_varname == varname:
print('Read was cached, using cached value!')
return self.___cached_value
else:
self.___cached_varname = varname
with open(variable_path, 'rb') as f:
self.___cached_value = pickle.load(f)
return self.___cached_value
else:
raise NameError(f"Variable on disk with name '{varname}' is not defined.") # Same error when you try to access a variable that was never defined.

用法：

# Create instance of VariableOnDisk
vardisk = VariableOnDisk(storage_path='./var_disk/')
# Example to define variable 'i'
vardisk.i = 4
# Since it already defined with same value, it skipped.
vardisk.i = 4
# Example to use variable 'i'
print(vardisk.i)
# Since it already used, it will using cached value
print(type(vardisk.i))
# Example to show available variable name
print(vardisk)

我添加了带有__setattr__的赋值运算符重载(=)。