在运行时构建多个轮子



我想在运行时构建包含一些脚本和一些有效载荷数据的轮子。 例如,在此示例中,target文件夹包含两个简单的构建build_123和要打包为轮子的build_124

main_project
├── __init__.py
├── whl_util.py               # wheel building script posted below
target/
├── build_123/                # contains one build to be packaged as a whl
│   └── mypkg
|       ├── __init__.py
│       ├── data
|       |   ├── __init__.py
|       |   └── mat.json
│       └── main
|           ├── __init__.py
|           └── dumpmat.py
└── build_124/                # contains another build to be packaged as a whl
└── mypkg
├── ...

在我的场景中,这个轮子被用作输出格式,包装不是主要的进展。轮子打包应该被视为一个简单的 IO 操作,它读取一个构建文件夹并输出一个没有副作用的轮子。为了执行此任务,我想出了以下解决方案:

# main_project/whl_util.py
from setuptools import setup, find_packages
import sys
import shutil
import os
def bdist_wheel(build_dir=".", dist_dir=None):
# backing up argv to restore them afterwards
argv_bak = sys.argv[:]
# clear args from running script with "bdist_wheel"
file = sys.argv[0]
sys.argv.clear()
sys.argv.extend([file, "bdist_wheel"])
if dist_dir is not None and "--dist-dir" not in sys.argv:
sys.argv.extend(["--dist-dir", dist_dir])
sys.argv.extend(["clean", "--all"])
setup(
name="mypkg",
version=0.1,
packages=find_packages(build_dir),
install_requires=[],
include_package_data=True,
package_dir={'': build_dir},
package_data={"mypkg.data": ["mat.json"]}
)
# restore args
sys.argv.clear()
sys.argv.extend(argv_bak)

def main():
# Adding main method here for testing.
# As mentioned in my actual scenario the wheels should be built as an output format at runtime
print("BUILD 123")
bdist_wheel("target/build_123", dist_dir="target/dist_123")
print("BUILD 124")
bdist_wheel("target/build_124", dist_dir="target/dist_124")
if __name__ == "__main__":
main()

我也不太喜欢通过sys.argv将参数传递给setuptools的方式,但这似乎是唯一的方法。但是,主要问题是,第一个轮子正常构建,而bdist_wheel/setup的第二个调用引发错误:

python3 -m main_project.whl_util
BUILD 123
running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/mypkg
copying target/build_123/mypkg/__init__.py -> build/lib/mypkg
creating build/lib/mypkg/data
copying target/build_123/mypkg/data/__init__.py -> build/lib/mypkg/data
creating build/lib/mypkg/main
copying target/build_123/mypkg/main/__init__.py -> build/lib/mypkg/main
copying target/build_123/mypkg/main/dumpmat.py -> build/lib/mypkg/main
running egg_info
writing target/build_123/mypkg.egg-info/PKG-INFO
writing dependency_links to target/build_123/mypkg.egg-info/dependency_links.txt
writing top-level names to target/build_123/mypkg.egg-info/top_level.txt
writing manifest file 'target/build_123/mypkg.egg-info/SOURCES.txt'
copying target/build_123/mypkg/data/mat.json -> build/lib/mypkg/data
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/mypkg
copying build/lib/mypkg/__init__.py -> build/bdist.linux-x86_64/wheel/mypkg
creating build/bdist.linux-x86_64/wheel/mypkg/data
copying build/lib/mypkg/data/__init__.py -> build/bdist.linux-x86_64/wheel/mypkg/data
copying build/lib/mypkg/data/mat.json -> build/bdist.linux-x86_64/wheel/mypkg/data
creating build/bdist.linux-x86_64/wheel/mypkg/main
copying build/lib/mypkg/main/__init__.py -> build/bdist.linux-x86_64/wheel/mypkg/main
copying build/lib/mypkg/main/dumpmat.py -> build/bdist.linux-x86_64/wheel/mypkg/main
running install_egg_info
Copying target/build_123/mypkg.egg-info to build/bdist.linux-x86_64/wheel/mypkg-0.1-py3.7.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/mypkg-0.1.dist-info/WHEEL
creating 'target/dist_123/mypkg-0.1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'mypkg/__init__.py'
adding 'mypkg/data/__init__.py'
adding 'mypkg/data/mat.json'
adding 'mypkg/main/__init__.py'
adding 'mypkg/main/dumpmat.py'
adding 'mypkg-0.1.dist-info/METADATA'
adding 'mypkg-0.1.dist-info/WHEEL'
adding 'mypkg-0.1.dist-info/top_level.txt'
adding 'mypkg-0.1.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
BUILD 124
running bdist_wheel
running build
running build_py
copying target/build_124/mypkg/__init__.py -> build/lib/mypkg
error: could not create 'build/lib/mypkg/__init__.py': No such file or directory
Process finished with exit code 1

no such file or directory错误表明setuptools模块跟踪它已创建的文件夹,并假定这些文件夹仍然存在。但是,在构建第一个轮子后,clean脚本将删除构建的文件夹(这是必需的,因为setuptools将重用该文件夹而不清除它(

我唯一可行的解决方案是在应用setup之前分叉该过程:

pid = os.fork()
if pid == 0:
setup(...)
sys.exit(0)
os.waitpid(pid, 0)

但是由于这看起来很脏,而且我的主要进程非常占用内存,所以我宁愿避免使用这种方法。

所以我的主要问题是:有没有办法制造一个没有任何副作用的轮子?还是有没有办法在应用setup后重置setuptools模块的状态?在最佳世界中,我想在内存PyFilesystem中创建轮子,并且只将轮子写入磁盘。

我不确定setuptools是否应该以这种方式使用。据我所知,pipand co.(wheel,setuptools等(并没有真正的公共API,或者至少没有友好的API。

distlib库看起来像是具有实际 API 的有前途的替代方案。请参阅distlib关于"使用 wheel API"的文档。

如果这不起作用,那么我可能会尝试其中之一:

subprocess.check_call([sys.executable, '-m', 'wheel', 'pack', 'target/build123'])

请参阅wheel pack文档。

subprocess.check_call([sys.executable, '-m', 'pip', 'wheel', 'target/build123'])

请参阅pip wheel文档。它不能与 API 调用一起使用的原因在 pip 文档的"从你的程序中使用pip"部分有说明。


有一个有点类似的问题,有一些有趣的想法:

  • 如何从 Python 中运行"python setup.py install"?

也许在运行时动态生成setup.py会有所帮助。

最新更新