我在共享主机上运行 git(uberspace.de),并在执行git push
或git gc
时达到内存限制。他们报告的内存消耗约为 1500 MB。
所以我听从了
https://stackoverflow.com/a/8761464/288568
它似乎曾经工作过一次,但是在大量提交之后,它不再有帮助,内存消耗再次约为1500MB,git pack进程被杀死。
我提交一个 120 MB 的 SQL 转储,更改很少(每次大约 5 行) - 所以打包的存储库只有大约 150 MB,但解压缩了大约 3 GB(我达到了 bitbucket 的限制)。
我看到哪个pmap正在打开很多包对象 - 是增加高内存的吗?如何限制这种情况?
最后,我收到"打包对象死于信号 15"和来自主机的邮件,我的进程被杀死了。
sh-4.1$ git config --list|grep pack
pack.windowmemory=25m
pack.packsizelimit=25m
pack.threads=1
pack.deltecachesize=25m
sh-4.1$ git gc
Counting objects: 2586, done.
^Zmpressing objects: 15% (163/1085)
[1]+ Stopped(SIGTSTP) git gc
sh-4.1$ bg
[1] git gc &
sh-4.1$ pmap 14190
14190: git pack-objects --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects --unpack-unreachable=2.weeks.ago --local --delta-base-offset .git/objects/pack/.tmp-14187-pack
0000000000400000 1848K r-x-- /home/foo/.toast/pkg/git/v2.11.1/1/root/libexec/git-core/git
00000000007ce000 36K rw--- /home/foo/.toast/pkg/git/v2.11.1/1/root/libexec/git-core/git
00000000007d7000 280K rw--- [ anon ]
0000000000da1000 1680K rw--- [ anon ]
00007f735dc5d000 161648K rw--- [ anon ]
00007f7377555000 149580K rw--- [ anon ]
00007f73915ee000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-be75b796793b738f7566a4be06491cfd3ad9c236.pack
00007f73990b6000 125592K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-33af25946a2fcb1c3e284b3093ac0eafa440cf46.pack
00007f73a0b5c000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-dc37fbdaf5ad7f6c317a7c3d5ffecf53daa045b6.pack
00007f73a8624000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-c7d6e0b1e4da6a2aaa5eff3cad27f6fcbb38a5c9.pack
00007f73b00ec000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-bbef922979f633183f0bc207b5e52608b36d6434.pack
00007f73b7bb4000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-1468b8f7d29c999297b2411f9b94d86b0cd9965d.pack
00007f73bf67c000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-3ecc274dd42f4783926224457225968b1fe1d95b.pack
00007f73c7144000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-bb77ce8e83062bf2d26a3366b5b4e0f77c3a3810.pack
00007f73cec0c000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-937dae9140ac11d33831a697f9347f98baa25894.pack
00007f73d66d4000 125348K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-d497de57bd0e623e5338eb553e0940ea2442274f.pack
00007f73de13d000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-188241ed7cb01328a4c32b09b0aff82311e32a1c.pack
00007f73e5c05000 125720K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-5fc5649f65f8596a6b3dfd7c71d47d34e7b150e5.pack
00007f73ed6cb000 125720K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-aeb01e8db0b4d43181626d8b7a412e813a4a8757.pack
00007f73f5191000 120872K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-fcebe7c0d63e44015ecc186ed74d45cb998c22ef.pack
00007f73fc79b000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-e84a2620505a10807e64727283f608b9661c39c0.pack
00007f7404263000 125720K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-56f5137267429f72a6e57ec70a637dc944cfeec7.pack
00007f740bd29000 125720K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-564815c1b986e15fa385daf51d6ecc9f44a07679.pack
00007f74137ef000 120868K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-4c28c08cbf0279f71e3c0a97cc91639fd6a8b308.pack
00007f741adf8000 125728K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-7e2d986a0faf6e464ff240b5bb21bb1b446b329f.pack
00007f74228c0000 484K rw--- [ anon ]
00007f7422939000 96840K r---- /usr/lib/locale/locale-archive
00007f74287cb000 8K r-x-- /lib64/libdl-2.12.so
00007f74287cd000 2048K ----- /lib64/libdl-2.12.so
00007f74289cd000 4K r---- /lib64/libdl-2.12.so
00007f74289ce000 4K rw--- /lib64/libdl-2.12.so
00007f74289cf000 1576K r-x-- /lib64/libc-2.12.so
00007f7428b59000 2048K ----- /lib64/libc-2.12.so
00007f7428d59000 16K r---- /lib64/libc-2.12.so
00007f7428d5d000 8K rw--- /lib64/libc-2.12.so
00007f7428d5f000 16K rw--- [ anon ]
00007f7428d63000 92K r-x-- /lib64/libpthread-2.12.so
00007f7428d7a000 2048K ----- /lib64/libpthread-2.12.so
00007f7428f7a000 4K r---- /lib64/libpthread-2.12.so
00007f7428f7b000 4K rw--- /lib64/libpthread-2.12.so
00007f7428f7c000 16K rw--- [ anon ]
00007f7428f80000 28K r-x-- /lib64/librt-2.12.so
00007f7428f87000 2044K ----- /lib64/librt-2.12.so
00007f7429186000 4K r---- /lib64/librt-2.12.so
00007f7429187000 4K rw--- /lib64/librt-2.12.so
00007f7429188000 1768K r-x-- /usr/lib64/libcrypto.so.1.0.1e
00007f7429342000 2044K ----- /usr/lib64/libcrypto.so.1.0.1e
00007f7429541000 108K r---- /usr/lib64/libcrypto.so.1.0.1e
00007f742955c000 48K rw--- /usr/lib64/libcrypto.so.1.0.1e
00007f7429568000 16K rw--- [ anon ]
00007f742956c000 84K r-x-- /lib64/libz.so.1.2.3
00007f7429581000 2044K ----- /lib64/libz.so.1.2.3
00007f7429780000 4K r---- /lib64/libz.so.1.2.3
00007f7429781000 4K rw--- /lib64/libz.so.1.2.3
00007f7429782000 128K r-x-- /lib64/ld-2.12.so
00007f74297a6000 464K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-004517110da3d0eaeea50b4eace06633eab4294c.pack
00007f742981a000 52K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-ee7d89bc7f8e5cd3e36c3655d967dfc8af3e6c7b.pack
00007f7429827000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-0381d5fab5a4f18e02eb18c3e12c5585702c9564.pack
00007f7429828000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-f26f972bc47c3a18cf47493e680c90fa3d5c9046.pack
00007f7429829000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-30fe739ba7e266ddf6f09a29fc2aa49eedd9e90b.pack
00007f742982a000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-fb13617b92c223a9d017935c5ed27c9c8b0e62de.pack
00007f742982b000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-d4bc55fff1083c9601bafeb49abf378cdc6f0727.pack
00007f742982c000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-e104e0bffca85b6346ef588c8c63d6ddadc567e3.pack
00007f742982d000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-5eeddad5db2c0a44f33a3c41cd9216a7af0417df.pack
00007f742982e000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-fc2614aa21f9284e0b76b6cc3ad9a90a0bae371e.pack
00007f742982f000 8K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-739869658456c4a7627279fceafa7c7c4c408acb.pack
00007f7429831000 20K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-feabd42e4392c52c8864a0397d10a73d7ada6fc8.pack
00007f7429836000 8K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-26462c3338a2e9302852788bad7b827a98748b65.pack
00007f7429838000 8K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-69b7829b456d813335b339545583ac9585ba9faa.pack
00007f742983a000 8K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-7e87b7b2f517ff2097026c55e98be6d1cec14587.pack
00007f742983c000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-eec773d189c951b5768eb85d33390d3d43db1d9f.pack
00007f742983d000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-25232ec34a92ab095486707f73fe5ad0a92bcb14.pack
00007f742983e000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-5c633980ef7d381a4af2e7bf433c1c86154848c1.pack
00007f742983f000 20K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-2ec52679fc048e2db73cedc5bab30680f9b14bc9.pack
00007f7429875000 4K rw--- [ anon ]
00007f7429876000 400K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-d18bf8e1b6a421fac697607e204da80b6c4c0370.pack
00007f74298da000 512K rw--- [ anon ]
00007f742995a000 44K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-004517110da3d0eaeea50b4eace06633eab4294c.idx
00007f7429965000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-4c28c08cbf0279f71e3c0a97cc91639fd6a8b308.idx
00007f7429966000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-ee7d89bc7f8e5cd3e36c3655d967dfc8af3e6c7b.idx
00007f7429967000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-fcebe7c0d63e44015ecc186ed74d45cb998c22ef.idx
00007f7429968000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-0381d5fab5a4f18e02eb18c3e12c5585702c9564.idx
00007f7429969000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-d497de57bd0e623e5338eb553e0940ea2442274f.idx
00007f742996a000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-f26f972bc47c3a18cf47493e680c90fa3d5c9046.idx
00007f742996b000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-33af25946a2fcb1c3e284b3093ac0eafa440cf46.idx
00007f742996c000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-30fe739ba7e266ddf6f09a29fc2aa49eedd9e90b.idx
00007f742996d000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-56f5137267429f72a6e57ec70a637dc944cfeec7.idx
00007f742996e000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-fb13617b92c223a9d017935c5ed27c9c8b0e62de.idx
00007f742996f000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-aeb01e8db0b4d43181626d8b7a412e813a4a8757.idx
00007f7429970000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-d4bc55fff1083c9601bafeb49abf378cdc6f0727.idx
00007f7429971000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-564815c1b986e15fa385daf51d6ecc9f44a07679.idx
00007f7429972000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-e104e0bffca85b6346ef588c8c63d6ddadc567e3.idx
00007f7429973000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-5fc5649f65f8596a6b3dfd7c71d47d34e7b150e5.idx
00007f7429974000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-5eeddad5db2c0a44f33a3c41cd9216a7af0417df.idx
00007f7429975000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-937dae9140ac11d33831a697f9347f98baa25894.idx
00007f7429976000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-fc2614aa21f9284e0b76b6cc3ad9a90a0bae371e.idx
00007f7429977000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-1468b8f7d29c999297b2411f9b94d86b0cd9965d.idx
00007f7429978000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-739869658456c4a7627279fceafa7c7c4c408acb.idx
00007f7429979000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-bb77ce8e83062bf2d26a3366b5b4e0f77c3a3810.idx
00007f742997a000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-feabd42e4392c52c8864a0397d10a73d7ada6fc8.idx
00007f742997b000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-dc37fbdaf5ad7f6c317a7c3d5ffecf53daa045b6.idx
00007f742997c000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-26462c3338a2e9302852788bad7b827a98748b65.idx
00007f742997d000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-3ecc274dd42f4783926224457225968b1fe1d95b.idx
00007f742997e000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-69b7829b456d813335b339545583ac9585ba9faa.idx
00007f742997f000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-7e2d986a0faf6e464ff240b5bb21bb1b446b329f.idx
00007f7429980000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-7e87b7b2f517ff2097026c55e98be6d1cec14587.idx
00007f7429981000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-bbef922979f633183f0bc207b5e52608b36d6434.idx
00007f7429982000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-5c633980ef7d381a4af2e7bf433c1c86154848c1.idx
00007f7429983000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-e84a2620505a10807e64727283f608b9661c39c0.idx
00007f7429984000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-eec773d189c951b5768eb85d33390d3d43db1d9f.idx
00007f7429985000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-c7d6e0b1e4da6a2aaa5eff3cad27f6fcbb38a5c9.idx
00007f7429986000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-25232ec34a92ab095486707f73fe5ad0a92bcb14.idx
00007f7429987000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-188241ed7cb01328a4c32b09b0aff82311e32a1c.idx
00007f7429988000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-be75b796793b738f7566a4be06491cfd3ad9c236.idx
00007f7429989000 24K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-d18bf8e1b6a421fac697607e204da80b6c4c0370.idx
00007f742998f000 20K rw--- [ anon ]
00007f7429998000 4K r---- /var/www/virtual/foo/bar/.git/objects/pack/pack-2ec52679fc048e2db73cedc5bab30680f9b14bc9.idx
00007f7429999000 28K r--s- /usr/lib64/gconv/gconv-modules.cache
00007f74299a0000 4K rw--- [ anon ]
00007f74299a1000 4K r---- /lib64/ld-2.12.so
00007f74299a2000 4K rw--- /lib64/ld-2.12.so
00007f74299a3000 4K rw--- [ anon ]
00007ffe76939000 84K rw--- [ stack ]
00007ffe7698b000 4K r-x-- [ anon ]
ffffffffff600000 4K r-x-- [ anon ]
total 2809104K
sh-4.1$ error: pack-objects died of signal 15
error: failed to run repack
我有一些大存储库,我必须压缩,我没有注意到任何与内存相关的选项帮助。如果git-pack-objects
必须重新打包一些对象,它将不得不映射大致相同的大内存。管理它的唯一方法是限制它必须重新打包的对象数量。
在您的情况下,对象已经压缩在小包文件中(每次推送一个)。因此,git gc
必须将这些小包文件重新打包成大文件。您可以尝试在上次成功重新打包创建的大包文件旁边创建空pack-<HASH>.keep
文件。那么也许接下来git gc
不会再尝试破坏这些对象。
Git 2.18(2018 年第 2 季度)应该有助于检测和缓解过多的内存消耗。
大型存储库中的"
git gc
"需要花费大量时间,因为它考虑 默认情况下将所有对象重新打包到一个包中.
该命令被教导假装好像最大的现有包文件是 标有".keep
",以便在物体在 其他包装和松散的包装将被重新包装。
请参阅提交 5af0504、提交 9806f5a、提交 8fc6776、提交 55dfe13、提交 ae4e89e、提交 ed7e5fc、提交 e9e33ab(2018 年 4 月 15 日),作者 Nguyễn Thái Ngọc Duy (pclouds
).
(由 Junio C Hamano --gitster
-- 合并于 commit 30b015b,2018 年 5 月 23 日)
GC --自动:如果没有足够的 MEM 来"重新包装 -AD",则排除基本包装
pack-objects
可能是一个很大的内存消耗,尤其是在大型存储库中, 每个人都知道.
在巨型基础包上粘贴.keep
文件以避免此问题的建议也早已为人所知。最近的补丁添加了一个选项来执行此操作,但它必须是 手动配置或激活。
此修补程序允许git gc --auto
在认为repack -ad
将使用大量内存并由于交换或刷新操作系统缓存而开始影响系统时自动激活此模式。
gc --auto
决定根据对包对象的估计来执行此操作 内存使用情况,至少对于堆部分来说是相当准确的,并且 这是否适合一半的系统内存(这里的假设是 运行许多其他应用程序的桌面环境)。仅当未配置
gc.bigBasePackThreshold
时,此机制才会启动。 如果是,则假定用户已经知道他们想要什么。
git gc
手册页现在指出:
如果包数超过
gc.autoPackLimit
的值, 然后是现有包(标有.keep
文件的包除外) 或超过gc.bigPackThreshold
限制) 通过使用-A
选项合并到单个包中 "git 重新打包"。如果估计内存量不足以
git repack
运行平稳,gc.bigPackThreshold
未设置,最大 包也将被排除在外(这相当于运行git gc
与--keep-base-pack
)。
将gc.autoPackLimit
设置为 0 将禁用自动合并 包。
Git 2.30(2021 年第 1 季度)修复了"gc
"文档中的选项名称。
参见提交 793c146,提交 db5368b(2020 年 11 月 20 日),作者:Ævar Arnfjörð Bjarmason (avar
).
(由 Junio C Hamano --gitster
-- 合并于 2020 年 11 月 30 日提交的 1c04cdd)
gc docs
: 将--keep-base-pack
更改为--keep-largest-pack
报告人: Luc Van Oostenryck
签名者: Ævar Arnfjörð Bjarmason
--keep-base-pack
选项在 git.git 中从未存在过。它是该系列早期修订版中--keep-largest-pack
选项的名称,然后它登陆为 ae4e89e549("gc
:add
(man)--keep-largest-pack option",2018-04-15,Git v2.18.0-rc0 - 合并列在批次 #6 中)。该系列中后来的补丁没有更改为也引用
--keep-largest-pack
,因此自从该功能最初登陆以来,我们就一直引用了一个不存在的选项。这些提交是:
- 55dfe13df9 ("
gc
: 添加 gc.bigPackThreshold 配置", 2018-04-15, Git v2.18.0-rc0 -- 合并列在批次 #6)- 9806f5a7bf ("
gc --auto
: 如果没有足够的 mem 来"repack -ad
",则排除基础包,2018-04-15,Git v2.18.0-rc0)
也许是解决方案
我观察到,当我选择 packSizeLimit 非常小时,这是相反的生产 - 它增加了 .git 目录而不是减少它,这也可能导致达到位桶大小限制(未经验证)
1) 将存储库同步到另一台具有更多 RAM 的计算机并正确打包
2)在内存限制下将其同步回服务器
3)增加到pack.packSizeLimit=300m
4)现在经过git gc
回购不会疯狂增长
5) 提交数据库转储
6)回购将其大小翻倍
7)直接做git gc
8)尺寸变细
9)现在推送
让我们看看几天后的情况如何...