Git gc 使用大量内存,即使我限制了它

  • 本文关键字:内存 gc Git git memory
  • 更新时间 :
  • 英文 :


我在共享主机上运行 git(uberspace.de),并在执行git pushgit gc时达到内存限制。他们报告的内存消耗约为 1500 MB。

所以我听从了

https://stackoverflow.com/a/8761464/288568

它似乎曾经工作过一次,但是在大量提交之后,它不再有帮助,内存消耗再次约为1500MB,git pack进程被杀死。

我提交一个 120 MB 的 SQL 转储,更改很少(每次大约 5 行) - 所以打包的存储库只有大约 150 MB,但解压缩了大约 3 GB(我达到了 bitbucket 的限制)。

我看到哪个pmap正在打开很多包对象 - 是增加高内存的吗?如何限制这种情况?

最后,我收到"打包对象死于信号 15"和来自主机的邮件,我的进程被杀死了。

sh-4.1$ git config --list|grep pack
pack.windowmemory=25m
pack.packsizelimit=25m
pack.threads=1
pack.deltecachesize=25m
sh-4.1$ git gc
Counting objects: 2586, done.
^Zmpressing objects:  15% (163/1085)   
[1]+  Stopped(SIGTSTP)        git gc
sh-4.1$ bg
[1] git gc &
sh-4.1$ pmap 14190
14190:   git pack-objects --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects --unpack-unreachable=2.weeks.ago --local --delta-base-offset .git/objects/pack/.tmp-14187-pack
0000000000400000   1848K r-x--  /home/foo/.toast/pkg/git/v2.11.1/1/root/libexec/git-core/git
00000000007ce000     36K rw---  /home/foo/.toast/pkg/git/v2.11.1/1/root/libexec/git-core/git
00000000007d7000    280K rw---    [ anon ]
0000000000da1000   1680K rw---    [ anon ]
00007f735dc5d000 161648K rw---    [ anon ]
00007f7377555000 149580K rw---    [ anon ]
00007f73915ee000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-be75b796793b738f7566a4be06491cfd3ad9c236.pack
00007f73990b6000 125592K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-33af25946a2fcb1c3e284b3093ac0eafa440cf46.pack
00007f73a0b5c000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-dc37fbdaf5ad7f6c317a7c3d5ffecf53daa045b6.pack
00007f73a8624000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-c7d6e0b1e4da6a2aaa5eff3cad27f6fcbb38a5c9.pack
00007f73b00ec000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-bbef922979f633183f0bc207b5e52608b36d6434.pack
00007f73b7bb4000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-1468b8f7d29c999297b2411f9b94d86b0cd9965d.pack
00007f73bf67c000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-3ecc274dd42f4783926224457225968b1fe1d95b.pack
00007f73c7144000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-bb77ce8e83062bf2d26a3366b5b4e0f77c3a3810.pack
00007f73cec0c000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-937dae9140ac11d33831a697f9347f98baa25894.pack
00007f73d66d4000 125348K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-d497de57bd0e623e5338eb553e0940ea2442274f.pack
00007f73de13d000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-188241ed7cb01328a4c32b09b0aff82311e32a1c.pack
00007f73e5c05000 125720K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-5fc5649f65f8596a6b3dfd7c71d47d34e7b150e5.pack
00007f73ed6cb000 125720K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-aeb01e8db0b4d43181626d8b7a412e813a4a8757.pack
00007f73f5191000 120872K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-fcebe7c0d63e44015ecc186ed74d45cb998c22ef.pack
00007f73fc79b000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-e84a2620505a10807e64727283f608b9661c39c0.pack
00007f7404263000 125720K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-56f5137267429f72a6e57ec70a637dc944cfeec7.pack
00007f740bd29000 125720K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-564815c1b986e15fa385daf51d6ecc9f44a07679.pack
00007f74137ef000 120868K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-4c28c08cbf0279f71e3c0a97cc91639fd6a8b308.pack
00007f741adf8000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-7e2d986a0faf6e464ff240b5bb21bb1b446b329f.pack
00007f74228c0000    484K rw---    [ anon ]
00007f7422939000  96840K r----  /usr/lib/locale/locale-archive
00007f74287cb000      8K r-x--  /lib64/libdl-2.12.so
00007f74287cd000   2048K -----  /lib64/libdl-2.12.so
00007f74289cd000      4K r----  /lib64/libdl-2.12.so
00007f74289ce000      4K rw---  /lib64/libdl-2.12.so
00007f74289cf000   1576K r-x--  /lib64/libc-2.12.so
00007f7428b59000   2048K -----  /lib64/libc-2.12.so
00007f7428d59000     16K r----  /lib64/libc-2.12.so
00007f7428d5d000      8K rw---  /lib64/libc-2.12.so
00007f7428d5f000     16K rw---    [ anon ]
00007f7428d63000     92K r-x--  /lib64/libpthread-2.12.so
00007f7428d7a000   2048K -----  /lib64/libpthread-2.12.so
00007f7428f7a000      4K r----  /lib64/libpthread-2.12.so
00007f7428f7b000      4K rw---  /lib64/libpthread-2.12.so
00007f7428f7c000     16K rw---    [ anon ]
00007f7428f80000     28K r-x--  /lib64/librt-2.12.so
00007f7428f87000   2044K -----  /lib64/librt-2.12.so
00007f7429186000      4K r----  /lib64/librt-2.12.so
00007f7429187000      4K rw---  /lib64/librt-2.12.so
00007f7429188000   1768K r-x--  /usr/lib64/libcrypto.so.1.0.1e
00007f7429342000   2044K -----  /usr/lib64/libcrypto.so.1.0.1e
00007f7429541000    108K r----  /usr/lib64/libcrypto.so.1.0.1e
00007f742955c000     48K rw---  /usr/lib64/libcrypto.so.1.0.1e
00007f7429568000     16K rw---    [ anon ]
00007f742956c000     84K r-x--  /lib64/libz.so.1.2.3
00007f7429581000   2044K -----  /lib64/libz.so.1.2.3
00007f7429780000      4K r----  /lib64/libz.so.1.2.3
00007f7429781000      4K rw---  /lib64/libz.so.1.2.3
00007f7429782000    128K r-x--  /lib64/ld-2.12.so
00007f74297a6000    464K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-004517110da3d0eaeea50b4eace06633eab4294c.pack
00007f742981a000     52K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-ee7d89bc7f8e5cd3e36c3655d967dfc8af3e6c7b.pack
00007f7429827000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-0381d5fab5a4f18e02eb18c3e12c5585702c9564.pack
00007f7429828000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-f26f972bc47c3a18cf47493e680c90fa3d5c9046.pack
00007f7429829000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-30fe739ba7e266ddf6f09a29fc2aa49eedd9e90b.pack
00007f742982a000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-fb13617b92c223a9d017935c5ed27c9c8b0e62de.pack
00007f742982b000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-d4bc55fff1083c9601bafeb49abf378cdc6f0727.pack
00007f742982c000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-e104e0bffca85b6346ef588c8c63d6ddadc567e3.pack
00007f742982d000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-5eeddad5db2c0a44f33a3c41cd9216a7af0417df.pack
00007f742982e000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-fc2614aa21f9284e0b76b6cc3ad9a90a0bae371e.pack
00007f742982f000      8K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-739869658456c4a7627279fceafa7c7c4c408acb.pack
00007f7429831000     20K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-feabd42e4392c52c8864a0397d10a73d7ada6fc8.pack
00007f7429836000      8K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-26462c3338a2e9302852788bad7b827a98748b65.pack
00007f7429838000      8K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-69b7829b456d813335b339545583ac9585ba9faa.pack
00007f742983a000      8K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-7e87b7b2f517ff2097026c55e98be6d1cec14587.pack
00007f742983c000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-eec773d189c951b5768eb85d33390d3d43db1d9f.pack
00007f742983d000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-25232ec34a92ab095486707f73fe5ad0a92bcb14.pack
00007f742983e000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-5c633980ef7d381a4af2e7bf433c1c86154848c1.pack
00007f742983f000     20K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-2ec52679fc048e2db73cedc5bab30680f9b14bc9.pack
00007f7429875000      4K rw---    [ anon ]
00007f7429876000    400K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-d18bf8e1b6a421fac697607e204da80b6c4c0370.pack
00007f74298da000    512K rw---    [ anon ]
00007f742995a000     44K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-004517110da3d0eaeea50b4eace06633eab4294c.idx
00007f7429965000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-4c28c08cbf0279f71e3c0a97cc91639fd6a8b308.idx
00007f7429966000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-ee7d89bc7f8e5cd3e36c3655d967dfc8af3e6c7b.idx
00007f7429967000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-fcebe7c0d63e44015ecc186ed74d45cb998c22ef.idx
00007f7429968000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-0381d5fab5a4f18e02eb18c3e12c5585702c9564.idx
00007f7429969000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-d497de57bd0e623e5338eb553e0940ea2442274f.idx
00007f742996a000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-f26f972bc47c3a18cf47493e680c90fa3d5c9046.idx
00007f742996b000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-33af25946a2fcb1c3e284b3093ac0eafa440cf46.idx
00007f742996c000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-30fe739ba7e266ddf6f09a29fc2aa49eedd9e90b.idx
00007f742996d000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-56f5137267429f72a6e57ec70a637dc944cfeec7.idx
00007f742996e000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-fb13617b92c223a9d017935c5ed27c9c8b0e62de.idx
00007f742996f000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-aeb01e8db0b4d43181626d8b7a412e813a4a8757.idx
00007f7429970000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-d4bc55fff1083c9601bafeb49abf378cdc6f0727.idx
00007f7429971000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-564815c1b986e15fa385daf51d6ecc9f44a07679.idx
00007f7429972000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-e104e0bffca85b6346ef588c8c63d6ddadc567e3.idx
00007f7429973000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-5fc5649f65f8596a6b3dfd7c71d47d34e7b150e5.idx
00007f7429974000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-5eeddad5db2c0a44f33a3c41cd9216a7af0417df.idx
00007f7429975000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-937dae9140ac11d33831a697f9347f98baa25894.idx
00007f7429976000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-fc2614aa21f9284e0b76b6cc3ad9a90a0bae371e.idx
00007f7429977000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-1468b8f7d29c999297b2411f9b94d86b0cd9965d.idx
00007f7429978000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-739869658456c4a7627279fceafa7c7c4c408acb.idx
00007f7429979000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-bb77ce8e83062bf2d26a3366b5b4e0f77c3a3810.idx
00007f742997a000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-feabd42e4392c52c8864a0397d10a73d7ada6fc8.idx
00007f742997b000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-dc37fbdaf5ad7f6c317a7c3d5ffecf53daa045b6.idx
00007f742997c000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-26462c3338a2e9302852788bad7b827a98748b65.idx
00007f742997d000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-3ecc274dd42f4783926224457225968b1fe1d95b.idx
00007f742997e000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-69b7829b456d813335b339545583ac9585ba9faa.idx
00007f742997f000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-7e2d986a0faf6e464ff240b5bb21bb1b446b329f.idx
00007f7429980000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-7e87b7b2f517ff2097026c55e98be6d1cec14587.idx
00007f7429981000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-bbef922979f633183f0bc207b5e52608b36d6434.idx
00007f7429982000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-5c633980ef7d381a4af2e7bf433c1c86154848c1.idx
00007f7429983000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-e84a2620505a10807e64727283f608b9661c39c0.idx
00007f7429984000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-eec773d189c951b5768eb85d33390d3d43db1d9f.idx
00007f7429985000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-c7d6e0b1e4da6a2aaa5eff3cad27f6fcbb38a5c9.idx
00007f7429986000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-25232ec34a92ab095486707f73fe5ad0a92bcb14.idx
00007f7429987000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-188241ed7cb01328a4c32b09b0aff82311e32a1c.idx
00007f7429988000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-be75b796793b738f7566a4be06491cfd3ad9c236.idx
00007f7429989000     24K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-d18bf8e1b6a421fac697607e204da80b6c4c0370.idx
00007f742998f000     20K rw---    [ anon ]
00007f7429998000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-2ec52679fc048e2db73cedc5bab30680f9b14bc9.idx
00007f7429999000     28K r--s-  /usr/lib64/gconv/gconv-modules.cache
00007f74299a0000      4K rw---    [ anon ]
00007f74299a1000      4K r----  /lib64/ld-2.12.so
00007f74299a2000      4K rw---  /lib64/ld-2.12.so
00007f74299a3000      4K rw---    [ anon ]
00007ffe76939000     84K rw---    [ stack ]
00007ffe7698b000      4K r-x--    [ anon ]
ffffffffff600000      4K r-x--    [ anon ]
total          2809104K
sh-4.1$ error: pack-objects died of signal 15
error: failed to run repack

我有一些大存储库,我必须压缩,我没有注意到任何与内存相关的选项帮助。如果git-pack-objects必须重新打包一些对象,它将不得不映射大致相同的大内存。管理它的唯一方法是限制它必须重新打包的对象数量。

在您的情况下,对象已经压缩在小包文件中(每次推送一个)。因此,git gc必须将这些小包文件重新打包成大文件。您可以尝试在上次成功重新打包创建的大包文件旁边创建空pack-<HASH>.keep文件。那么也许接下来git gc不会再尝试破坏这些对象。

Git 2.18(2018 年第 2 季度)应该有助于检测和缓解过多的内存消耗。

大型存储库中的"git gc"需要花费大量时间,因为它考虑 默认情况下将所有对象重新打包到一个包中.
该命令被教导假装好像最大的现有包文件是 标有".keep",以便在物体在 其他包装和松散的包装将被重新包装。

请参阅提交 5af0504、提交 9806f5a、提交 8fc6776、提交 55dfe13、提交 ae4e89e、提交 ed7e5fc、提交 e9e33ab(2018 年 4 月 15 日),作者 Nguyễn Thái Ngọc Duy (pclouds).
(由 Junio C Hamano --gitster-- 合并于 commit 30b015b,2018 年 5 月 23 日)

GC --自动:如果没有足够的 MEM 来"重新包装 -AD",则排除基本包装

pack-objects可能是一个很大的内存消耗,尤其是在大型存储库中, 每个人都知道.
在巨型基础包上粘贴.keep文件以避免此问题的建议也早已为人所知。

最近的补丁添加了一个选项来执行此操作,但它必须是 手动配置或激活。
此修补程序允许git gc --auto在认为repack -ad将使用大量内存并由于交换或刷新操作系统缓存而开始影响系统时自动激活此模式。

gc --auto决定根据对包对象的估计来执行此操作 内存使用情况,至少对于堆部分来说是相当准确的,并且 这是否适合一半的系统内存(这里的假设是 运行许多其他应用程序的桌面环境)。

仅当未配置gc.bigBasePackThreshold时,此机制才会启动。 如果是,则假定用户已经知道他们想要什么。

git gc手册页现在指出:

如果包数超过gc.autoPackLimit的值, 然后是现有包(标有.keep文件的包除外) 或超过gc.bigPackThreshold限制) 通过使用-A选项合并到单个包中 "git 重新打包"。

如果估计内存量不足以git repack运行平稳,gc.bigPackThreshold未设置,最大 包也将被排除在外(这相当于运行git gc--keep-base-pack)。
gc.autoPackLimit设置为 0 将禁用自动合并 包。


Git 2.30(2021 年第 1 季度)修复了"gc"文档中的选项名称。

参见提交 793c146,提交 db5368b(2020 年 11 月 20 日),作者:Ævar Arnfjörð Bjarmason (avar).
(由 Junio C Hamano --gitster-- 合并于 2020 年 11 月 30 日提交的 1c04cdd)

gc docs: 将--keep-base-pack更改为--keep-largest-pack

报告人: Luc Van Oostenryck
签名者: Ævar Arnfjörð Bjarmason

--keep-base-pack选项在 git.git 中从未存在过。它是该系列早期修订版中--keep-largest-pack选项的名称,然后它登陆为 ae4e89e549("gcadd(man)--keep-largest-pack option",2018-04-15,Git v2.18.0-rc0 - 合并列在批次 #6 中)。

该系列中后来的补丁没有更改为也引用--keep-largest-pack,因此自从该功能最初登陆以来,我们就一直引用了一个不存在的选项。

这些提交是:

  1. 55dfe13df9 ("gc: 添加 gc.bigPackThreshold 配置", 2018-04-15, Git v2.18.0-rc0 -- 合并列在批次 #6)
  2. 9806f5a7bf ("gc --auto: 如果没有足够的 mem 来"repack -ad",则排除基础包,2018-04-15,Git v2.18.0-rc0)

也许是解决方案

我观察到,当我选择 packSizeLimit 非常小时,这是相反的生产 - 它增加了 .git 目录而不是减少它,这也可能导致达到位桶大小限制(未经验证)

1) 将存储库同步到另一台具有更多 RAM 的计算机并正确打包

2)在内存限制下将其同步回服务器

3)增加到pack.packSizeLimit=300m

4)现在经过git gc回购不会疯狂增长

5) 提交数据库转储

6)回购将其大小翻倍

7)直接做git gc

8)尺寸变细

9)现在推送

让我们看看几天后的情况如何...

最新更新