Closed
Description
System information
| Type | Version/Name |
|---|---|
| Distribution Name | Proxmox VE |
| Distribution Version | 8.2 |
| Kernel Version | Linux 6.8.12-2-pve |
| Architecture | x86_64 |
| OpenZFS Version | zfs-2.2.6-pve1 + #16546, zfs-kmod-2.2.6-pve1 |
Describe the problem you're observing
This is a continuation of #16521. I'm still seeing a similar problem after applying PR #16546. The MFU size (mfusz) kept shrinking despite having a much higher MFU target (mfutg) and some ghost hits. arcstat was updated every minute. There're no file deletions or overwrites during the whole period. The MFU size still kept shrinking after this but the trace would get very long. On the other hand, MRU size (mrusz) kept growing despite being way over target (mrutg).
./arcstat.py -f hits,hit%,mfudt,mfuda,mfumt,mfume,mfutg,mfusz,mfued,mfuem,mfu,mfug,mfugsz,mrudt,mruda,mrumt,mrume,mrutg,mrusz,mrued,mruem,mru,mrug,mrugsz,l2hits,l2hit% 60
hits hit% mfudt mfuda mfumt mfume mfutg mfusz mfued mfuem mfu mfug mfugsz mrudt mruda mrumt mrume mrutg mrusz mrued mruem mru mrug mrugsz l2hits l2hit%
0.0 100 24.6G 13.6G 4.2G 1.9G 28.8G 15.5G 12.8G 1.6G 0.0 0.0 16.4G 13.4G 30.7G 4.2G 205M 17.6G 30.9G 28.9G 143M 0.0 0.0 7.9G 0.0 0.0
3.9K 98.5 24.6G 13.6G 4.2G 1.9G 28.8G 15.5G 12.7G 1.6G 2.6K 3.0 16.3G 13.4G 30.6G 4.2G 206M 17.5G 30.8G 29.0G 143M 1.3K 5.6 7.8G 21.0 34.9
3.7K 98.5 24.7G 13.5G 4.2G 1.9G 28.9G 15.3G 12.8G 1.6G 2.2K 0.3 16.4G 13.4G 30.9G 4.2G 205M 17.6G 31.1G 29.1G 143M 1.5K 0.2 7.8G 16.8 28.8
3.5K 98.5 24.7G 13.3G 4.2G 1.9G 28.9G 15.2G 12.7G 1.6G 2.2K 0.3 16.5G 13.4G 31.2G 4.2G 205M 17.6G 31.4G 29.2G 143M 1.3K 0.1 7.7G 17.4 33.2
4.7K 98.7 24.8G 13.4G 4.2G 1.9G 29.0G 15.2G 12.7G 1.6G 3.3K 0.6 16.6G 13.5G 31.2G 4.2G 205M 17.7G 31.4G 29.3G 143M 1.4K 0.2 7.7G 13.7 22.2
3.8K 98.6 24.8G 13.3G 4.2G 1.9G 29.0G 15.2G 12.6G 1.6G 2.4K 0.1 16.6G 13.4G 31.2G 4.2G 204M 17.6G 31.4G 29.4G 143M 1.3K 0.0 7.7G 13.2 24.2
3.1K 98.6 24.7G 13.3G 4.2G 1.9G 28.8G 15.2G 12.5G 1.6G 2.0K 0.5 16.5G 13.4G 31.0G 4.2G 205M 17.6G 31.2G 29.2G 142M 1.1K 0.0 7.7G 13.4 29.4
3.0K 98.5 24.6G 13.3G 4.2G 1.9G 28.8G 15.2G 12.5G 1.6G 2.0K 0.1 16.5G 13.3G 30.9G 4.2G 206M 17.5G 31.1G 29.1G 142M 1.0K 0.1 7.7G 8.8 19.1
3.3K 98.5 24.7G 13.4G 4.2G 1.9G 28.9G 15.3G 12.5G 1.6G 2.2K 0.1 16.5G 13.4G 30.9G 4.2G 206M 17.6G 31.1G 29.3G 142M 1.1K 0.1 7.7G 10.2 20.1
3.6K 98.8 24.7G 13.4G 4.2G 1.9G 28.9G 15.3G 12.5G 1.6G 2.5K 0.1 16.6G 13.4G 31.0G 4.2G 206M 17.6G 31.2G 29.4G 142M 1.1K 0.0 7.7G 10.0 22.9
3.5K 98.7 24.7G 13.4G 4.2G 1.9G 28.9G 15.3G 12.5G 1.6G 2.4K 0.1 16.5G 13.4G 31.0G 4.2G 206M 17.6G 31.2G 29.4G 142M 1.1K 0.0 7.8G 9.4 20.0
3.2K 98.5 24.6G 13.2G 4.2G 1.9G 28.7G 15.1G 12.3G 1.6G 2.0K 0.2 16.5G 13.3G 30.9G 4.2G 205M 17.5G 31.1G 29.2G 142M 1.2K 0.0 7.7G 11.3 23.3
Include any warning/errors/backtraces from the system logs
It's strange that arc_evict_impl() for MFU data (the 4th ones?) would still be positive despite MFU size that's way lower than MFU target. Perhaps the amount to be evicted should be based on the difference from its target instead?
bpftrace -e 'kretfunc:arc_evict { printf("%-10u: %12llu arc_evict()\n", elapsed / 1e6, retval); } kretfunc:arc_evict_impl { printf("%-10u: %12llu arc_evict_impl(%llu, %d, %12lld)\n", elapsed / 1e6, retval, args->state, args->type, args->bytes); }' > trace.txt
Attaching 2 probes...
6684 : 0 arc_evict_impl(18446744072650004352, 1, -4261878060)
6684 : 0 arc_evict_impl(18446744072650003840, 1, -6721473112)
6685 : 71876608 arc_evict_impl(18446744072650004352, 0, 71827456)
6685 : 9961472 arc_evict_impl(18446744072650003840, 0, 9912320)
6685 : 0 arc_evict_impl(18446744072650004096, 0, -15076608)
6685 : 0 arc_evict_impl(18446744072650004096, 1, -24729084928)
6685 : 37617664 arc_evict_impl(18446744072650003584, 0, 37522944)
6685 : 0 arc_evict_impl(18446744072650003584, 1, -23827943424)
6685 : 81838080 arc_evict()
6685 : 0 arc_evict_impl(18446744072650004352, 1, -4258904096)
6685 : 0 arc_evict_impl(18446744072650003840, 1, -6715525185)
6685 : 131072 arc_evict_impl(18446744072650004352, 0, 41224)
6685 : 0 arc_evict_impl(18446744072650003840, 0, -89848)
6685 : 0 arc_evict_impl(18446744072650004096, 0, -14945536)
6685 : 0 arc_evict_impl(18446744072650004096, 1, -24729084928)
6685 : 0 arc_evict_impl(18446744072650003584, 0, -94720)
6685 : 0 arc_evict_impl(18446744072650003584, 1, -23827943424)
6685 : 131072 arc_evict()
15816 : 0 arc_evict_impl(18446744072650004352, 1, -4262470526)
15816 : 0 arc_evict_impl(18446744072650003840, 1, -6723243261)
15817 : 54554624 arc_evict_impl(18446744072650004352, 0, 54525952)
15818 : 27262976 arc_evict_impl(18446744072650003840, 0, 27234304)
15818 : 52559872 arc_evict_impl(18446744072650004096, 0, 52531712)
15818 : 0 arc_evict_impl(18446744072650004096, 1, -24723512576)
15818 : 20840448 arc_evict_impl(18446744072650003584, 0, 20770048)
15818 : 0 arc_evict_impl(18446744072650003584, 1, -23822663680)
15818 : 81817600 arc_evict()
26170 : 0 arc_evict_impl(18446744072650004352, 1, -4260845910)
26170 : 0 arc_evict_impl(18446744072650003840, 1, -6721949357)
26171 : 69754880 arc_evict_impl(18446744072650004352, 0, 69730304)
26171 : 12161024 arc_evict_impl(18446744072650003840, 0, 12034048)
26172 : 70778880 arc_evict_impl(18446744072650004096, 0, 70656256)
26172 : 0 arc_evict_impl(18446744072650004096, 1, -24720819712)
26172 : 14942208 arc_evict_impl(18446744072650003584, 0, 14870016)
26172 : 0 arc_evict_impl(18446744072650003584, 1, -23820948480)
26172 : 81915904 arc_evict()
36348 : 0 arc_evict_impl(18446744072650004352, 1, -4261536779)
36348 : 0 arc_evict_impl(18446744072650003840, 1, -6723004438)
36349 : 60919808 arc_evict_impl(18446744072650004352, 0, 60817408)
36350 : 20922368 arc_evict_impl(18446744072650003840, 0, 20869120)
36350 : 66453504 arc_evict_impl(18446744072650004096, 0, 66369792)
36350 : 0 arc_evict_impl(18446744072650004096, 1, -24719767296)
36350 : 17039360 arc_evict_impl(18446744072650003584, 0, 16988416)
36350 : 0 arc_evict_impl(18446744072650003584, 1, -23819732736)
36350 : 81842176 arc_evict()
67618 : 0 arc_evict_impl(18446744072650004352, 1, -4275821808)
67618 : 0 arc_evict_impl(18446744072650003840, 1, -6754869801)
67619 : 34095104 arc_evict_impl(18446744072650004352, 0, 34079112)
67620 : 47710208 arc_evict_impl(18446744072650003840, 0, 47694216)
67621 : 81395712 arc_evict_impl(18446744072650004096, 0, 81352448)
67621 : 0 arc_evict_impl(18446744072650004096, 1, -24790857472)
67621 : 0 arc_evict_impl(18446744072650003584, 0, -72238848)
67621 : 0 arc_evict_impl(18446744072650003584, 1, -23892090112)
67621 : 81805312 arc_evict()
67621 : 0 arc_evict_impl(18446744072650004352, 1, -4269879464)
67621 : 0 arc_evict_impl(18446744072650003840, 1, -6742984107)
67621 : 131072 arc_evict_impl(18446744072650004352, 0, 113992)
67621 : 0 arc_evict_impl(18446744072650003840, 0, -17080)
67621 : 131072 arc_evict_impl(18446744072650004096, 0, 87808)
67621 : 0 arc_evict_impl(18446744072650004096, 1, -24790923008)
67621 : 0 arc_evict_impl(18446744072650003584, 0, -72304384)
67621 : 0 arc_evict_impl(18446744072650003584, 1, -23892155648)
67621 : 131072 arc_evict()
76487 : 0 arc_evict_impl(18446744072650004352, 1, -4272755501)
76487 : 0 arc_evict_impl(18446744072650003840, 1, -6749594702)
76489 : 63545344 arc_evict_impl(18446744072650004352, 0, 63438848)
76489 : 18350080 arc_evict_impl(18446744072650003840, 0, 18243584)
76489 : 71172096 arc_evict_impl(18446744072650004096, 0, 71105024)
76489 : 0 arc_evict_impl(18446744072650004096, 1, -24783807744)
76489 : 0 arc_evict_impl(18446744072650003584, 0, -52513792)
76489 : 0 arc_evict_impl(18446744072650003584, 1, -23885469440)
76489 : 81895424 arc_evict()
96047 : 0 arc_evict_impl(18446744072650004352, 1, -4279804485)
96047 : 0 arc_evict_impl(18446744072650003840, 1, -6766208742)
96048 : 53604352 arc_evict_impl(18446744072650004352, 0, 53477376)
96048 : 28184576 arc_evict_impl(18446744072650003840, 0, 28184576)
96049 : 66322432 arc_evict_impl(18446744072650004096, 0, 66242048)
96049 : 0 arc_evict_impl(18446744072650004096, 1, -24817620480)
96049 : 0 arc_evict_impl(18446744072650003584, 0, -69669376)
96049 : 0 arc_evict_impl(18446744072650003584, 1, -23920539648)
96049 : 81788928 arc_evict()
106342 : 0 arc_evict_impl(18446744072650004352, 1, -4278327781)
106342 : 0 arc_evict_impl(18446744072650003840, 1, -6764556567)
106343 : 58224640 arc_evict_impl(18446744072650004352, 0, 58195968)
106343 : 12582912 arc_evict_impl(18446744072650003840, 0, 12554240)
106343 : 66191360 arc_evict_impl(18446744072650004096, 0, 66077952)
106343 : 0 arc_evict_impl(18446744072650004096, 1, -24813767168)
106343 : 0 arc_evict_impl(18446744072650003584, 0, -59833856)
106343 : 0 arc_evict_impl(18446744072650003584, 1, -23917337088)
106343 : 70807552 arc_evict()
116642 : 0 arc_evict_impl(18446744072650004352, 1, -4276836922)
116642 : 0 arc_evict_impl(18446744072650003840, 1, -6764254936)
116643 : 67178496 arc_evict_impl(18446744072650004352, 0, 67108864)
116643 : 14680064 arc_evict_impl(18446744072650003840, 0, 14610432)
116644 : 75104256 arc_evict_impl(18446744072650004096, 0, 75088128)
116644 : 0 arc_evict_impl(18446744072650004096, 1, -24806566656)
116644 : 0 arc_evict_impl(18446744072650003584, 0, -46053632)
116644 : 0 arc_evict_impl(18446744072650003584, 1, -23911476736)
116644 : 81858560 arc_evict()
125871 : 0 arc_evict_impl(18446744072650004352, 1, -4277824720)
125871 : 0 arc_evict_impl(18446744072650003840, 1, -6769202904)
125872 : 41472000 arc_evict_impl(18446744072650004352, 0, 41418752)
125873 : 40427520 arc_evict_impl(18446744072650003840, 0, 40316928)
125873 : 66191360 arc_evict_impl(18446744072650004096, 0, 66132736)
125873 : 0 arc_evict_impl(18446744072650004096, 1, -24801835264)
125873 : 0 arc_evict_impl(18446744072650003584, 0, -23940096)
125873 : 0 arc_evict_impl(18446744072650003584, 1, -23908231424)
125873 : 81899520 arc_evict()
155690 : 0 arc_evict_impl(18446744072650004352, 1, -4291273264)
155690 : 0 arc_evict_impl(18446744072650003840, 1, -6795745371)
155690 : 25223168 arc_evict_impl(18446744072650004352, 0, 25165824)
155691 : 56623104 arc_evict_impl(18446744072650003840, 0, 56565760)
155691 : 52690944 arc_evict_impl(18446744072650004096, 0, 52680192)
155691 : 0 arc_evict_impl(18446744072650004096, 1, -24869155584)
155691 : 0 arc_evict_impl(18446744072650003584, 0, -61692928)
155691 : 0 arc_evict_impl(18446744072650003584, 1, -23975373312)
155691 : 81846272 arc_evict()
176397 : 0 arc_evict_impl(18446744072650004352, 1, -4292092503)
176397 : 0 arc_evict_impl(18446744072650003840, 1, -6799604492)
176398 : 69320704 arc_evict_impl(18446744072650004352, 0, 69206016)
176398 : 12513280 arc_evict_impl(18446744072650003840, 0, 12468224)
176399 : 85065728 arc_evict_impl(18446744072650004096, 0, 84956160)
176399 : 0 arc_evict_impl(18446744072650004096, 1, -24891399680)
176399 : 0 arc_evict_impl(18446744072650003584, 0, -84867328)
176399 : 0 arc_evict_impl(18446744072650003584, 1, -23998727680)
176399 : 81833984 arc_evict()
185689 : 0 arc_evict_impl(18446744072650004352, 1, -4290072052)
185689 : 0 arc_evict_impl(18446744072650003840, 1, -6795507422)
185689 : 50884608 arc_evict_impl(18446744072650004352, 0, 50855936)
185690 : 31014912 arc_evict_impl(18446744072650003840, 0, 30904320)
185690 : 62390272 arc_evict_impl(18446744072650004096, 0, 62281216)
185690 : 0 arc_evict_impl(18446744072650004096, 1, -24872522752)
185690 : 0 arc_evict_impl(18446744072650003584, 0, -46061056)
185690 : 0 arc_evict_impl(18446744072650003584, 1, -23979822848)
185690 : 81899520 arc_evict()
206344 : 0 arc_evict_impl(18446744072650004352, 1, -4302018529)
206344 : 0 arc_evict_impl(18446744072650003840, 1, -6821287096)
206345 : 45088768 arc_evict_impl(18446744072650004352, 0, 45088768)
206346 : 36749312 arc_evict_impl(18446744072650003840, 0, 36700160)
206346 : 9306112 arc_evict_impl(18446744072650004096, 0, 9199616)
206346 : 0 arc_evict_impl(18446744072650004096, 1, -24937528320)
206346 : 0 arc_evict_impl(18446744072650003584, 0, -38930688)
206346 : 0 arc_evict_impl(18446744072650003584, 1, -24045770752)
206346 : 81838080 arc_evict()
206363 : 0 arc_evict_impl(18446744072650004352, 1, -4293655249)
206363 : 0 arc_evict_impl(18446744072650003840, 1, -6804559122)
206364 : 65904640 arc_evict_impl(18446744072650004352, 0, 65868608)
206364 : 17281024 arc_evict_impl(18446744072650003840, 0, 17265472)
206365 : 75235328 arc_evict_impl(18446744072650004096, 0, 75118592)
206365 : 0 arc_evict_impl(18446744072650004096, 1, -24896656384)
206365 : 10616832 arc_evict_impl(18446744072650003584, 0, 10602240)
206365 : 0 arc_evict_impl(18446744072650003584, 1, -24004898816)
206365 : 83185664 arc_evict()
225999 : 0 arc_evict_impl(18446744072650004352, 1, -4301906210)
225999 : 0 arc_evict_impl(18446744072650003840, 1, -6821043781)
226000 : 59351040 arc_evict_impl(18446744072650004352, 0, 59244544)
226001 : 22491136 arc_evict_impl(18446744072650003840, 0, 22437888)
226001 : 44305408 arc_evict_impl(18446744072650004096, 0, 44254976)
226001 : 0 arc_evict_impl(18446744072650004096, 1, -24943371008)
226001 : 0 arc_evict_impl(18446744072650003584, 0, -12317440)
226001 : 0 arc_evict_impl(18446744072650003584, 1, -24051986688)
226001 : 81842176 arc_evict()
236395 : 0 arc_evict_impl(18446744072650004352, 1, -4304070803)
236395 : 0 arc_evict_impl(18446744072650003840, 1, -6825000231)
236395 : 32579584 arc_evict_impl(18446744072650004352, 0, 32505856)
236396 : 49324032 arc_evict_impl(18446744072650003840, 0, 49209344)
236397 : 50859008 arc_evict_impl(18446744072650004096, 0, 50757888)
236397 : 0 arc_evict_impl(18446744072650004096, 1, -24941874432)
236397 : 20447232 arc_evict_impl(18446744072650003584, 0, 20425984)
236397 : 0 arc_evict_impl(18446744072650003584, 1, -24050303744)
236397 : 81903616 arc_evict()
246592 : 0 arc_evict_impl(18446744072650004352, 1, -4304221177)
246592 : 0 arc_evict_impl(18446744072650003840, 1, -6825915378)
246594 : 62406656 arc_evict_impl(18446744072650004352, 0, 62390272)
246594 : 19398656 arc_evict_impl(18446744072650003840, 0, 19382272)
246595 : 72089600 arc_evict_impl(18446744072650004096, 0, 72029440)
246595 : 0 arc_evict_impl(18446744072650004096, 1, -24957583360)
246595 : 0 arc_evict_impl(18446744072650003584, 0, -5312000)
246595 : 0 arc_evict_impl(18446744072650003584, 1, -24066319872)
246595 : 81805312 arc_evict()
255500 : 0 arc_evict_impl(18446744072650004352, 1, -4301396266)
255500 : 0 arc_evict_impl(18446744072650003840, 1, -6820802133)
255501 : 61358080 arc_evict_impl(18446744072650004352, 0, 61341696)
255502 : 20539904 arc_evict_impl(18446744072650003840, 0, 20430848)
255502 : 61603840 arc_evict_impl(18446744072650004096, 0, 61541120)
255502 : 0 arc_evict_impl(18446744072650004096, 1, -24941160960)
255503 : 31719424 arc_evict_impl(18446744072650003584, 0, 31601664)
255503 : 0 arc_evict_impl(18446744072650003584, 1, -24050165760)
255503 : 81897984 arc_evict()
256171 : 0 arc_evict_impl(18446744072650004352, 1, -4289918705)
256171 : 0 arc_evict_impl(18446744072650003840, 1, -6797724130)
256173 : 81825792 arc_evict_impl(18446744072650004352, 0, 81788928)
256173 : 0 arc_evict_impl(18446744072650003840, 0, -36864)
256173 : 82837504 arc_evict_impl(18446744072650004096, 0, 82833152)
256173 : 0 arc_evict_impl(18446744072650004096, 1, -24899761664)
256174 : 41287680 arc_evict_impl(18446744072650003584, 0, 41253888)
256174 : 0 arc_evict_impl(18446744072650003584, 1, -24008705024)
256174 : 81825792 arc_evict()
266540 : 0 arc_evict_impl(18446744072650004352, 1, -4293052574)
266540 : 0 arc_evict_impl(18446744072650003840, 1, -6803540797)
266541 : 66650112 arc_evict_impl(18446744072650004352, 0, 66584576)
266541 : 15204352 arc_evict_impl(18446744072650003840, 0, 15138816)
266542 : 66322432 arc_evict_impl(18446744072650004096, 0, 66273280)
266542 : 0 arc_evict_impl(18446744072650004096, 1, -24894225408)
266542 : 21889024 arc_evict_impl(18446744072650003584, 0, 21838080)
266542 : 0 arc_evict_impl(18446744072650003584, 1, -24002943232)
266542 : 81854464 arc_evict()
286276 : 0 arc_evict_impl(18446744072650004352, 1, -4296793968)
286276 : 0 arc_evict_impl(18446744072650003840, 1, -6807367905)
286278 : 67137536 arc_evict_impl(18446744072650004352, 0, 67108864)
286278 : 14680064 arc_evict_impl(18446744072650003840, 0, 14651392)
286278 : 61997056 arc_evict_impl(18446744072650004096, 0, 61874688)
286278 : 0 arc_evict_impl(18446744072650004096, 1, -24915537920)
286278 : 0 arc_evict_impl(18446744072650003584, 0, -2342656)
286278 : 0 arc_evict_impl(18446744072650003584, 1, -24022427904)
286278 : 81817600 arc_evict()
286278 : 0 arc_evict_impl(18446744072650004352, 1, -4293447413)
286278 : 0 arc_evict_impl(18446744072650003840, 1, -6800674795)
286278 : 655360 arc_evict_impl(18446744072650004352, 0, 531040)
286278 : 0 arc_evict_impl(18446744072650003840, 0, -124320)
286278 : 393216 arc_evict_impl(18446744072650004096, 0, 336384)
286278 : 0 arc_evict_impl(18446744072650004096, 1, -24915537920)
286278 : 0 arc_evict_impl(18446744072650003584, 0, -2146048)
286278 : 0 arc_evict_impl(18446744072650003584, 1, -24022427904)
286278 : 655360 arc_evict()
305853 : 0 arc_evict_impl(18446744072650004352, 1, -4300664698)
305853 : 0 arc_evict_impl(18446744072650003840, 1, -6812330228)
305854 : 43585536 arc_evict_impl(18446744072650004352, 0, 43515904)
305854 : 38248448 arc_evict_impl(18446744072650003840, 0, 38203392)
305855 : 40239104 arc_evict_impl(18446744072650004096, 0, 40152320)
305855 : 0 arc_evict_impl(18446744072650004096, 1, -24929458944)
305855 : 23986176 arc_evict_impl(18446744072650003584, 0, 23953408)
305855 : 0 arc_evict_impl(18446744072650003584, 1, -24034959360)
305855 : 81833984 arc_evict()
306013 : 0 arc_evict_impl(18446744072650004352, 1, -4292780085)
306013 : 0 arc_evict_impl(18446744072650003840, 1, -6796561003)
306014 : 55488512 arc_evict_impl(18446744072650004352, 0, 55469208)
306014 : 31006720 arc_evict_impl(18446744072650003840, 0, 30913688)
306015 : 71303168 arc_evict_impl(18446744072650004096, 0, 71243776)
306015 : 0 arc_evict_impl(18446744072650004096, 1, -24888580096)
306015 : 56623104 arc_evict_impl(18446744072650003584, 0, 56547328)
306015 : 0 arc_evict_impl(18446744072650003584, 1, -23994080512)
306015 : 86495232 arc_evict()
316248 : 0 arc_evict_impl(18446744072650004352, 1, -4291800128)
316248 : 0 arc_evict_impl(18446744072650003840, 1, -6790611584)
316249 : 51957760 arc_evict_impl(18446744072650004352, 0, 51904512)
316250 : 29958144 arc_evict_impl(18446744072650003840, 0, 29831168)
316250 : 47054848 arc_evict_impl(18446744072650004096, 0, 47041280)
316250 : 0 arc_evict_impl(18446744072650004096, 1, -24882132992)
316251 : 40108032 arc_evict_impl(18446744072650003584, 0, 40072448)
316251 : 0 arc_evict_impl(18446744072650003584, 1, -23985638656)
316251 : 81915904 arc_evict()
326519 : 0 arc_evict_impl(18446744072650004352, 1, -4287893340)
326519 : 0 arc_evict_impl(18446744072650003840, 1, -6782350521)
326520 : 67234816 arc_evict_impl(18446744072650004352, 0, 67108864)
326521 : 14585856 arc_evict_impl(18446744072650003840, 0, 14554112)
326521 : 63700992 arc_evict_impl(18446744072650004096, 0, 63611648)
326521 : 0 arc_evict_impl(18446744072650004096, 1, -24869262336)
326521 : 30539776 arc_evict_impl(18446744072650003584, 0, 30472448)
326521 : 0 arc_evict_impl(18446744072650003584, 1, -23972544256)
326521 : 81820672 arc_evict()
326599 : 0 arc_evict_impl(18446744072650004352, 1, -4280034307)
326599 : 0 arc_evict_impl(18446744072650003840, 1, -6766591495)
326600 : 70815744 arc_evict_impl(18446744072650004352, 0, 70807960)
326600 : 11534336 arc_evict_impl(18446744072650003840, 0, 11526552)
326601 : 77070336 arc_evict_impl(18446744072650004096, 0, 76980992)
326601 : 0 arc_evict_impl(18446744072650004096, 1, -24828457984)
326601 : 46661632 arc_evict_impl(18446744072650003584, 0, 46645504)
326601 : 0 arc_evict_impl(18446744072650003584, 1, -23931719424)
326601 : 82350080 arc_evict()
335823 : 0 arc_evict_impl(18446744072650004352, 1, -4273227441)
335823 : 0 arc_evict_impl(18446744072650003840, 1, -6749978978)
335825 : 81850368 arc_evict_impl(18446744072650004352, 0, 81788928)
335825 : 0 arc_evict_impl(18446744072650003840, 0, -61440)
335825 : 74186752 arc_evict_impl(18446744072650004096, 0, 74142720)
335825 : 0 arc_evict_impl(18446744072650004096, 1, -24809163008)
335825 : 25690112 arc_evict_impl(18446744072650003584, 0, 25560064)
335825 : 0 arc_evict_impl(18446744072650003584, 1, -23910925056)
335825 : 81850368 arc_evict()
346131 : 0 arc_evict_impl(18446744072650004352, 1, -4274956693)
346131 : 0 arc_evict_impl(18446744072650003840, 1, -6753127722)
346132 : 61870080 arc_evict_impl(18446744072650004352, 0, 61865984)
346132 : 20037632 arc_evict_impl(18446744072650003840, 0, 19918848)
346132 : 64749568 arc_evict_impl(18446744072650004096, 0, 64687616)
346132 : 0 arc_evict_impl(18446744072650004096, 1, -24796529152)
346133 : 28180480 arc_evict_impl(18446744072650003584, 0, 28144640)
346133 : 0 arc_evict_impl(18446744072650003584, 1, -23898136320)
346133 : 81907712 arc_evict()
356471 : 0 arc_evict_impl(18446744072650004352, 1, -4272904165)
356471 : 0 arc_evict_impl(18446744072650003840, 1, -6748594634)
356472 : 54124544 arc_evict_impl(18446744072650004352, 0, 54001664)
356473 : 27746304 arc_evict_impl(18446744072650003840, 0, 27664384)
356473 : 65404928 arc_evict_impl(18446744072650004096, 0, 65400320)
356473 : 0 arc_evict_impl(18446744072650004096, 1, -24779870720)
356474 : 33816576 arc_evict_impl(18446744072650003584, 0, 33744896)
356474 : 0 arc_evict_impl(18446744072650003584, 1, -23881263872)
356474 : 81870848 arc_evict()
365761 : 0 arc_evict_impl(18446744072650004352, 1, -4270644875)
365761 : 0 arc_evict_impl(18446744072650003840, 1, -6739784983)
365763 : 67158016 arc_evict_impl(18446744072650004352, 0, 67108864)
365763 : 14680064 arc_evict_impl(18446744072650003840, 0, 14630912)
365763 : 72876032 arc_evict_impl(18446744072650004096, 0, 72846336)
365763 : 0 arc_evict_impl(18446744072650004096, 1, -24774011904)
365763 : 13500416 arc_evict_impl(18446744072650003584, 0, 13437696)
365763 : 0 arc_evict_impl(18446744072650003584, 1, -23873259520)
365763 : 81838080 arc_evict()
375996 : 0 arc_evict_impl(18446744072650004352, 1, -4268358954)
375996 : 0 arc_evict_impl(18446744072650003840, 1, -6732575316)
375997 : 59346944 arc_evict_impl(18446744072650004352, 0, 59244544)
375997 : 22552576 arc_evict_impl(18446744072650003840, 0, 22441984)
375998 : 61213696 arc_evict_impl(18446744072650004096, 0, 61183488)
375998 : 0 arc_evict_impl(18446744072650004096, 1, -24757458432)
375998 : 36962304 arc_evict_impl(18446744072650003584, 0, 36925184)
375998 : 0 arc_evict_impl(18446744072650003584, 1, -23855387136)
375998 : 81899520 arc_evict()
386274 : 0 arc_evict_impl(18446744072650004352, 1, -4264567590)
386274 : 0 arc_evict_impl(18446744072650003840, 1, -6723605068)
386276 : 81334272 arc_evict_impl(18446744072650004352, 0, 81264640)
386276 : 524288 arc_evict_impl(18446744072650003840, 0, 454656)
386276 : 66060288 arc_evict_impl(18446744072650004096, 0, 65934592)
386276 : 0 arc_evict_impl(18446744072650004096, 1, -24744105216)
386276 : 28311552 arc_evict_impl(18446744072650003584, 0, 28196096)
386276 : 0 arc_evict_impl(18446744072650003584, 1, -23841340160)
386276 : 81858560 arc_evict()
396515 : 0 arc_evict_impl(18446744072650004352, 1, -4263138776)
396515 : 0 arc_evict_impl(18446744072650003840, 1, -6718576561)
396516 : 51380224 arc_evict_impl(18446744072650004352, 0, 51380224)
396517 : 30412800 arc_evict_impl(18446744072650003840, 0, 30408704)
396517 : 55574528 arc_evict_impl(18446744072650004096, 0, 55550208)
396517 : 0 arc_evict_impl(18446744072650004096, 1, -24727584512)
396517 : 40894464 arc_evict_impl(18446744072650003584, 0, 40785152)
396517 : 0 arc_evict_impl(18446744072650003584, 1, -23823734016)
396517 : 81793024 arc_evict()
405788 : 0 arc_evict_impl(18446744072650004352, 1, -4262465517)
405788 : 0 arc_evict_impl(18446744072650003840, 1, -6716480474)
405789 : 57720832 arc_evict_impl(18446744072650004352, 0, 57671680)
405789 : 24105984 arc_evict_impl(18446744072650003840, 0, 24068096)
405790 : 64356352 arc_evict_impl(18446744072650004096, 0, 64303104)
405790 : 0 arc_evict_impl(18446744072650004096, 1, -24724881408)
405790 : 19660800 arc_evict_impl(18446744072650003584, 0, 19550464)
405790 : 0 arc_evict_impl(18446744072650003584, 1, -23820656128)
405790 : 81826816 arc_evict()
405790 : 0 arc_evict_impl(18446744072650004352, 1, -4257987431)
405790 : 0 arc_evict_impl(18446744072650003840, 1, -6707524302)
405790 : 131072 arc_evict_impl(18446744072650004352, 0, 99616)
405790 : 0 arc_evict_impl(18446744072650003840, 0, -31456)
405790 : 131072 arc_evict_impl(18446744072650004096, 0, 77824)
405790 : 0 arc_evict_impl(18446744072650004096, 1, -24724946944)
405790 : 0 arc_evict_impl(18446744072650003584, 0, -175872)
405790 : 0 arc_evict_impl(18446744072650003584, 1, -23820721664)
405790 : 131072 arc_evict()
416026 : 0 arc_evict_impl(18446744072650004352, 1, -4263983107)
416026 : 0 arc_evict_impl(18446744072650003840, 1, -6716366855)
416027 : 47820800 arc_evict_impl(18446744072650004352, 0, 47710208)
416028 : 34058240 arc_evict_impl(18446744072650003840, 0, 33968128)
416028 : 54919168 arc_evict_impl(18446744072650004096, 0, 54882816)
416028 : 0 arc_evict_impl(18446744072650004096, 1, -24728397312)
416028 : 22282240 arc_evict_impl(18446744072650003584, 0, 22248192)
416028 : 0 arc_evict_impl(18446744072650003584, 1, -23822597632)
416028 : 81879040 arc_evict()
426280 : 0 arc_evict_impl(18446744072650004352, 1, -4261691190)
426280 : 0 arc_evict_impl(18446744072650003840, 1, -6710312557)
426281 : 65561600 arc_evict_impl(18446744072650004352, 0, 65536000)
426282 : 16352256 arc_evict_impl(18446744072650003840, 0, 16227328)
426282 : 57016320 arc_evict_impl(18446744072650004096, 0, 56915200)
426282 : 0 arc_evict_impl(18446744072650004096, 1, -24727681536)
426282 : 24772608 arc_evict_impl(18446744072650003584, 0, 24714752)
426282 : 0 arc_evict_impl(18446744072650003584, 1, -23821146624)
426282 : 81913856 arc_evict()
456106 : 0 arc_evict_impl(18446744072650004352, 1, -4273832527)
456106 : 0 arc_evict_impl(18446744072650003840, 1, -6731828894)
456108 : 50892800 arc_evict_impl(18446744072650004352, 0, 50855936)
456108 : 30896128 arc_evict_impl(18446744072650003840, 0, 30896128)
456108 : 20316160 arc_evict_impl(18446744072650004096, 0, 20264960)
456108 : 0 arc_evict_impl(18446744072650004096, 1, -24785158144)
456108 : 2359296 arc_evict_impl(18446744072650003584, 0, 2232320)
456108 : 0 arc_evict_impl(18446744072650003584, 1, -23877240064)
456108 : 81788928 arc_evict()
456108 : 0 arc_evict_impl(18446744072650004352, 1, -4268920859)
456108 : 0 arc_evict_impl(18446744072650003840, 1, -6721743414)
456108 : 925696 arc_evict_impl(18446744072650004352, 0, 884712)
456108 : 0 arc_evict_impl(18446744072650003840, 0, -40984)
456108 : 524288 arc_evict_impl(18446744072650004096, 0, 407552)
456108 : 0 arc_evict_impl(18446744072650004096, 1, -24785154048)
456108 : 393216 arc_evict_impl(18446744072650003584, 0, 335872)
456108 : 0 arc_evict_impl(18446744072650003584, 1, -23877104896)
456108 : 925696 arc_evict()
466364 : 0 arc_evict_impl(18446744072650004352, 1, -4268984430)
466364 : 0 arc_evict_impl(18446744072650003840, 1, -6722771677)
466366 : 58744832 arc_evict_impl(18446744072650004352, 0, 58720256)
466366 : 23093248 arc_evict_impl(18446744072650003840, 0, 23044096)
466367 : 62914560 arc_evict_impl(18446744072650004096, 0, 62814208)
466367 : 0 arc_evict_impl(18446744072650004096, 1, -24765405184)
466367 : 39583744 arc_evict_impl(18446744072650003584, 0, 39477248)
466367 : 0 arc_evict_impl(18446744072650003584, 1, -23857806592)
466367 : 81838080 arc_evict()
515981 : 0 arc_evict_impl(18446744072650004352, 1, -4291261688)
515981 : 0 arc_evict_impl(18446744072650003840, 1, -6770275825)
515982 : 51916800 arc_evict_impl(18446744072650004352, 0, 51904512)
515983 : 29950976 arc_evict_impl(18446744072650003840, 0, 29872128)
515983 : 5111808 arc_evict_impl(18446744072650004096, 0, 5102336)
515983 : 0 arc_evict_impl(18446744072650004096, 1, -24882740736)
515983 : 0 arc_evict_impl(18446744072650003584, 0, -39820288)
515983 : 0 arc_evict_impl(18446744072650003584, 1, -23976616960)
515983 : 81867776 arc_evict()
526230 : 0 arc_evict_impl(18446744072650004352, 1, -4287587506)
526230 : 0 arc_evict_impl(18446744072650003840, 1, -6762597732)
526231 : 51933184 arc_evict_impl(18446744072650004352, 0, 51904512)
526232 : 29876224 arc_evict_impl(18446744072650003840, 0, 29855744)
526232 : 69074944 arc_evict_impl(18446744072650004096, 0, 69047040)
526232 : 0 arc_evict_impl(18446744072650004096, 1, -24864807936)
526232 : 0 arc_evict_impl(18446744072650003584, 0, -9128960)
526232 : 0 arc_evict_impl(18446744072650003584, 1, -23958519296)
526232 : 81809408 arc_evict()
536700 : 0 arc_evict_impl(18446744072650004352, 1, -4286624942)
536700 : 0 arc_evict_impl(18446744072650003840, 1, -6760631645)
536701 : 59355136 arc_evict_impl(18446744072650004352, 0, 59244544)
536702 : 22548480 arc_evict_impl(18446744072650003840, 0, 22433792)
536702 : 74186752 arc_evict_impl(18446744072650004096, 0, 74105088)
536702 : 0 arc_evict_impl(18446744072650004096, 1, -24860753408)
536702 : 3538944 arc_evict_impl(18446744072650003584, 0, 3443712)
536702 : 0 arc_evict_impl(18446744072650003584, 1, -23954444288)
536702 : 81903616 arc_evict()
544938 : 0 arc_evict_impl(18446744072650004352, 1, -4282579913)
544938 : 0 arc_evict_impl(18446744072650003840, 1, -6752801171)
544939 : 65048576 arc_evict_impl(18446744072650004352, 0, 65011712)
544940 : 16863232 arc_evict_impl(18446744072650003840, 0, 16740352)
544940 : 66715648 arc_evict_impl(18446744072650004096, 0, 66645504)
544940 : 0 arc_evict_impl(18446744072650004096, 1, -24842323456)
544941 : 33947648 arc_evict_impl(18446744072650003584, 0, 33821184)
544941 : 0 arc_evict_impl(18446744072650003584, 1, -23936144128)
544941 : 81911808 arc_evict()
556760 : 0 arc_evict_impl(18446744072650004352, 1, -4280877356)
556760 : 0 arc_evict_impl(18446744072650003840, 1, -6747372632)
556761 : 48869376 arc_evict_impl(18446744072650004352, 0, 48758784)
556761 : 33030144 arc_evict_impl(18446744072650003840, 0, 32919552)
556762 : 51511296 arc_evict_impl(18446744072650004096, 0, 51469824)
556762 : 0 arc_evict_impl(18446744072650004096, 1, -24826600960)
556762 : 45613056 arc_evict_impl(18446744072650003584, 0, 45511168)
556762 : 0 arc_evict_impl(18446744072650003584, 1, -23919409920)
556762 : 81899520 arc_evict()
576248 : 0 arc_evict_impl(18446744072650004352, 1, -4284262454)
576248 : 0 arc_evict_impl(18446744072650003840, 1, -6753334381)
576249 : 67670016 arc_evict_impl(18446744072650004352, 0, 67633152)
576249 : 14151680 arc_evict_impl(18446744072650003840, 0, 14118912)
576250 : 46137344 arc_evict_impl(18446744072650004096, 0, 46037248)
576250 : 0 arc_evict_impl(18446744072650004096, 1, -24852856320)
576250 : 9306112 arc_evict_impl(18446744072650003584, 0, 9181696)
576250 : 0 arc_evict_impl(18446744072650003584, 1, -23945261056)
576250 : 81821696 arc_evict()
586474 : 0 arc_evict_impl(18446744072650004352, 1, -4283969438)
586474 : 0 arc_evict_impl(18446744072650003840, 1, -6751663420)
586476 : 42035712 arc_evict_impl(18446744072650004352, 0, 41943040)
586476 : 39845888 arc_evict_impl(18446744072650003840, 0, 39753216)
586477 : 54788096 arc_evict_impl(18446744072650004096, 0, 54695936)
586477 : 0 arc_evict_impl(18446744072650004096, 1, -24840038912)
586477 : 39714816 arc_evict_impl(18446744072650003584, 0, 39711744)
586477 : 0 arc_evict_impl(18446744072650003584, 1, -23931901184)
586477 : 81881600 arc_evict()
594825 : 0 arc_evict_impl(18446744072650004352, 1, -4279037700)
594825 : 0 arc_evict_impl(18446744072650003840, 1, -6741105160)
594826 : 65630208 arc_evict_impl(18446744072650004352, 0, 65536000)
594826 : 16252928 arc_evict_impl(18446744072650003840, 0, 16158720)
594827 : 65798144 arc_evict_impl(18446744072650004096, 0, 65721344)
594827 : 0 arc_evict_impl(18446744072650004096, 1, -24822124032)
594827 : 34078720 arc_evict_impl(18446744072650003584, 0, 33984256)
594827 : 0 arc_evict_impl(18446744072650003584, 1, -23913638912)
594827 : 81883136 arc_evict()
595773 : 0 arc_evict_impl(18446744072650004352, 1, -4272474809)
595773 : 0 arc_evict_impl(18446744072650003840, 1, -6727725426)
595774 : 50900992 arc_evict_impl(18446744072650004352, 0, 50855936)
595775 : 30908416 arc_evict_impl(18446744072650003840, 0, 30887936)
595775 : 66715648 arc_evict_impl(18446744072650004096, 0, 66626560)
595775 : 0 arc_evict_impl(18446744072650004096, 1, -24780463616)
595776 : 57016320 arc_evict_impl(18446744072650003584, 0, 57009920)
595776 : 0 arc_evict_impl(18446744072650003584, 1, -23871851520)
595776 : 81809408 arc_evict()
606091 : 0 arc_evict_impl(18446744072650004352, 1, -4269646061)
606091 : 0 arc_evict_impl(18446744072650003840, 1, -6720523738)
606092 : 55177216 arc_evict_impl(18446744072650004352, 0, 55050240)
606092 : 26738688 arc_evict_impl(18446744072650003840, 0, 26611712)
606092 : 55053312 arc_evict_impl(18446744072650004096, 0, 55007232)
606092 : 0 arc_evict_impl(18446744072650004096, 1, -24764541440)
606093 : 42991616 arc_evict_impl(18446744072650003584, 0, 42905344)
606093 : 0 arc_evict_impl(18446744072650003584, 1, -23855157248)
606093 : 81915904 arc_evict()
616435 : 0 arc_evict_impl(18446744072650004352, 1, -4269701569)
616435 : 0 arc_evict_impl(18446744072650003840, 1, -6718652290)
616436 : 62967808 arc_evict_impl(18446744072650004352, 0, 62914560)
616436 : 18870272 arc_evict_impl(18446744072650003840, 0, 18821120)
616437 : 72482816 arc_evict_impl(18446744072650004096, 0, 72359680)
616437 : 0 arc_evict_impl(18446744072650004096, 1, -24763276544)
616437 : 9961472 arc_evict_impl(18446744072650003584, 0, 9890048)
616437 : 0 arc_evict_impl(18446744072650003584, 1, -23852901120)
616437 : 81838080 arc_evict()
625669 : 0 arc_evict_impl(18446744072650004352, 1, -4267514483)
625669 : 0 arc_evict_impl(18446744072650003840, 1, -6716049358)
625670 : 49840128 arc_evict_impl(18446744072650004352, 0, 49807360)
625670 : 31981568 arc_evict_impl(18446744072650003840, 0, 31948800)
625671 : 66715648 arc_evict_impl(18446744072650004096, 0, 66676224)
625671 : 0 arc_evict_impl(18446744072650004096, 1, -24753357568)
625671 : 24772608 arc_evict_impl(18446744072650003584, 0, 24693248)
625671 : 0 arc_evict_impl(18446744072650003584, 1, -23843488000)
625671 : 81821696 arc_evict()
625754 : 0 arc_evict_impl(18446744072650004352, 1, -4259609190)
625754 : 0 arc_evict_impl(18446744072650003840, 1, -6700196471)
625755 : 59191296 arc_evict_impl(18446744072650004352, 0, 59079824)
625756 : 24117248 arc_evict_impl(18446744072650003840, 0, 24005776)
625756 : 71434240 arc_evict_impl(18446744072650004096, 0, 71394816)
625756 : 0 arc_evict_impl(18446744072650004096, 1, -24712499968)
625757 : 53084160 arc_evict_impl(18446744072650003584, 0, 52978176)
625757 : 0 arc_evict_impl(18446744072650003584, 1, -23802609920)
625757 : 83308544 arc_evict()
635964 : 0 arc_evict_impl(18446744072650004352, 1, -4259411152)
635964 : 0 arc_evict_impl(18446744072650003840, 1, -6699366110)
635965 : 45158400 arc_evict_impl(18446744072650004352, 0, 45088768)
635966 : 36753408 arc_evict_impl(18446744072650003840, 0, 36630528)
635966 : 69468160 arc_evict_impl(18446744072650004096, 0, 69445376)
635966 : 0 arc_evict_impl(18446744072650004096, 1, -24701868800)
635966 : 23461888 arc_evict_impl(18446744072650003584, 0, 23393024)
635966 : 0 arc_evict_impl(18446744072650003584, 1, -23791761664)
635966 : 81911808 arc_evict()
646275 : 0 arc_evict_impl(18446744072650004352, 1, -4256539175)
646275 : 0 arc_evict_impl(18446744072650003840, 1, -6694163872)
646276 : 60915712 arc_evict_impl(18446744072650004352, 0, 60817408)
646276 : 20901888 arc_evict_impl(18446744072650003840, 0, 20873216)
646277 : 78774272 arc_evict_impl(18446744072650004096, 0, 78699520)
646277 : 0 arc_evict_impl(18446744072650004096, 1, -24693369088)
646277 : 11796480 arc_evict_impl(18446744072650003584, 0, 11701248)
646277 : 0 arc_evict_impl(18446744072650003584, 1, -23783533056)
646277 : 81817600 arc_evict()
656487 : 0 arc_evict_impl(18446744072650004352, 1, -4254165838)
656487 : 0 arc_evict_impl(18446744072650003840, 1, -6690134062)
656488 : 67117056 arc_evict_impl(18446744072650004352, 0, 67108864)
656489 : 14684160 arc_evict_impl(18446744072650003840, 0, 14671872)
656489 : 80216064 arc_evict_impl(18446744072650004096, 0, 80135936)
656489 : 0 arc_evict_impl(18446744072650004096, 1, -24683825408)
656489 : 12320768 arc_evict_impl(18446744072650003584, 0, 12315136)
656489 : 0 arc_evict_impl(18446744072650003584, 1, -23774348032)
656489 : 81801216 arc_evict()
665752 : 0 arc_evict_impl(18446744072650004352, 1, -4254005841)
665752 : 0 arc_evict_impl(18446744072650003840, 1, -6689716266)
665753 : 58286080 arc_evict_impl(18446744072650004352, 0, 58195968)
665753 : 23527424 arc_evict_impl(18446744072650003840, 0, 23502848)
665754 : 76677120 arc_evict_impl(18446744072650004096, 0, 76559360)
665754 : 0 arc_evict_impl(18446744072650004096, 1, -24678737664)
665754 : 10092544 arc_evict_impl(18446744072650003584, 0, 10026752)
665754 : 0 arc_evict_impl(18446744072650003584, 1, -23769211392)
665754 : 81813504 arc_evict()
688965 : 0 arc_evict_impl(18446744072650004352, 1, -4269671341)
688965 : 0 arc_evict_impl(18446744072650003840, 1, -6725813645)
688966 : 47788032 arc_evict_impl(18446744072650004352, 0, 47710208)
688966 : 34075136 arc_evict_impl(18446744072650003840, 0, 34000896)
688967 : 83886080 arc_evict_impl(18446744072650004096, 0, 83775232)
688967 : 0 arc_evict_impl(18446744072650004096, 1, -24761656064)
688967 : 0 arc_evict_impl(18446744072650003584, 0, -83821312)
688967 : 0 arc_evict_impl(18446744072650003584, 1, -23854511616)
688967 : 81863168 arc_evict()