Pebble upgrade plan

[rjl493456442]

Pebble has released Pebble v2, which is unfortunately incompatible with
the v1 version currently used in go-ethereum.

Upgrading to v2 is certainly a desirable long-term goal. However, before
proceeding with a database version upgrade, several measurements and
evaluations must be performed.

System metrics

There is no major difference in terms of system metrics, one notable change
is pebble-v2 (bench05) has higher IO wait.

Database metrics

The notable difference is Pebble-v2 (bench05) has more aggressive compaction workload.

Overall chain progressing metrics

The node with Pebble-v2 (bench05) is slightly faster, roughly 1h ahead after running for 3 days.

Preliminary conclusion

There is no major performance gain from upgrading Pebble directly. However, Pebble team
is actively working on the Blob notion in pebble, which implements WiscKey idea to separate
value from the LSM tree to minimize read and write amplification. It will be a huge gain once
this feature is shipped and well-tested.

All the following updates regarding Pebble will be posted/tracked in this issue.

[rjl493456442]

Second round, by enabling the key-value separation (Pebble-tableformat-v6) feature in the Pebble-v2

---

The benchmark results:

• The size of LSM tree is 50% less
• The compaction workload is about 50% less
• The overall disk read/write is similar
• The state read is slightly slower due to the additional blob log read
• The blockchain processing is slightly slower due to state read

[[ Pebble-v2 ]]
/datadir/geth # du -sh geth/chaindata/ancient/
335.7G	geth/chaindata/ancient/
/datadir/geth # du -sh geth/chaindata/
736.9G	geth/chaindata/

[[ Pebble-v1 ]]
/datadir/geth # du -sh geth/chaindata/ancient/
336.9G	geth/chaindata/ancient/
/datadir/geth # du -sh geth/chaindata/
514.8G	geth/chaindata/

The pebble v2 disk usage (401.2GB) is significantly larger than v1 (177.9GB) due to the stale blob logs.

---