This blog post reveals a threat against the ethereum network that was present from the merger to the Dencun hard fork.
Background
Before the merger, different message size limits were set for RPC communication to protect clients from denial of service (DOS) attacks. These limits, applied to messages received via HTTP endpoints, were moved to the engine API, which plays a crucial role in connecting execution and consensus layer clients during block production. Due to the involvement of the engine API in block production, it was possible to produce blocks that exceeded the RPC size limits of some customers but remained within the acceptable range for others.
<img alt="" src="https://storage.googleapis.com/ethereum-hackmd/upload_63a6c5b3b29a2b29d6076ce38e03cf82.png” class=”chakra-image css-hw6q2r”/>
If an attacker creates a message that exceeds the client size limit with the lowest configuration, while still meeting the gas limit requirements, and then waits for a block to be produced, it could result in a situation where Some customers consider the block. as valid, while others reject it and issue an HTTP error code “413: Content too large”.
Impact
An attacker who could craft these messages could force the majority of nodes (=geth) to reject blocks that a minority would accept. These blocks would be forked and the proposer would lose rewards.
<img alt="" src="https://storage.googleapis.com/ethereum-hackmd/upload_043bfcd82c1fbd1d8485048b8b7874f2.png” class=”chakra-image css-hw6q2r”/>
At first we thought it was only possible to create these blocks using constructors or a modified version of a client. Geth has a built-in limit of 128 KB for transactions, which means that a large transaction like the one we are analyzing would not end up in the transaction groups of any geth node. However, it was still possible to trigger the limit by having a client with a higher limit propose the block and the CL request validation of this proposed larger block.
We proposed a solution to temporarily reduce the RPC limit on all clients to the lowest value (5 MB). This would invalidate the block and an attacker would be greatly limited in the chaos they can cause on the network as most nodes would reject their blocks.
However, on February 7 we discovered that it was possible to create a block that would hit the 5 MB limit with a set of transactions that are below the 128 KB limit and do not exceed 30 million gas.
This is a bigger problem because we realized that an attacker could create a bunch of high-payout transactions and send them to the network. Since it pays more than everyone else in the mempool, each node (even geth nodes) would include the attack transactions in its block, thus creating a block that would not be accepted by the majority of the network, resulting in many forks. (all considered valid by the minority nodes) and the chain continues to reorganize itself over and over again.
Later on February 7, we concluded that everyone increasing their RPC limits would be the safest alternative.
Timeline
- 2024-02-06 13:00: Toni (EF), Pari (EF) and Justin (Besu) try to send a specifically ground transaction to the network. The transaction contributes blocks of up to 2.7 MB when quickly compressed.
- 2024-02-06 13:25: Pari receives errors from her local Geth node even though the transaction should be valid.
- 2024-02-06 15:14: Justin managed to put the transaction in a block and sent it through the Besu client.
- 2024-02-06 20:46: Sam (EF) alerts Pari (special thanks to misticryuujin in X), Toni and Alex about certain Sepolia nodes fighting.
- 2024-02-06 21:05: The team checks again with Marius of Geth and confirms the error.
- 2024-02-06 21:10: The gang meets to purge it.
- 2024-02-07 23:40: We decided to have all clients limit their RPC request limit to 5 MB
- 2024-02-07 6:40: We discovered that there could be a larger problem and that the attack can be executed with transactions smaller than 128 KB.
- 2024-02-07 10:00: We decided to have all clients increase the RPC request limit.
- 2024-02-07 21:00: Solution merged into geth.
- 2024-02-09: Geth was released
While Geth was the only customer affected by this bug, other customers also updated their defaults to be safe from this attack even if gas limits are increased. Customer teams indicated that the following updates have safe rpc limits:
Geth: v1.13.12
Lesser Mind: v1.25.4
Believe: 24.1.2
Erigon: v2.58.0
Reth: v0.1.0-alpha.18
