Riding the Roller Coaster: Gas Fee Spikes Demystified

Polygon Labs

July 3, 2023

The dynamics of network fees can be complicated and gas prices are one of the most common questions people have about Polygon PoS. This is a deep dive into EIP-1559 and a closer look at what happens on the network when the gas prices change rapidly.

Let’s take a look at a specific event that occurred in March. Researchers at Polygon Labs received inquiries about user transaction fees going over 4 MATIC when the median average should have been around 0.05 MATIC. That prompted an investigation to identify the root cause of the fee spikes and find a way to solve the issue in order to ensure that the Polygon network remains affordable and usable.

The investigation targeted a 40-hour period between 2023-03-21 00:00:00 and 2023-03-22 17:45:50. The Polygon Labs engineering team discovered that only a very small percentage (< 0.03%) of transactions were priced above 4 MATIC. Indeed, most of the other transactions were priced correctly.

Using a wider aperture, it is clear that high fees were rare. By narrowing down and focusing on time ranges when the fees are high, we can get some interesting insights. In this visual, each dot is a transaction. The blue dots were successful, and the red dots were failures. What stands out most are the inorganic and unnatural-looking failures. Those transactions are being priced unusually.

After filtering down, removing successful transactions and only looking at transactions that were priced at a very high level around 1 MATIC or more, we could see that there were around 1,340 failed transactions and 98% of these transactions came from the same smart contract. The average cost per transaction should be around ~$0.015 on Polygon PoS if we take into account that 1 MATIC is worth ~$0.9. These transactions were priced 60 times higher than the usual transactions. This contract was overpaying for gas, running to exhaustion and utilizing a lot of space in blocks. Suspicious right? Let's dive in.

We used investigation tools such as PolygonScan, Tenderly, and Arkham Intelligence to understand what the smart contract was doing. We found that the smart contract in question was involved in arbitrage activities. It would acquire assets from decentralized exchanges (DEXes) such as Uniswap and SushiSwap at lower prices and then sell them on other exchanges at higher prices, bringing both exchange prices back to equilibrium while earning a profit. The primary functions invoked in this contract were `arbV3Iterative3`, `uniswapV3SwapCallback` and `jetswapCall`, leaving no room for ambiguity regarding the contract's purpose.

The individuals responsible for deploying the smart contract also deployed a proxy contract. This proxy contract enabled them to update the bytecode of the arbitrage smart contract without changing its address. Additionally, they funded multiple bot accounts with MATIC, which interacted with the arbitrage smart contract. These bots competed against each other by sending identical transactions with the same parameters to the arbitrage smart contract simultaneously. Here are a few examples of these bots: found here, here, and here. By the way, these bots are still actively operating on Polygon PoS.

A visual representation of the network of bots can be seen here. The MATIC dispenser smart contract is positioned in the center, surrounded by an "army" of bots, while the account owner can be identified on the left side. This strongly echoes the Dark Forest concept as described by G. Konstantopoulos, portraying the Ethereum mempool as a fierce battleground where competing bots engage in relentless competition.

This strategy has proven to be highly profitable, indeed, these bots managed to extract around $230,000 in a few months! Unfortunately, the smart contract is not verified on PolygonScan so the inner workings of the smart contract remain unclear. One approach to gaining insights into its implementation would involve decompiling the bytecode, which can be done using tools like Dedaub. Though, even if the contract can be decompiled, understanding the logic behind it can be highly complex.

But why was this smart contract spamming expensive transactions on the network? That's a great reason to dive into the MEV rabbit hole! 🐇Fasten your seat belt, this is going to be fun!

Wait, what does MEV mean? MEV stands for Maximal Extractable Value which refers to "the maximum value that can be extracted from block production in excess of the standard block reward and gas fees by including, excluding, and changing the order of transactions in a block", credits to ethereum.org for the definition. There are different forms of MEV, including DEX arbitrage, liquidations, sandwich trading, NFT MEV, etc.

Coming back to our suspicious smart contract spamming transactions. On the Polygon network and other Layer 2 blockchains, without the existence and adoption of solutions like Flashbots by the validators, most of the MEV activity resolves into Priority Gas Auctions. PGAs create an environment where bots compete against each other by bidding up transaction fees (gas) which drives fees for other users. This is considered "bad" MEV because it impacts users negatively, causing huge gas spikes on the network.

MEV is not all bad. Some MEV activities are useful for DeFi projects as well as users such as arbitraging MEV, to ensure that users get the best prices for their tokens. In the context of the arbitrage contract, due to the high throughput and low transaction fees on the Polygon PoS network, a strategy of spamming transactions can be seen as advantageous. Initially, engaging in transaction spam to drive up fees may result in some financial loss. However, over time, the expectation is that the profitability of the arbitrage contract will outweigh these initial losses, ultimately compensating for them.

Some potential solutions we are working on to address such behavior would be to quantify the impact of MEV on Polygon PoS. This initiative was initially started with the FlashBabies report which focuses on the impact of MEV on L2s such as Polygon and Optimism. Such an initiative was also started on the Flashbots forum in October 2022 or by other projects such as mev-bor and Fastlane.

Why is it important? In proof-of-work and proof-of-stake systems, miners and validators have the power to manipulate the order of transactions to their advantage, potentially extracting additional value from the transactions, before a block is added to the blockchain. The Proposer-Builder Separation (PBS) concept aims to mitigate this by separating the decision-making process of transaction ordering from the block proposal process. By separating these stages, the goal is to reduce the influence of miners or validators in manipulating the order of transactions and capturing additional value, ensuring a fairer and more secure blockchain ecosystem. Bots will have a dedicated channel to make their bids, which will reduce the number of spam, front-running and sandwiching transactions on the network and this will also encourage the use of good MEV which is better for users.

Base fee mechanics of EIP-1559 on Polygon

The other topic worth digging into is the base fee mechanics of EIP-1559.

This EIP simplifies Ethereum's fee market. Previously, gas fees were calculated via a price-first auction method, as on the Bitcoin network. Users would bid an amount of ether for miners/validators to validate their transactions. Since miners/validators' goal is to make a profit, the higher bidder would always win. This would lead to massive inefficiencies. Users would have to wait multiple blocks before seeing their transaction included in a block. And it was very challenging to predict how much fees you would pay. In the end, most users would overpay for gas, introducing elements of instability. The UX was very very bad.

EIP-1559 splits fees into two parts, base fee and priority fee (or tip). The base fee is the amount required to get your transaction introduced in the next block. It is adjusted algorithmically based on network congestion by ±12.5%, which makes fees more predictable. The priority fee is an optional fee that incentivises a miner to verify a transaction sooner during times of high congestion. Note that the miner/validator only receives the priority fee. Indeed, the base fee is burned.

This resulted in significant improvement in fee stability as illustrated by this graph.

It is important to note that the EIP-1599 did not lower the average gas price, instead it enables for easier fee estimation for users. This concept is well explained in this article: “Before EIP-1559, users paid the entirety of their bids, so they risk overpaying transaction fees if the network condition turns out less congested after they bid. With the new TFM (transaction fee dynamics), however, such risks are avoided, because users can set two parameters in their bids: a fee cap and a tip for the miner on top of the base fee.”

High gas fees and spikes on Ethereum is a scalability issue, it is not related to the transaction fee mechanism. Because Ethereum can only process a few hundred transactions per block on average, fees are going to be higher and higher as we onboard new users. There is a lot of work going on on L2 scaling solutions to increase the amount of transactions that can be processed per block, which will help lower the fees.

The formula to compute the next base fee is a geometric sequence with:

`b` as the base fee (`b_n+1` for the base fee of the next block and `b_n` for the base fee of the previous block),
`u` as the total amount of gas used in the current block,
`t` as the target amount of gas for a block divided by the elasticity multiplier coefficient (set to 2 on both Ethereum and Polygon PoS]),
`d` as the base fee change denominator (set to 8 on Ethereum and 16 on Polygon PoS after the Delhi hard fork - this is the only difference between the computation of EIP-1559 on Ethereum and Polygon PoS).

Assuming that blocks are fully utilized on Polygon PoS (t_n=u_n/2), we get b_n+1=(17/16)*b_n. This means that base fees swing between ±6.25%, illustrated by the diagram below, half as much in terms of percentage variation compared to fees on Ethereum.

The Polygon team uncovered that in a ~20-block period, the base fee jumped from 439 Gwei to 1,601 Gwei. This is something we can see in the first red incline of the third figure. Given that each block on Polygon PoS can increase the base by 6.25%. When blocks are fully utilized, like what this smart contract was doing, the base fee could increase by roughly 500% in about 1 minute.

A solution could be to try to prevent the base fee from increasing and decreasing too rapidly. Since Polygon produces blocks around 6 times faster than Ethereum (around 12s for Ethereum block time and 2s for Polygon PoS), we are investigating ways to decrease the `ElasticityMultiplier` or increase the `BaseFeeChangeDenominator` to dampen these swings.

Conclusion

The investigation into fee spikes on the Polygon network revealed the importance of addressing such behavior to ensure a cheap and usable network. Potential solutions currently being investigated involve adjusting some parameters of the EIP-1559 computation such as `ElasticityMultiplier` and `BaseFeeChangeDenominator`, to prevent rapid fluctuations in the base fee.

Additionally, the team is working on quantifying the impact of MEV on Polygon PoS and exploring options and mitigations. By implementing these measures, the Polygon network can maintain a fairer, more secure, and cost-effective ecosystem for its users.

Check out the Technical Blog for more in-depth content on the workings of Polygon protocols. Tune into the Polygon Labs Blog and our social channels to keep up with all of the updates about the Polygon ecosystem.

Together, we can build an equitable future for all through the mass adoption of Web3!

‍