All You Need to Know About the Recent Network Upgrade

Earlier this month, Polygon’s core development team with help from bug bounty platform Immunefi successfully fixed a critical network vulnerability. Considering the nature of this upgrade, it had to be executed without attracting too much attention. We are now ready to give the full account of what happened.

A group of whitehat hackers notified Immunefi, which hosts our bug bounty, of a vulnerability in the Polygon PoS genesis contract on Dec. 3. The Polygon core team engaged with the group and Immunefi’s expert team and immediately introduced a fix. The validator and full node communities were notified, and they rallied behind the core devs to upgrade 80% of the network within 24 hours without stoppage.

The upgrade was executed on Dec. 5 at block #22156660 without impacting liveness and performance of the network in any major way. The vulnerability was fixed and damage was mitigated, with there being no material harm to the protocol and its end-users. All Polygon contracts and node implementations remain fully open source.

“All projects that achieve any measure of success sooner or later find themselves in this situation,” said Polygon’s co-founder Jaynti Kanani. “What’s important is that this was a test of our network’s resilience as well as our ability to act decisively under pressure. Considering how much was at stake, I believe our team has made the best decisions possible given the circumstances.”

There is a natural tension between security and transparency, both of which are cornerstone values at Polygon. Our initial disclosure was minimal because we follow the “silent patches” policy introduced and used by the Geth team. All in all, the core development team struck the best possible balance between openness and doing what is best for the community, partners and the broader ecosystem in handling this extremely urgent and sensitive issue. But you can be the judge of that.

Here is the timeline of the events as they unfolded:

Dec. 3, 10:11 (UTC) The first white hat hacker submits a report of a possible exploit to Immunefi, which hosts Polygon’s $2 million bounty program.Dec. 3, 16:18 (UTC)Polygon confirms the vulnerability. Within one hour, various options are considered. The decision is made to upgrade the mainnet as soon as possible.Dec. 3, 20:18 (UTC)The Polygon team provides release Bor v0.2.12-beta1 to validators on Mumbai testnet at Block #22244000. Dec. 4, 04:26 (UTC)Mumbai update is complete. The Polygon team, white hat and Immunefi validate the fix and prepare for the update of the mainnet.Dec. 4, 13:46 (UTC)The vulnerability is used to steal MATIC tokens, the first in a series of transfers that removes 801,601 MATIC in total.Dec. 4, 18:53 (UTC)The second white hat submits a report to Immunefi.Dec. 4, 21:08 (UTC)The Polygon team informs Validators of an “Emergency Bor Upgrade for Mainnet.”Dec. 5, 07:27 (UTC)Mainnet update is complete for +90% validators at Block #22156660.

Polygon paid a total of about $3.46 million as bounty to two white hats who helped discover the bug. Despite our best efforts, a malicious hacker was able to use the exploit to steal 801,601 MATIC before the network upgrade took effect. The foundation will bear the cost of the theft.

“The Polygon team’s response to this disclosure was swift and effective,” said Immunefi's Chief Technology Officer Duncan Townsend. “That this incident had a happy ending is a testament to their expertise. Tight coordination with the Polygon validators helped avert what could’ve been a major disaster.”

In the days after the upgrade, Polygon’s core team has carried out an extensive post mortem and identified a number of existing processes that can be improved as well as actions that will make the network and our community more resilient in the future. These measures include the following:

Updating our critical response processes;
Consolidating partner contact info and communications channels;
Identifying and formalizing backups for key internal resources to eliminate single points of failure during time sensitive situations

This experience highlighted the importance of investing into an ecosystem of security expert partners. We are very grateful to Immunefi for all their help. At the end of the day, this brought Polygon a step closer to becoming the most battle-tested scaling solution for Ethereum.

You can read Immunefi’s technical report on the fix here. A security audit certificate completed by Quantstamp can be viewed here. To keep up with the latest developments, check out our blog.

Let’s bring the world to Ethereum!