Earlier this month, Polygon’s core development team with help from bug bounty platform Immunefi successfully fixed a critical network vulnerability. Considering the nature of this upgrade, it had to be executed without attracting too much attention. We are now ready to give the full account of what happened.
A group of whitehat hackers notified Immunefi, which hosts our bug bounty, of a vulnerability in the Polygon PoS genesis contract on Dec. 3. The Polygon core team engaged with the group and Immunefi’s expert team and immediately introduced a fix. The validator and full node communities were notified, and they rallied behind the core devs to upgrade 80% of the network within 24 hours without stoppage.
The upgrade was executed on Dec. 5 at block #22156660 without impacting liveness and performance of the network in any major way. The vulnerability was fixed and damage was mitigated, with there being no material harm to the protocol and its end-users. All Polygon contracts and node implementations remain fully open source.
“All projects that achieve any measure of success sooner or later find themselves in this situation,” said Polygon’s co-founder Jaynti Kanani. “What’s important is that this was a test of our network’s resilience as well as our ability to act decisively under pressure. Considering how much was at stake, I believe our team has made the best decisions possible given the circumstances.”
There is a natural tension between security and transparency, both of which are cornerstone values at Polygon. Our initial disclosure was minimal because we follow the “silent patches” policy introduced and used by the Geth team. All in all, the core development team struck the best possible balance between openness and doing what is best for the community, partners and the broader ecosystem in handling this extremely urgent and sensitive issue. But you can be the judge of that.
Here is the timeline of the events as they unfolded:
|Dec. 3, 10:11 (UTC) The first white hat hacker submits a report of a possible exploit to Immunefi, which hosts Polygon’s $2 million bounty program.|
|Dec. 3, 16:18 (UTC)Polygon confirms the vulnerability. Within one hour, various options are considered. The decision is made to upgrade the mainnet as soon as possible.|
|Dec. 3, 20:18 (UTC)The Polygon team provides release Bor v0.2.12-beta1 to validators on Mumbai testnet at Block #22244000.|
|Dec. 4, 04:26 (UTC)Mumbai update is complete. The Polygon team, white hat and Immunefi validate the fix and prepare for the update of the mainnet.|
|Dec. 4, 13:46 (UTC)The vulnerability is used to steal MATIC tokens, the first in a series of transfers that removes 801,601 MATIC in total.|
|Dec. 4, 18:53 (UTC)The second white hat submits a report to Immunefi.|
|Dec. 4, 21:08 (UTC)The Polygon team informs Validators of an “Emergency Bor Upgrade for Mainnet.”|
|Dec. 5, 07:27 (UTC)Mainnet update is complete for +90% validators at Block #22156660.|
Polygon paid a total of about $3.46 million as bounty to two white hats who helped discover the bug. Despite our best efforts, a malicious hacker was able to use the exploit to steal 801,601 MATIC before the network upgrade took effect. The foundation will bear the cost of the theft.
“The Polygon team’s response to this disclosure was swift and effective,” said Immunefi's Chief Technology Officer Duncan Townsend. “That this incident had a happy ending is a testament to their expertise. Tight coordination with the Polygon validators helped avert what could’ve been a major disaster.”
In the days after the upgrade, Polygon’s core team has carried out an extensive post mortem and identified a number of existing processes that can be improved as well as actions that will make the network and our community more resilient in the future. These measures include the following:
This experience highlighted the importance of investing into an ecosystem of security expert partners. We are very grateful to Immunefi for all their help. At the end of the day, this brought Polygon a step closer to becoming the most battle-tested scaling solution for Ethereum.
Let’s bring the world to Ethereum!
Meroku—world's first decentralized app store protocol—and Polygon Labs are proud to announce the launch of dApp Store Kit, an open-source software development tech stack that enables anyone to build and launch their own EVM-compatible dApp store where they can set their own rules for distribution, curation, governance, branding, and pricing, and simultaneously upload and distribute...
For developers, creators, and brands building in Web3, identifying the most promising networks requires accurate analytics. Many lenses used to analyze chain activity are incomplete, inaccurate, and may be misleading. Comparing blockchain protocols can be difficult because different chains have wildly different infrastructures. Metrics that go by the same name across two different ecosystems may...
The final Polygon zkEVM testnet, which launched last year, has seen a surge of developers and users eager to experience the future of Ethereum scaling. The years-long technical achievements to get to this point have been enormous. Researchers working on zero-knowledge (ZK) to scale Ethereum have had to answer and address a bevy of design...