Toward Reliable Ethereum Infrastructure

On September 19, in the early morning hours in Shanghai, the Ethereum community was about to wake up in excitement for the annual developer conference, Devcon2. Instead, we woke up to find the Ethereum network under attack. More specifically, an attacker had found a specific issue with the go-ethereum client, allowing specific transactions to be sent causing the go-ethereum client to hang and not be able to process blocks.

The go-ethereum team quickly responded and issued a patch within a few hours, and Devcon2 went on with only a minor delay in start time.

Infura: “For the People”

At the Infura project, we have the mission of providing secure, stable, fault tolerant and scalable Ethereum and IPFS nodes for the Ethereum community. Our experience in Shanghai was also highly impacted by the attack and subsequent activity. We woke up to a red alert that the Ethereum network was under attack. Our nodes were not broadcasting transactions and were falling behind. Further complicating things was the restricted internet access we had in China. We initially spent some time walking the streets of Shanghai looking for a reliable Wi-fi connection to assess the situation. As we were walking, Joseph Lubin, founder of ConsenSys, commented with a grin, “It’s hard standing up an entire ecosystem from scratch!” We eventually landed in the lobby of the Astor Place hotel which is a short walk from the Hyatt on the Bund where the conference was taking place.

As it became clear that the issue at the time was only impacting the go-ethereum client, we quickly made the decision to route all incoming traffic in Infura to our Parity nodes. This allowed us to quickly get service restored and any application that relied on us for their connection into the Ethereum blockchain was back online and able to behave predictably.

A month later, we have seen that that hiccup on the morning of Devcon2 was just the first in a series of attacks that has been ongoing and has impacted both go-ethereum and Parity. The following timeline summarizes the attacks over the last month.

The Infura team has been actively monitoring the health of the Ethereum network and contributing data back into the development teams as the attacks progress. We are actively developing an early warning detection within our “Ferryman” service layer that will help us to quickly identify attacks that hit particular clients and eventually automatically route incoming requests to healthy Ethereum nodes. We will be supporting the protocol hard fork, originally proposed by Gavin Wood and outlined in this EIP to put this particular attack vector behind us. It is our firm belief that these attacks make our network stronger.

It is our goal to provide stable and reliable core infrastructure so that those who choose to use our service can focus on developing and deploying decentralized applications that behave predictably rather than worrying about maintaining their own infrastructure.