Compare Ethereum API Performance with Versus

Deciding whether to run your own Ethereum node (or nodes), outsource your infrastructure to a Web3 provider like Infura, or deploy a hybrid node solution really depends on your API workload. To help developers evaluate their options, we’ve built Versus: an open source API benchmarking tool. Versus compares your API workload across different node implementations and Web3 providers so you can make data-based decisions.

Tests run using the Versus tool are 99% reproducible. In this post we step through a sample test to demonstrate how the tool works, and share the test parameters and results so others can follow the same method and reproduce them.

Getting Started

This project lives here. Simply grab the latest release binary for your platform, extract it, put it in your $PATH and you’re ready to roll.

How it Works

Versus is a command-line tool that reads lines of JSON-RPC requests from STDIN and concurrently sends them to multiple endpoints. It validates that the same results are returned, and compares the time to retrieve them, across each endpoint.

For example: let’s say we have a file queries.json with lines of the types of queries that our application performs:

$ cat queries.json

{"jsonrpc":"2.0","id":1,"method":"eth_blockNumber"}

{"jsonrpc":"2.0","id":1,"method":"eth_getBalance","params" ["0x49318b3970C8a82275B440900bE45f27bf682108","latest"]}

{"jsonrpc":"2.0","id":1,"method":"eth_getCode","params":["0x2a0c0dbecc7e4d658f48e01e3fa353f44050c208","latest"]}

We can send those queries to our local node: `cat queries.json | versus "http://127.0.0.1:8545"

The results would be returned super fast. That's because these are fairly simple queries – and there are only three of them. If we want to run a few thousand queries, it’s going to take a bit more time. We can bump up the concurrency of versus with the --concurrency=20 flag. Add --verbose once or twice to get more output as the process is going.

We can also run the same queries against multiple endpoints at the same time:

$ export INFURA_API_KEY=”... grab this from your https://infura.io/dashboard”
$ cat many-queries.json | versus -vv --concurrency=50 \
"http://geth-node:8545" \
	"http://parity-node:8545" \
	"https://mainnet.infura.io/v3/${INFURA_API_KEY}"

Some responses may be mismatched, but that’s to be expected. Geth and Parity don’t always agree on the exact output that certain types of queries should generate. Note that Websocket support is not implemented yet - pull requests are welcome!

Generating Realistic Queries

Not many of us have a juicy many-queries.json sitting around on our hard drive - and even if we do, these queries get stale really fast. The performance of a node on queries that touch recently-mutated state will vary from state that is super old.

To generate realistic queries, we can use Ethspam - another handy open source tool. Ethspam generates an infinite stream of realistic queries, based on the typical pattern of RPC methods that Infura receives. It’s not perfect, but it’s very useful in combination with Versus. After grabbing a binary release, we can pipe it directly:

$ ethspam | versus -vv --concurrency=50 --stop-after=10000 \
"http://geth-node:8545" \
	"http://parity-node:8545" \
	"https://mainnet.infura.io/v3/${INFURA_API_KEY}"

Output

Below is an example of the output you can expect, once you’ve piped your query into Versus:

In this example, we’ve used Infura (Core Tier) as the Web3 provider, and a local “anonymous” node on an i3.xlarge EC2 instance, and ran a test to compare the two.

Results

In this test, the local node was quite limited. Despite many optimizations (high-IO storage, high memory with increased cache size, etc), requests were bottlenecked. By comparison, Infura’s concurrency proved to be unbounded (i.e., there was no limit to how many requests Infura could respond to, per second).
Some requests were very fast to serve from the local node, which resulted in a lower minimum latency time -- however, the average and maximum latency was significantly and consistently lower for requests through Infura (0.1291s standard deviation for Infura vs 3.2035s for the local node).

Key Learnings

1. The mismatched responses presented above were predominantly due to the local node disagreeing with Infura about what the latest block number was. If Versus is run with the --verbose flag, it will print mismatched results. Ethspam included some queries that required an archival node to retrieve. Neither the local node nor the Infura Core plan had access to archive-tier data, so these particular results matched up;
2. Ethspam input is random, which means that if you’re using Versus to compare separate runs, your results won’t be consistent. When comparing nodes, make a single Versus run against multiple endpoints. A single run will confirm that the results match. Only make multiple Versus runs against each single endpoint if you’re using the exact same input;
3. When running Versus against a single node, make sure that node is fully synced and functional. Nodes like geth will happily respond with an empty non-error response, even when it’s not synchronized, without doing any actual work. Running simultaneously against a known functioning node helps ensure that results are not mismatched.
4. When running with high concurrency, make sure there are enough samples to reach consistent performance. At minimum, we suggest at least x100 requests per unit of concurrency. Higher is better!

Get Testing!

Though Versus was built by Infura, it’s intended to be an impartial and open source project. We encourage ideas and pull requests from the Ethereum community that seek to make Versus a better, more objective tool that anyone can run.

You can start playing around with Versus now to compare different node implementations and Web3 providers. Compare a light node to a full node, compare different kinds of nodes against each other, or compare Infura against other Web3 providers in high load/concurrency situations.

Our hope is that developers will find Versus a useful tool to help them evaluate what’s out there and ultimately find the best solution to power their DApp.

Want more Web3 tools? Check out this post for open source tools that help you interact more seamlessly with IPFS. Ready to test out Infura? Sign up to start building for free!