Launching 0xFast Stream: 100x Faster Ethereum Block Downloads

Update (Oct 7): 0xFast Stream now also supports Polygon Mainnet. Easy to click links:

Motivation

As I’ve ventured into the web3 space, I’ve been astounded by how unintuitive getting your hands on web3 data is. The first thing I wanted to do was to download the blockchain data and play with it. Seems simple enough, right? Not quite.

While there are several web3 API services available, they can be excruciatingly slow. To test this out, I wrote a simple Go script to fetch an Ethereum block, all its transactions, its receipts, and logs. And I ran it over these services to see what kind of throughput and latency I could get.

The numbers were surprisingly bad. Here’s the code, and these are the results:

Download Alchemy Cloudflare Infura Quicknode
Blocks per Second (higher is better) 0.17 1 4.75 0.78
75p Latency in Seconds (lower is better) 60s 120s 25s 13s

Ethereum mainnet has 15M+ blocks. At these speeds, it would take forever to download blockchain data. Despite a wide array of services, this approach is a no-go.

After this, I ran Geth node locally, used its JSON-RPC interface, then GraphQL interface, and the ethereum exporter — even those didn’t provide a good download speed. Ultimately, I gave up on all API methods and wrote a script to parse Geth’s underlying storage format. I then put together the various pieces needed to form an entire block worth of data. This allowed me to download 15.3M Ethereum blocks — within 12 hours.

To say that the path leading up to this was time-consuming and cumbersome would be an understatement. Being able to query or download entire block data is fundamental to building any indexing or analytics system. Having a way to quickly download any block, a bunch of blocks, or even the entire blockchain should be easy, but it’s not. That changes today.

Introducing 0xFast Stream

Over the last few days, I’ve put together a blazingly fast, high throughput system that can take a block number or a block range and give you everything there is about those blocks — the metadata, the transactions, the receipts, and the logs in JSON format. Behind the scenes, this system holds all the blocks — equivalent of a full node.

Starting today, you can access this system with a simple HTTP GET.

# To download one block -- replace 1 with any block number in decimal or hex.
curl https://stream.0xfast.com/beta/eth-mainnet/1
curl https://stream.0xfast.com/beta/eth-mainnet/0x1

# To download a series of blocks. Decimal or hex separated by a dash.
curl https://stream.0xfast.com/beta/eth-mainnet/1-10
curl https://stream.0xfast.com/beta/eth-mainnet/0x1-0xa

# To download the latest block.
curl https://stream.0xfast.com/beta/eth-mainnet/latest

# To download everything from 1 to the latest
curl https://stream.0xfast.com/beta/eth-mainnet/1-latest
curl https://stream.0xfast.com/beta/eth-mainnet/0x1-latest

# To get gzipped output, add Accept-Encoding: gzip header.
curl -H "Accept-Encoding: gzip" https://stream.0xfast.com/beta/eth-mainnet/latest -s | wc -c
214969

# Use --compressed flag to automatically decompress.
curl --compressed -H "Accept-Encoding: gzip" https://stream.0xfast.com/beta/eth-mainnet/latest -s | wc -c
877870

curl -H "Accept-Encoding: notgzip" https://stream.0xfast.com/beta/eth-mainnet/latest -s | wc -c
877870

At the Speed of Optical Fiber

Comparing this streaming service to the web3 API providers won’t exactly be an apples-to-apples comparison — downloading full blocks isn’t what they are optimizing for. But still evident of the kind of performance you can expect from 0xFast Stream. These are the results:

Download Alchemy Cloudflare Infura Quicknode 0xFast Stream
Blocks per Second (higher is better) 0.17 1 4.75 0.78 220
75p Latency in Seconds (lower is better) 60s 120s 25s 13s 1s

At 220 blocks/sec, the 0xFast Stream is 100x faster than some of these services. The numbers are even better if you download a range of blocks. To test the speed of the system, I ran a download of all the blocks for over a minute via localhost and a Digital Ocean droplet.

# localhost -- for theoretical maximum speed
$ curl localhost:1248/block/1-latest -s | pv -trab > /dev/null
67.4GiB 0:01:00 [1.60GiB/s] [1.12GiB/s]

# Digital Ocean droplet
$ curl https://stream.0xfast.com/beta/eth-mainnet/1-latest -s | pv -trab > /dev/null
8.01GiB 0:01:00 [ 139MiB/s] [ 136MiB/s]

The droplet was able to download data at 136 MiBs ~ 1 Gbps, which is the max network throughput provided by Digital Ocean. Via localhost, I could download data at 1.12 GiBps ~ 9 Gbps, with minimal CPU usage behind the scenes.

And this is the fun part about the 0xFast Stream. The download speed is only limited by the network speed. Now that’s fast!

What’s 0xFast?

0xFast, pronounced exa-fast is a new web3 API service that I’m working on — to provide 100x faster and 3x cheaper web3 APIs compared to what’s available today.

This streaming service is different from Outserv, which is a distributed indexing system. 0xFast Stream is focused on streaming data. I plan to add indices within the streaming system as well to make it easy to find blocks, transactions by hash, corresponding to a smart contract, or an account, etc.

Thus, 0xFast stream is an indexed streaming service.

Conclusion

https://stream.0xfast.com makes it trivial to download Ethereum blocks as JSON, providing a download speed that is 100x faster than available services. The service is currently in beta and free to access. It’s serving the first 15.3M Ethereum mainnet blocks. Over the next few weeks, I plan to:

  1. Ensure that latest Ethereum blocks are available to stream as soon as they get produced, and
  2. Extend the system to include other chains.

The Stream service does not do any searches or run any Ethereum APIs. It’s aim is very simple — allow developers to download blocks efficiently.

Try it out, and join Discord to let me know your thoughts and what it would enable you to build.



Date
September 22, 2022