Launching 0xFast Stream: 100x Faster Ethereum Block Downloads
Motivation
As I’ve ventured into the web3 space, I’ve been astounded by how unintuitive getting your hands on web3 data is. The first thing I wanted to do was to download the blockchain data and play with it. Seems simple enough, right? Not quite.
While there are several web3 API services available, they can be excruciatingly slow. To test this out, I wrote a simple Go script to fetch an Ethereum block, all its transactions, its receipts, and logs. And I ran it over these services to see what kind of throughput and latency I could get.
The numbers were surprisingly bad. Here’s the code, and these are the results:
| Download | Alchemy | Cloudflare | Infura | Quicknode |
|---|---|---|---|---|
| Blocks per Second (higher is better) | 0.17 | 1 | 4.75 | 0.78 |
| 75p Latency in Seconds (lower is better) | 60s | 120s | 25s | 13s |
Ethereum mainnet has 15M+ blocks. At these speeds, it would take forever to download blockchain data. Despite a wide array of services, this approach is a no-go.
After this, I ran Geth node locally, used its JSON-RPC interface, then GraphQL interface, and the ethereum exporter — even those didn’t provide a good download speed. Ultimately, I gave up on all API methods and wrote a script to parse Geth’s underlying storage format. I then put together the various pieces needed to form an entire block worth of data. This allowed me to download 15.3M Ethereum blocks — within 12 hours.
To say that the path leading up to this was time-consuming and cumbersome would be an understatement. Being able to query or download entire block data is fundamental to building any indexing or analytics system. Having a way to quickly download any block, a bunch of blocks, or even the entire blockchain should be easy, but it’s not. That changes today.
Introducing 0xFast Stream
Over the last few days, I’ve put together a blazingly fast, high throughput system that can take a block number or a block range and give you everything there is about those blocks — the metadata, the transactions, the receipts, and the logs in JSON format. Behind the scenes, this system holds all the blocks — equivalent of a full node.
Starting today, you can access this system with a simple HTTP GET.
# To download one block -- replace 1 with any block number in decimal or hex.
curl "https://eth-uswest.0xfast.com/stream/free?range=1"
curl "https://eth-uswest.0xfast.com/stream/free?range=0x1"
# To download a series of blocks. Decimal or hex separated by a dash.
curl "https://eth-uswest.0xfast.com/stream/free?range=1-10"
curl "https://eth-uswest.0xfast.com/stream/free?range=0x1-0xa"
# To download the latest block.
curl "https://eth-uswest.0xfast.com/stream/free?range=latest"
# To download everything from 1 to the latest
curl "https://eth-uswest.0xfast.com/stream/free?range=1-latest"
curl "https://eth-uswest.0xfast.com/stream/free?range=0x1-latest"
# To get gzipped output, add Accept-Encoding: gzip header.
curl -H "Accept-Encoding: gzip" "https://eth-uswest.0xfast.com/stream/free?range=latest" -s | wc -c
214969
# Use --compressed flag to automatically decompress.
curl --compressed -H "Accept-Encoding: gzip" "https://eth-uswest.0xfast.com/stream/free?range=latest" -s | wc -c
877870
curl -H "Accept-Encoding: notgzip" "https://eth-uswest.0xfast.com/stream/free?range=latest" -s | wc -c
877870At the Speed of Optical Fiber
Comparing this streaming service to the web3 API providers won’t exactly be an apples-to-apples comparison — downloading full blocks isn’t what they are optimizing for. But still evident of the kind of performance you can expect from 0xFast Stream. These are the results:
| Download | Alchemy | Cloudflare | Infura | Quicknode | 0xFast Stream |
|---|---|---|---|---|---|
| Blocks per Second (higher is better) | 0.17 | 1 | 4.75 | 0.78 | 220 |
| 75p Latency in Seconds (lower is better) | 60s | 120s | 25s | 13s | 1s |
At 220 blocks/sec, the 0xFast Stream is 100x faster than some of these services. The numbers are even better if you download a range of blocks. To test the speed of the system, I ran a download of all the blocks for over a minute via localhost and a Digital Ocean droplet.
# localhost -- for theoretical maximum speed
$ curl localhost:1248/block/1-latest -s | pv -trab > /dev/null
67.4GiB 0:01:00 [1.60GiB/s] [1.12GiB/s]
# Digital Ocean droplet
$ curl "https://eth-uswest.0xfast.com/stream/free?range=1-latest" -s | pv -trab > /dev/null
8.01GiB 0:01:00 [ 139MiB/s] [ 136MiB/s]The droplet was able to download data at 136 MiBs ~ 1 Gbps, which is the max network throughput provided by Digital Ocean. Via localhost, I could download data at 1.12 GiBps ~ 9 Gbps, with minimal CPU usage behind the scenes.
And this is the fun part about the 0xFast Stream. The download speed is only limited by the network speed. Now that’s fast!
What’s 0xFast?
0xFast, pronounced exa-fast is a new web3 API service that I’m working on — to provide 100x faster and 3x cheaper web3 APIs compared to what’s available today.
This streaming service is different from Outserv, which is a distributed indexing system. 0xFast Stream is focused on streaming data. I plan to add indices within the streaming system as well to make it easy to find blocks, transactions by hash, corresponding to a smart contract, or an account, etc.
Thus, 0xFast stream is an indexed streaming service.
Conclusion
https://stream.0xfast.com makes it trivial to download Ethereum blocks as JSON, providing a download speed that is 100x faster than available services. The service is currently in beta and free to access. It’s serving the first 15.3M Ethereum mainnet blocks. Over the next few weeks, I plan to:
- Ensure that latest Ethereum blocks are available to stream as soon as they get produced, and
- Extend the system to include other chains.
The Stream service does not do any searches or run any Ethereum APIs. It’s aim is very simple — allow developers to download blocks efficiently.
Try it out, and join Discord to let me know your thoughts and what it would enable you to build.