0xFast Indexed Streams: Search Blockchain at the Speed of Light
I’m excited to announce the launch of Indexed Blockchain Streams at 0xFast. What are indexed streams, you ask? Indexed Streams build an index on top of the Blockchain Stream, to allow searching within and streaming blocks, transactions (txns), and logs.
With indexed streams, you can
- Search a block by hash.
- Search a txn by hash.
- Stream blocks by miner address.
- Stream transactions by from address, to address, or both.
- Stream logs by contract address, topic, or both.
0xFast Stream now enables a developer to quickly filter and stream only the information relevant to their needs. The filters can be mixed and matched to only stream logs corresponding to a contract address, and within a certain block range. The same goes for transactions, miners, etc.
For example, if you are interested only in NFT data, you can stream just the logs corresponding to the NFT smart contracts. You can further paginate this stream by block ranges.
Bits here, Maps there
Behind the scenes, I use 64-bit Serialized Roaring Bitmaps (sroar) to generate bitmaps across the blockchain on relevant fields parsed from blocks. Sroar is a library that my team and I built while at Dgraph Labs. Quoting from the blog post:
Based on the benchmarks, Sroar consumes 15x less memory and does 25x fewer memory allocations. And it achieves this, while also being 7x faster!
When querying for a field, a bitmap is fetched corresponding to that filter. Multiple filters generate multiple bitmaps, and depending upon the context could be intersected or combined to generate the final bitmap.
The final bitmap is then used to identify and fetch the right blocks to search inside and match the relevant fields that the user asked for to stream those out.
Streaming Search
Here are the searches that can be performed, along with working examples.
Stream Blocks
# By block number or 'latest'
$ curl "https://eth-uswest.0xfast.com/stream/free?range=0"
$ curl "https://eth-uswest.0xfast.com/stream/free?range=latest"
# By a range of block numbers
$ curl "https://eth-uswest.0xfast.com/stream/free?range=1-1000"
$ curl "https://eth-uswest.0xfast.com/stream/free?range=0-latest"
# By block hash -- returns one block
$ curl "https://eth-uswest.0xfast.com/stream/free?block.hash=0xfcaeebe3150cf3132a27bcf2657551d69317556c212f79ac191c39ec788388a3" -s
# By miner address -- returns a list of blocks
$ curl "https://eth-uswest.0xfast.com/stream/free?miner.addr=0xdafea492d9c6733ae3d56b7ed1adb60692c98bc5&range=15560000-15570000" -sStream Transactions
# By txn hash -- returns one txn
$ curl "https://eth-uswest.0xfast.com/stream/free?txn.hash=0xf6f57a97551dfc96fbd7f7b5c62ff7a1f3767e229cd5ae0c9f9faa7dc8182fe9" -s
# By from address -- returns a list of txns from that address
$ curl "https://eth-uswest.0xfast.com/stream/free?from.addr=0xa5a13f62ce1113838e0d9b4559b8caf5f76463c0&range=15560000-15570000" -s
# By to address -- returns a list of txns to that address
$ curl "https://eth-uswest.0xfast.com/stream/free?to.addr=0xdac17f958d2ee523a2206206994597c13d831ec7&range=15560000-15561000" -s
# By both from and to address -- returns a list of txns involving both addresses.
$ curl "https://eth-uswest.0xfast.com/stream/free?from.addr=0x77ad3a15b78101883af36ad4a875e17c86ac65d1&to.addr=0x00000000a991c429ee2ec6df19d40fe0c80088b8&range=15560000-15570000" -s
# By address -- list of txns involving that address in either the to or the from field.
$ curl "https://eth-uswest.0xfast.com/stream/free?addr=0x77ad3a15b78101883af36ad4a875e17c86ac65d1&range=15560000-15570000" -sStream Logs
# By log address -- returns a list of logs
$ curl "https://eth-uswest.0xfast.com/stream/free?range=10000000-11000000&log.addr=0x95ad61b0a150d79219dcf64e1e6cc01f0b64c4ce" -s
# By log topic -- returns a list of logs
$ curl "https://eth-uswest.0xfast.com/stream/free?range=10000000-11000000&log.topic=0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef" -s
# By both log address and topic -- returns a list of logs
$ curl "https://eth-uswest.0xfast.com/stream/free?range=10000000-11000000&log.addr=0x95ad61b0a150d79219dcf64e1e6cc01f0b64c4ce&log.topic=0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef" -sKeeping up with the (karda-)Chains
In the 0xFast launch blog post, I mentioned that the ETH stream contains the first 15.3 million blocks. And I plan to:
Ensure that the latest Ethereum blocks are available to stream as soon as they get produced.
This is now happening. Ethereum stream is now keeping up to date with the latest block produced by the chain. And the indexing system indexes these blocks as soon as they get added to the stream. All within a couple of seconds from block generation. In fact, sometimes I’ve noticed this system to be a few blocks ahead of available blockchain API service providers.
Conclusion
Next on 0xFast, I want to enable indexing and syncing Polygon stream with the latest block. And next, create a special cross-chain stream for the most revered data on Blockchain — NFTs.
0xFast Streams are the perfect blocks to build web3 APIs or analytical systems. This is what some of our users said:
0xfast is compelling because it’s simple AF. And bash-fu friendly. It’s just so conducive to getting sh*t done.
Wish 0xfast had launched earlier. It would have saved us 2 months of work on indexing Ethereum.
So, do try it out. And come talk to us on Discord.