By Mark Blackman & Mica Cerone

The upcoming Hedera network upgrade introduces early access to HIP-1056: Block Streams, a new blockchain format for recording the Hedera network's activities. This innovation marks a big step forward with all transaction output and state change data being exposed in a single, cryptographically verifiable stream of data.

Block Streams will consolidate current output streams into a single, unified format. Moreover, the inclusion of Hedera state data redefines how developers and services interact with the network, enabling richer insights, improved efficiency, and streamlined integrations.

TL;DR

Block Streams are a new output stream that merges Hedera’s existing event, record, and sidecar streams into a single stream of verifiable data. This consolidated approach primarily simplifies data consumption; however, with the inclusion of state data, it also unlocks new capabilities such as state proofs and interledger communication.

Each block within this stream is a self-contained, verifiable entity that encompasses:

• All events and transactions within a configurable number of consensus rounds.
• All resulting transaction output and state change data.
• A single network signature that attests to the validity of the block by a majority of Hedera mainnet nodes.

Benefits of Blockstreams

1. Unified Data Stream
   Block Streams consolidate multiple data types into a single, cohesive stream, simplifying data consumption and processing for developers and services.

2. Enhanced Verifiability
   Each block includes a BLS network signature, proving it was signed by a majority of nodes representing a majority of the network's consensus stake weight. This feature simplifies the verification of the data, reducing mirror node processing and verification costs.

3. Simplified Consumption
   Block Streams are encoded using Protobuf, enabling them to be easily consumable by any programming language with minimal complexity or dependencies, which increases accessibility for developers to work with Hedera data.

Unlocking Potential with State Changes

Adding Hedera state change data unlocks significant advantages by enabling off-network computation and verification of state. This capability empowers developers and services to achieve high performance, enhanced efficiency, and greater flexibility.

Some examples include:

• Applications and off-chain services can cache and process Hedera state locally, reducing latency and dependency on external data retrieval.
• Applications and off-chain services can monitor and maintain contract state locally, enabling seamless interaction with Hedera-based smart contracts.

Technical Insights into Blockstreams

Block Items
A Block Stream is a continuous sequence of data elements, known as Block Items, streamed from Hedera consensus nodes to provide a verifiable record of Hedera consensus and transaction processing.

The sequence of Block Items is crucial for interpreting the Block Stream, as it not only defines the boundaries of individual blocks but also establishes the relationship between Block Items within a block.

This ordered structure determines how blocks are extracted from the Block Stream, how transactions are linked to specific events, how events correlate with rounds, and how these components collectively define a block.

The graphic below illustrates how the sequential ordering of Block Items represents the structure of a block and the data relationships connecting blocks, rounds, events, and transactions.

Block Hash

Within each block, the Block Items are organized into a Merkle-based tree structure, culminating in a root hash, known as the Block Hash. This structure ensures that each block’s data is consistent and accurately represented.

This Block Hash is signed by a majority of consensus nodes using an aggregated signature, referred to as the Block Proof. The Block Proof provides cryptographic assurance that it has been processed, attested, and signed by a majority of Hedera mainnet nodes.

Block Item Proof

The Block Hash, Block Proof, and Merkle tree structure of a block open up new opportunities for leveraging Hedera's output data. Among these advancements are Block Item Proofs, which are cryptographic proofs for individual block items that allow verification of specific data within a block.

These proofs empower users to independently verify all of Hedera’s transactional history at minimal cost, even for offline devices. This capability not only enhances trust and transparency but also significantly broadens the range of use cases for Hedera, from lightweight applications to secure offline verifications.

Blockchain Structure

Hedera functions as a modular blockchain, achieving consensus through the Hedera Hashgraph while outputting data into a blockchain structure. This architecture ensures the security and immutability of the network's historical records.

Block Streams build on this foundation by incorporating the hash of each preceding block into the next block, forming an immutable chain of data derived from Hedera.

The diagram below illustrates the Merkle tree structure within a block, showcasing how the previous block's hash is seamlessly integrated into the current one. This structure not only ensures data integrity but also enhances the transparency and auditability of the network.

Transition to Block Streams

When transitioning from the existing Hedera Record Stream to Block Streams, the hash of the final record file will be included as input to the first block. This ensures cryptographic continuity between the existing record stream and the new Block Stream format.

Impacts on Mirror Nodes

The Mirror Node software ensures a smooth transition from records to Block Streams and Block Nodes, offering an intuitive and straightforward upgrade process for operators. Additionally, the introduction of Block Streams brings significant features and benefits to Mirror Node operators and consumers, including the following:

1. Aggregated Signature – Less cost, less latency
   Currently, each node generates a v6 record stream and a signature file for verification. Mirror Nodes must download signature files from a strong minority (1/3 of consensus stake) along with the record file, which makes frequent cloud storage access expensive. Blocks now include a single aggregated signature to cut costs and simplify the process.

2. File Naming and Retrieval – Less cost
   Record stream files are named after the first transaction's consensus timestamp, forcing mirror nodes to perform costly LIST operations on cloud storage to find new record files. The new Block Streams use sequential block numbers, removing the need for these operations.

3. Sidecar Inclusion – Lower complexity and processing
   In record streams, sidecar files store extra data—like smart contract logs—separately from the main record stream. Block Streams integrate this sidecar data into a unified stream, simplifying data management for Mirror Node operators.

4. Reduced Egress
   Currently, records streams are stored in cloud buckets, incurring egress charges for requesters. With the introduction of Block Nodes, Mirror Node operators have the flexibility to select their preferred Block Node, offering greater control over geolocation and reducing egress costs.

Final Thoughts

Block Streams mark a significant step in Hedera's evolution, providing a powerful new way to leverage network data for verification, transparency, and innovation.

As we roll out this feature, we encourage your feedback and collaboration to explore the full potential of Block Streams to transform how we interact with and verify blockchain data. Together, we’re paving the way for a more connected, transparent, and efficient decentralized future.

Starting with release 0.56, developers can access Block Streams, which will contain full mainnet blocks, including all data, except for the Block Proof, which will be mocked for development purposes.

The blocks will be published to the Google Cloud Service (GCS) utilized by Hedera's record stream, adhering to the following structure:

gs://hedera-mainnet-streams/block-preview/mainnet/0/3/000[..].blk.gz

For more information, please visit:
Hedera Documentation