Overlay Network Data Synchronization

Ty Everett ([email protected])

Abstract

This document proposes a solution for synchronizing data across UTXO-based overlay networks by specifying a mechanism for submitting transactions and receiving topic-specific admittance instructions. In current interoperable implementations, overlay services accept BEEF-formatted transaction submissions together with a set of topic names and return a per-topic result describing newly admitted outputs and retained historical inputs. This standard defines the parameters and processing steps needed to track topical UTXOs.

Motivation

As discussed in BRC-59, UTXO-based overlay networks provide a secure and scalable solution to managing states derived from the Bitcoin network. In order to maintain a consistent and up-to-date state across nodes, a method of data synchronization is required. Additionally, it is essential to provide a standard way for network participants to submit new transactions and notify network operators of their relevance to hosted topics. This document aims to address these needs by outlining a clear and well-defined process for submitting transactions and processing them based on topic-specific rules.

Specification

We start with a server that is set up to track the state of one or more topics. Every individual server implementing this specification decides what specific topic labels mean, and which rules they apply when admitting new UTXOs.

Status Note

The earlier JSON-and-BRC-8-envelope framing described by this document is now historical. Current interoperable implementations in bsv-blockchain/ts-sdk and services built around it use:

• a POST /submit endpoint,

• an application/octet-stream request body containing a serialized BEEF transaction,

• an X-Topics header carrying a JSON array of topic names,

• and a structured JSON acknowledgment describing topical admittance instructions.

Implementers SHOULD treat the older JSON envelope framing as deprecated unless they are explicitly maintaining backwards compatibility with a legacy deployment.

API Endpoint and Parameters

The overlay service hosts a POST /submit API endpoint.

Current interoperable request framing is:

• Body: BEEF for the submitted transaction.

• Content-Type: application/octet-stream

• Header: X-Topics, containing a JSON array of topic names.

• Optional Header: x-includes-off-chain-values: true when the request body appends off-chain values after the BEEF payload.

When x-includes-off-chain-values: true is present, current SDK facilitators frame the body as:

VarInt(beefByteLength) || beefBytes || offChainValueBytes

Without that header, the body is just the raw BEEF bytes.

Topic names used with current SDK tooling are expected to begin with tm_.

Overlay Network Node Processing Steps

Upon receiving a transaction, the overlay network node performs the following steps:

1. Parse and validate the submitted BEEF transaction.

2. Check whether the node hosts at least one of the requested topics. If none are hosted, the node returns an early response indicating that no outputs were admitted.

3. Verify the transaction according to the node's policy and validation stack. This commonly includes SPV and/or policy checks, but the exact validation pipeline is implementation-defined.

4. Apply topic-specific logic to the transaction by iterating through all specified topics and performing the following for each topic:

   • Check the transaction's inputs for UTXOs that are already part of the current topical overlay.

   • Apply the topic-specific management logic and protocol rules to determine:

     • which outputs are newly admitted into the topic,

     • which consumed topical coins should be retained for historical traversal,

     • and which topical coins are removed from the live set.

   • Update the node's tracking system, associating admitted outputs with the topic and marking removed coins as spent. If implementing BRC-24, notify lookup services about the new incoming and outgoing entries.

5. Return a JSON response object containing, for each topic, an Admittance Instructions object:

   • outputsToAdmit: output indices newly admitted into the topic.

   • coinsToRetain: input indices whose spent topical predecessors should remain queryable for history.

   • coinsRemoved (optional): input indices whose spent topical predecessors are removed from the live topical set.

6. Pursuant to any peering or synchronization arrangements the operator maintains, notify peer nodes about the transaction.

Authentication, payment, or additional transport requirements MAY be layered on top of this baseline by deployment-specific facilitators, but they are not part of the current minimum interoperable SHIP submission behavior.

Example

We provide an example of an HTTP request and response.

Request

POST /submit HTTP/1.1
Host: example-overlay-node.com
Content-Type: application/octet-stream
X-Topics: ["tm_example_1","tm_example_2"]

<binary BEEF bytes>

Response

HTTP/1.1 200 OK
Content-Type: application/json

{
  "tm_example_1": {
    "outputsToAdmit": [0, 2],
    "coinsToRetain": [1]
  },
  "tm_example_2": {
    "outputsToAdmit": [1, 2],
    "coinsToRetain": [],
    "coinsRemoved": [0]
  }
}

In this example, a client submits a BEEF transaction to the /submit endpoint together with two topics. The node determines which outputs are newly admitted and which consumed topical coins should be retained or removed, then returns topic-keyed admittance instructions.

Implementation

To implement this data synchronization solution, developers should expose a POST /submit endpoint that accepts BEEF transaction submissions, applies topic logic, and returns per-topic admittance instructions. Services that also support history queries should preserve enough ancestry information to satisfy BRC-64.