Overlay Network Data Synchronization
Last updated
Was this helpful?
Last updated
Was this helpful?
Ty Everett (ty@projectbabbage.com)
This document proposes a solution for synchronizing data across UTXO-based overlay networks by specifying a secure and efficient mechanism for submitting and processing transactions. By utilizing a protected server hosting an API endpoint, overlay network nodes can process and admit transactions based on their relevance to the hosted topics. This standard defines the parameters and processing steps needed to track topical UTXOs.
As discussed in , UTXO-based overlay networks provide a secure and scalable solution to managing states derived from the Bitcoin network. In order to maintain a consistent and up-to-date state across nodes, a method of data synchronization is required. Additionally, it is essential to provide a standard way for network participants to submit new transactions and notify network operators of their relevance to hosted topics. This document aims to address these needs by outlining a clear and well-defined process for submitting transactions and processing them based on topic-specific rules.
We start with a server that is set up to track the state of one or more topics. Every individual server implementing this specification decides what specific topic labels mean, and which rules they want to apply when admitting new UTXOs.
The protected server will host a POST /submit API endpoint, which accepts JSON request payloads. The JSON request body will include a transaction envelope (rawTx, inputs, mapiResponses, proof) and an additional field called topics. The transaction envelope facilitates SPV checking, while the topics field denotes which overlay network topics should be notified about the transaction.
Upon receiving a transaction, the overlay network node will perform the following steps:
Verify the identity of the sender and establish the required level of identity for submitting transactions, negotiating under the protocol to the satisfaction of the network operator and the transaction sender.
Check whether the node hosts at least one of the tagged topics from the payload. If none are hosted, the node returns an early response indicating no outputs were admitted. Otherwise, it can drop any unhosted topics from the list before proceeding.
Use the process to verify the submitted envelope and confirm the transaction's legitimacy.
Apply topic-specific logic to the transaction by iterating through all specified topics and performing the following for each topic:
Check the transaction's list of inputs for UTXOs that are part of the current topical overlay. Flag these inputs and communicate the information to the topic's management logic.
Apply the topic-specific management logic and protocol rules to determine output admittance for each transaction output.
Update the node's tracking system, associating each relevant output with the topic labels of topics that have admitted those outpoints. If implementing , notify lookup services about the new incoming entries.
Remove any now-spent inputs that were previously associated with a topic from the tracking system, as they have now become spent by this transaction. If implementing , notify lookup services about the old outgoing entries.
Generate and return a JSON response object containing the transaction processing results, including a status key with a value of success, and a topics key with an object containing topic labels and arrays of output numbers denoting which outputs were admitted into the specific topics.
Pursuant to any peering or data synchronization arrangements or contracts that the node operator may have negotiated with other node operators with whom they would like to remain in sync, use the agreed-upon exchange mechanisms to notify the other node operators about the received transaction, potentially in conjunction with an incoming or outgoing payment.
We provide an example of an HTTP request and response.
The overlay network node processes the transaction and determines that outputs 0 and 2 are admissible for "example_topic_1" and outputs 1 and 2 are admissible for "example_topic_2". The node then returns an HTTP response with a status of "success" and a topics object containing the admitted outputs for each topic.
Developers should ensure that their implementation adheres to the JSON request and response structures specified in this document, enabling seamless interaction between overlay network participants.
In this example, a client submits a transaction to the /submit API endpoint. The JSON payload contains the transaction envelope (rawTx, inputs, mapiResponses, proof) and an additional field called topics, which includes two example topics: "example_topic_1" and "example_topic_2".
To implement this data synchronization solution, developers should first set up a protected server to host the POST /submit API endpoint. The server should be capable of processing JSON request payloads and performing the necessary transaction validation using .
Overlay network nodes must be programmed to process incoming transaction submissions following the steps outlined in the specification. This includes verifying the sender's identity, checking for hosted topics, validating the transaction using , and applying topic-specific logic to determine output admittance.