Overlay Network Lookup Services

Ty Everett ([e-mail])

Abstract

This document proposes a solution for querying the state of UTXO-based overlay networks by specifying a mechanism for accessing data using lookup services. By utilizing a BRC-31 protected server hosting an API endpoint, overlay network nodes can return relevant UTXOs based on the topics they host and the user's query. This standard defines the parameters and processing steps needed to query for and access topical UTXOs.

Motivation

As introduced in BRC-22, UTXO-based overlay networks provide a secure and scalable solution to managing states derived from the Bitcoin network. In order to enable users to query the state of these overlay networks and retrieve relevant UTXOs, a standardized method for lookup services is required. This document aims to address this need by outlining a clear and well-defined process for submitting queries to overlay network nodes and retrieving the resultant UTXOs.

Specification

We build on the node first described in BRC-22 and hook into the output admittance and spend events that naturally occur as transactions are submitted. When transactions are received, topical logic determines which outputs will be part of which overlay networks, and we specify that the lookup services respond to these events.

API Endpoint and JSON Parameters

The BRC-31 protected server will host a POST /lookup API endpoint, which accepts JSON request payloads. The JSON request body will include a provider, which is a string denoting the chosen provider, and a query field, whose meaning is determined by the specific provider being chosen.

Overlay Network Node Processing Steps

Upon receiving a query, the overlay network node will perform the following steps:

1. Engage in BRC-31 authentication and learn the identity of the person making the request, to the extent required in order to fulfill the request.

2. If consistent with the policy set by the node operator, charge a BRC-41 payment based on the query being performed, providing a mechanism by which the node can monetize its operations.

3. Check that the provider stipulated by the request is supported on this node. If not, return a JSON error response with a status key of error, a code of ERR_LOOKUP_SERVICE_NOT_SUPPORTED, and a description comprising a human-readable error message.

4. Send the query to the provider for processing. The provider will use the current known state of the overlay in conjunction with its data storage and retrieval system to fulfill the query.

5. The provider returns a responsive list of current topical UTXO identifiers (topic labels + TXIDs + output numbers) to the overlay network node, where each identifier is for a UTXO that is currently admitted into a topical overlay.

6. After retrieving the list of UTXOs responsive to the query from the lookup service provider, the overlay network node will hydrate each UTXO with its output script, amount, encompassing transaction, and other information. The format for the UTXOs constructed by this process is defined by BRC-36.

7. The node will then return a JSON response to the consumer, comprising an array of BRC-36-style UTXOs.

Lookup Service Providers

Each lookup service is assigned a provider identifier by the overlay network node so that multiple can be installed simultaneously. For each provider, the overlay network node will send events when new UTXOs are added and when they later become spent. The events will contain the output script, the number of satoshis, the TXID, the output number from the transaction, and the topic identifier for the topic where the output was added or removed.

Each lookup service maintains its own data storage and retrieval mechanism independently. This allows each service to use the most appropriate solution for the specific data being managed. Each service chooses how it will respond to these events, which could involve adding, removing, incrementing, decrementing, replacing, updating, or otherwise mutating data storage and retrieval systems applicable to its specific use-case or indexing methodology. Providers are under no obligation to process events for all topics and can freely drop events they deem irrelevant to their operations. Each service will do this in a way so that it can later provide effective responses to queries.

Payments for Queries

The /lookup route may be behind a BRC-41 paywall to provide a monetization mechanism for overlay network server operators. The rules for this paywall are determined by the specific server being queried and can be based on any aspect of the query being made. All state consumers must be prepared to furnish a BRC-41 payment for lookup requests if enabled by the server, unless the client knows for certain that a payment will not be charged.

This means that consumers should be prepared to handle potential payment requirements when making lookup requests, as the server operator may have implemented a BRC-41 paywall to monetize access to the overlay network's state.

In either case, all lookup provider queries are always protected by BRC-31 authentication, as with BRC-22's /submit route, ensuring there are digitally signed records of all requests and responses between the parties.

Example Requests and Responses

Below are examples of an HTTP request and response for the /lookup route.

Request

POST /lookup HTTP/1.1
Host: example-overlay-node.com
Content-Type: application/json
X-Authrite: 0.1
X-Authrite-Identity-Key: ...
X-Authrite-Signature: ...
X-Authrite-Nonce: ...
X-Authrite-YourNonce: ...
X-Authrite-Certificates: ...

{
  "provider": "example_provider",
  "query": {
    "topic": "example_topic",
    "search": "example_search_criteria"
  }
}

Response

HTTP/1.1 200 OK
Content-Type: application/json
X-Authrite: 0.1
X-Authrite-Identity-Key: ...
X-Authrite-Signature: ...
X-Authrite-Nonce: ...
X-Authrite-YourNonce: ...
X-Authrite-Certificates: ...

[
  {
    "txid":"7001295a287fee8e7ad5de58ed30bd61923977ddc9b7a44830ebb9a68b12dc39",
    "vout":0,"outputScript":"4104ca0a8ce950bf2bc85115bd68818455ae2f187efc96c7527dfad98b69531ae65d13cff3e6f07263dcc64c8ccfd03884983a896b0c5887f2ec5bfd7ad739b76119ac213155485250596e4d4850755135546762334146384a5871774b6b6d5a56793568472231485438347a7272467645594658567a6b5343436f6968706d33437754586a74726b200c613f338ad70e228eaee180f65c34d72a63bdf4c9e167dd5bf70a61539e80a8096164766572746973654468747470733a2f2f73746167696e672d6e616e6f73746f72652e626162626167652e73797374656d732f63646e2f54436f5a366269715743634b574b4d6e5065695854620a31383338383430313032043834303046304402205da55600d8f260d760b37491bc3286b9faacd0d408c187e69119ddc1d98bf3fe02207996fda8d6aba1b7494d7fe8a9b7e5111ea492348f11162bc898b3f7ab2ebf486d6d6d6d",
    "topic":"UHRP",
    "satoshis":500,
    "rawTx":"0100000001e09afc3f6ed8842e3f231aac29d325c0f90137a5441bf861c7a9ee7499fafc9a020000006b483045022100cfef7385daedeb6f54a68c1188bf7d9061774f88eb391e132daea49cfcb883c60220010ba950bbb2952a1d4ace35c6721a254774c6370be75e804334be13544e4f3c412102dc35718aeb818a6075a926e8f68bd6656e5fa007a083d8f5b6193b7b1c34e3f7ffffffff03f401000000000000fd53014104ca0a8ce950bf2bc85115bd68818455ae2f187efc96c7527dfad98b69531ae65d13cff3e6f07263dcc64c8ccfd03884983a896b0c5887f2ec5bfd7ad739b76119ac213155485250596e4d4850755135546762334146384a5871774b6b6d5a56793568472231485438347a7272467645594658567a6b5343436f6968706d33437754586a74726b200c613f338ad70e228eaee180f65c34d72a63bdf4c9e167dd5bf70a61539e80a8096164766572746973654468747470733a2f2f73746167696e672d6e616e6f73746f72652e626162626167652e73797374656d732f63646e2f54436f5a366269715743634b574b4d6e5065695854620a31383338383430313032043834303046304402205da55600d8f260d760b37491bc3286b9faacd0d408c187e69119ddc1d98bf3fe02207996fda8d6aba1b7494d7fe8a9b7e5111ea492348f11162bc898b3f7ab2ebf486d6d6d6dc8000000000000001976a9148055582669432149141abcf887fced6881f7404288ac207c1202000000001976a9146a1a5f00deb78b2ef39e9a80d3e3ea2c97d0710c88ac00000000",
    "proof":null,
    "inputs":"{\"9afcfa9974eea9c761f81b44a53701f9c025d329ac1a233f2e84d86e3ffc9ae0\":{\"proof\":{\"flags\":2,\"index\":3,\"txOrId\":\"9afcfa9974eea9c761f81b44a53701f9c025d329ac1a233f2e84d86e3ffc9ae0\",\"target\":\"c99dfb20991b131142a915b086eb6413d78fb472477c31ef06495c7d712dc4e4\",\"nodes\":[\"5206842a42c144617bee40869bd73b0f65ecf6d19c75e73c7106ee2f5b271308\",\"624c68a9e2c8e83df9e55c0bc4b3a560924d6cb027285e6d6499f6445149e4ac\",\"4c02bfcd6098e40ecebab74783e1bd14d8495bccb02e4b5447057de31ed485d6\"],\"targetType\":\"merkleRoot\"},\"rawTx\":\"010000000166dee27f6d4ca528dbd664c1b6a686f9a3b18336ca396bcead7f2065ec0f07ac020000006b483045022100effd3358e1a602898e2a80b0d45b3a01bb36dfc9148b0766bc6e8185f4cf3c0c02203c3d6c6ff180d395b1300b89f96056d9ec915d6cfbf02c7cff2d6f4bb9611d1941210228ec497f7097a26ea1049ef732931d5abc40335e7693cac56d4534f15ff60aefffffffff03204e0000000000001976a9145b296c72cbec349171445808df953c2edaa3a1d388acc8000000000000001976a914d013936b4177bcbb0d934c6348b0d9ec3cc05dac88ac1e7f1202000000001976a914b281d7f257d79aebb910a5ba2c8e787f7ea635ab88ac00000000\"}}",
    "mapiResponses":"[{\"payload\":\"{\\\"apiVersion\\\":\\\"1.5.0\\\",\\\"timestamp\\\":\\\"2023-04-10T20:55:03.2794204Z\\\",\\\"txid\\\":\\\"7001295a287fee8e7ad5de58ed30bd61923977ddc9b7a44830ebb9a68b12dc39\\\",\\\"returnResult\\\":\\\"success\\\",\\\"resultDescription\\\":\\\"\\\",\\\"minerId\\\":\\\"030d1fe5c1b560efe196ba40540ce9017c20daa9504c4c4cec6184fc702d9f274e\\\",\\\"currentHighestBlockHash\\\":\\\"00000000000004863e21305bc8cee6dd66b80a443416f14ccc5c74895fdefcac\\\",\\\"currentHighestBlockHeight\\\":1546251,\\\"txSecondMempoolExpiry\\\":0,\\\"warnings\\\":[],\\\"failureRetryable\\\":false}\",\"publicKey\":\"030d1fe5c1b560efe196ba40540ce9017c20daa9504c4c4cec6184fc702d9f274e\",\"signature\":\"304402205cd7c1b8f64de17c5824fec87549ecf46d69f122ed96971878562795d432a00702202097e309da4db38876bcaecafdd9f6234fcee999113a5fd54c909383d223a24f\"}]"
  }
]

In this example, a client submits a query to the /lookup API endpoint. The JSON payload contains a provider field specifying the desired lookup service provider and a query field with the specific query parameters.

The overlay network node processes the query and returns an HTTP response containing an array of BRC-36-style UTXOs that match the query's criteria. The response includes the topic, TXID, output index, output script, number of satoshis, and other BRC-8 fields for each UTXO.

Implementation

Developers should extend their BRC-31 protected server to host the POST /lookup API endpoint. For monetization, a BRC-41 paywall may be configured in front of the /lookup endpoint, and may charge for lookups based on the queries being performed or the volume of information requested. Developers should ensure that their implementation adheres to the JSON request and response structures specified in this document.