[Part 1|3] Introduce SQL Payment schema into LND #9147
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Important Review skippedAuto reviews are limited to specific labels. 🏷️ Labels to auto review (1)
Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
|
|
Some questions I already had:
I think this is ok and we just include an interface for the |
As in a composite foreign key? That should be fine, though if there's a way we can avoid it in practice to simplify queries or instead add a |
|
@ziggie1984 I think the image upload on that last comment failed. |
|
I used Claude to create an mermaid diagram of the schema so far (zero shot, and first pass, might have some errors). This is nice, as we can check it into the repo, and the markdown updates will be shown in the git diff: erDiagram
mpp_payment ||--o{ first_hop_custom_records : has
mpp_payment ||--|| payment_status_types : has
mpp_payment ||--|| mpp_state : has
mpp_payment ||--o{ htlc_attempt : has
htlc_attempt ||--o| htlc_settle_info : has
htlc_attempt ||--o| htlc_fail_info : has
htlc_fail_info ||--|| htlc_fail_reason_types : has
htlc_fail_info ||--o| failure_msg_types : has
htlc_attempt ||--o| route : has
route ||--o{ hop : contains
hop ||--o| blinded_data : has
hop ||--o| mpp_record : has
hop ||--o| amp_record : has
hop ||--o{ hop_custom_records : has
mpp_payment {
integer id PK
integer payment_status FK
blob payment_hash
bigint amount_msat
blob payment_request
timestamp created_at
}
first_hop_custom_records {
bigint key FK
blob value
text payment_id
}
payment_status_types {
integer id PK
text description
}
mpp_state {
integer payment_id FK
integer num_attempts_in_flight
bigint remaining_amt
bigint FeesPaid
boolean has_settled_htlc
boolean payment_failed
}
htlc_attempt {
integer id PK,FK
blob session_key
timestamp attempt_time
integer payment_id FK
}
htlc_settle_info {
integer id PK
integer attempt_id FK
blob preimage
timestamp settle_time
}
htlc_fail_info {
integer id PK
integer attempt_id FK
integer failure_source_index
integer htlc_fail_reason FK
integer failure_msg FK
}
htlc_fail_reason_types {
integer id PK
text description
}
failure_msg_types {
integer id PK
text description
text error_msg
}
route {
integer id PK
integer htlc_attempt_id FK
integer total_timeLock
bigint total_amount
blob source_key
bigint first_hop_amount
}
hop {
integer id PK
integer route_id FK
blob pub_key
text chan_id
integer outgoing_time_lock
bigint amt_to_forward
blob meta_data
}
blinded_data {
integer hop_id FK
blob encrypted_data
blob blinding_point
bigint total_amt
}
mpp_record {
integer hop_id FK
blob payment_addr
bigint total_msat
}
amp_record {
integer hop_id FK
blob root_share
blob set_id
integer child_index
}
hop_custom_records {
bigint key FK
blob value
integer hop_id FK
}
|
Roasbeef
requested changes
Oct 7, 2024
| amount_msat BIGINT NOT NULL, | ||
|
|
||
| -- payment_request is the payment request of the payment. | ||
| payment_request BLOB UNIQUE NOT NULL, |
There was a problem hiding this comment.
I wonder if we should split this out, as in the future we'll have more than one type of payment request encoding. I think it's also possible to actually have duplicates here: someone pays a zero value invoice multiple times.
| -- invoice_event_types defines the different types of invoice events. | ||
| INSERT INTO payment_status_types (id, description) | ||
| VALUES | ||
| -- status_unknown is a deprecated status and describes the unkown payment |
|
|
||
|
|
||
| -- invoice_event_types defines the different types of invoice events. | ||
| INSERT INTO payment_status_types (id, description) |
There was a problem hiding this comment.
Do these status values capture the latest fixes/wisdom w.r.t edge cases we've seen in the wild with payments in "stuck" states?
| CREATE TABLE legacy_payments ( | ||
| payment_id INTEGER PRIMARY KEY REFERENCES payments(id) ON DELETE CASCADE, | ||
|
|
||
| payment_hash BLOB UNIQUE NOT NULL |
There was a problem hiding this comment.
I think there were instances of duplicate payment hashes in the past?
|
|
||
| remaining_amt BIGINT NOT NULL, | ||
|
|
||
| FeesPaid BIGINT NOT NULL, |
|
|
||
| has_settled_htlc BOOLEAN NOT NULL, | ||
|
|
||
| payment_failed BOOLEAN NOT NULL |
There was a problem hiding this comment.
Do these actually need to be NOT NULL, iiuc that'll just force a caller to always specify something. Might have some implications at the sqlc API level as well.
| -- first_hop_custom_records table stores the first hop custom records which are | ||
| -- only send as a wire message to the first hop and are used for overlay | ||
| -- channels. | ||
| CREATE TABLE "first_hop_custom_records" ( |
There was a problem hiding this comment.
WIP, but can drop the quotes from each of the tables below.
|
|
||
| value BLOB NOT NULL, | ||
|
|
||
| payment_id INTEGER REFERENCES payments(id) ON DELETE CASCADE |
There was a problem hiding this comment.
Seems we could further denormalize this with another layer of indirection. So we store a table of TLV KV pairs, then an associative table that relates a payment_id to an entry in the TLV KV table.
| CREATE INDEX IF NOT EXISTS amp_record_set_id_idx ON amp_record(set_id); | ||
|
|
||
| CREATE TABLE "hop_custom_records" ( | ||
| key BIGINT NOT NULL, |
There was a problem hiding this comment.
Yeah so if we go with the suggestion above, then this would become another associative table (dedup the KV pairs, then one table to map a given hop_id to the set of KV pairs).
|
Updated version: erDiagram
payments {
BIGINT id PK
INTEGER payment_status FK
INTEGER payment_type FK
BIGINT amount_msat
BLOB payment_request
TIMESTAMP created_at
}
payment_status_types {
INTEGER id PK
TEXT description
}
payment_types {
INTEGER id PK
TEXT description
}
legacy_payments {
INTEGER payment_id PK,FK
BLOB payment_hash
}
mpp_payments {
INTEGER payment_id PK,FK
INTEGER mpp_record_id FK
}
amp_payments {
INTEGER payment_id PK,FK
INTEGER amp_record_id FK
}
mpp_state {
INTEGER payment_id PK,FK
INTEGER num_attempts_in_flight
BIGINT remaining_amt
BIGINT FeesPaid
BOOLEAN has_settled_htlc
BOOLEAN payment_failed
}
first_hop_custom_records {
BIGINT key
BLOB value
INTEGER payment_id FK
}
htlc_attempt {
INTEGER id PK
BLOB session_key
TIMESTAMP attempt_time
INTEGER payment_id FK
}
route {
INTEGER htlc_attempt_id PK,FK
INTEGER total_timeLock
BIGINT total_amount
BLOB source_key
BIGINT first_hop_amount
}
hop {
INTEGER id PK
INTEGER route_id FK
BLOB pub_key
TEXT chan_id
INTEGER outgoing_time_lock
BIGINT amt_to_forward
BLOB meta_data
}
mpp_record {
INTEGER hop_id PK,FK
BLOB payment_addr
BIGINT total_msat
}
amp_record {
INTEGER hop_id PK,FK
BLOB root_share
BLOB set_id
INTEGER child_index
}
hop_custom_records {
BIGINT key
BLOB value
BIGINT hop_id FK
}
blinded_data {
INTEGER hop_id PK,FK
BLOB encrypted_data
BLOB blinding_point
BIGINT total_amt
}
htlc_settle_info {
INTEGER htlc_attempt_id PK,FK
BLOB preimage
TIMESTAMP settle_time
}
htlc_fail_info {
INTEGER htlc_attempt_id PK,FK
INTEGER htlc_fail_reason FK
TEXT failure_msg
}
htlc_fail_reason_types {
INTEGER id PK
TEXT description
}
payments ||--o{ legacy_payments : has
payments ||--o{ mpp_payments : has
payments ||--o{ amp_payments : has
payments ||--o{ mpp_state : has
payments ||--o{ first_hop_custom_records : has
payments ||--o{ htlc_attempt : has
payments }|--|| payment_status_types : references
payments }|--|| payment_types : references
mpp_payments }|--|| mpp_record : references
amp_payments }|--|| amp_record : references
htlc_attempt ||--o| route : has
route ||--o{ hop : contains
hop ||--o| mpp_record : has
hop ||--o| amp_record : has
hop ||--o{ hop_custom_records : has
hop ||--o| blinded_data : has
htlc_attempt ||--o| htlc_settle_info : has
htlc_attempt ||--o| htlc_fail_info : has
htlc_fail_info }|--|| htlc_fail_reason_types : references
|
|
Updated graph: erDiagram
payments {
BIGINT id PK
INTEGER payment_status FK
INTEGER payment_type FK
BIGINT amount_msat
TIMESTAMP created_at
}
payment_requests {
INTEGER id PK
INTEGER payment_id FK
BLOB payment_request
}
payment_status_types {
INTEGER id PK
TEXT description
}
payment_types {
INTEGER id PK
TEXT description
}
legacy_payments {
INTEGER payment_id PK,FK
BLOB payment_hash
}
mpp_payments {
INTEGER payment_id PK,FK
INTEGER mpp_record_id FK
}
amp_payments {
INTEGER payment_id PK,FK
INTEGER amp_record_id FK
}
payment_state {
INTEGER payment_id PK,FK
INTEGER num_attempts_in_flight
BIGINT remaining_amt
BIGINT fees_paid
BOOLEAN has_settled_htlc
BOOLEAN payment_failed
}
htlc_attempt {
INTEGER id PK
BLOB session_key
TIMESTAMP attempt_time
INTEGER payment_id FK
}
route {
INTEGER htlc_attempt_id PK,FK
INTEGER total_timeLock
BIGINT total_amount
BLOB source_key
BIGINT first_hop_amount
}
hop {
INTEGER id PK
INTEGER route_id FK
BLOB pub_key
TEXT chan_id
INTEGER outgoing_time_lock
BIGINT amt_to_forward
BLOB meta_data
}
mpp_record {
INTEGER hop_id PK,FK
BLOB payment_addr
BIGINT total_msat
}
amp_record {
INTEGER hop_id PK,FK
BLOB root_share
BLOB set_id
INTEGER child_index
}
tlv_records {
INTEGER id PK
BIGINT key
BLOB value
}
custom_records {
INTEGER tlv_record_id PK,FK
INTEGER hop_id FK
}
first_hop_custom_records {
INTEGER tlv_record_id PK,FK
INTEGER payment_id FK
}
blinded_data {
INTEGER hop_id PK,FK
BLOB encrypted_data
BLOB blinding_point
BIGINT total_amt
}
htlc_settle_info {
INTEGER htlc_attempt_id PK,FK
BLOB preimage
TIMESTAMP settle_time
}
htlc_fail_info {
INTEGER htlc_attempt_id PK,FK
INTEGER htlc_fail_reason FK
TEXT failure_msg
}
htlc_fail_reason_types {
INTEGER id PK
TEXT description
}
payments ||--o{ payment_requests : has
payments ||--o| legacy_payments : has
payments ||--o| mpp_payments : has
payments ||--o| amp_payments : has
payments ||--o| payment_state : has
payments ||--o{ htlc_attempt : has
payments }o--|| payment_status_types : has
payments }o--|| payment_types : has
htlc_attempt ||--o| route : has
route ||--o{ hop : has
hop ||--o| mpp_record : has
hop ||--o| amp_record : has
hop ||--o{ custom_records : has
hop ||--o| blinded_data : has
htlc_attempt ||--o| htlc_settle_info : has
htlc_attempt ||--o| htlc_fail_info : has
htlc_fail_info }o--|| htlc_fail_reason_types : has
mpp_record ||--|| mpp_payments : references
amp_record ||--|| amp_payments : references
tlv_records ||--o{ custom_records : has
tlv_records ||--o{ first_hop_custom_records : has
payments ||--o{ first_hop_custom_records : has
|
ziggie1984
force-pushed
the
introduce-sql-schema-payments
branch
from
9741e37
to
616b851
Compare
ziggie1984
force-pushed
the
introduce-sql-schema-payments
branch
from
616b851
to
9817290
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Created a draft of the schema, will now proceed to create the whole sqlc harness now. If you have already feedback feel free to reach out.
Payment SQL Schema