lnd v0.9.0-beta

This marks the first major release in the `v0.9.x` series, as well as the first `lnd` release of the new decade! This release contains several new features, optimizations, new protocol-level features surfaced on the RPC layer, cross-implementation protocol compatibility fixes, and as usual a ton of bug fixes. Shouts out to all the contributors (over 30+ for this release!) as well as all those that reported bugs and tested, that helped make this new release happen. # Database Migrations This release contains a single migration which modifies the on-disk representation of the invoices to be more flexible by utilizing the TLV serialization defined in the BOLT specifications. This change allows us to easily add more information to the invoices, which will be especially useful as we finalize our support for AMP within the daemon. The migration should look something like this upon initial start up: ``` 2020-01-09 16:35:39.586 [INF] LTND: Version: 0.9.0-beta commit=v0.9.0-beta, build=development, logging=default 2020-01-09 16:35:39.587 [INF] LTND: Active chain: Bitcoin (network=simnet) 2020-01-09 16:35:39.589 [INF] CHDB: Checking for schema update: latest_version=12, db_version=11 2020-01-09 16:35:39.589 [INF] CHDB: Performing database schema migration 2020-01-09 16:35:39.589 [INF] CHDB: Applying migration #12 2020-01-09 16:35:39.589 [INF] CHDB: Migrating invoice bodies to TLV, adding payment addresses and feature vectors. 2020-01-09 16:35:39.589 [INF] CHDB: Migration to TLV invoice bodies, payment address, and features complete! ``` # Verifying the Release In order to verify the release, you'll need to have `gpg` or `gpg2` installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already: ``` curl https://keybase.io/roasbeef/pgp_keys.asc | gpg --import ``` Once you have the required PGP keys, you can verify the release (assuming `manifest-v0.9.0-beta.txt` and `manifest-v0.9.0-beta.txt.sig` are in the current directory) with: ``` gpg --verify manifest-v0.9.0-beta.txt.sig ``` You should see the following if the verification was successful: ``` gpg: assuming signed data in 'manifest-v0.9.0-beta.txt' gpg: Signature made Wed Jan 22 09:02:56 2020 PST gpg: using RSA key 4AB7F8DA6FAEBB3B70B1F903BC13F65E2DC84465 gpg: Good signature from "Olaoluwa Osuntokun " [ultimate] ``` That will verify the signature of the manifest file, which ensures integrity and authenticity of the archive you've downloaded locally containing the binaries. Next, depending on your operating system, you should then re-compute the `sha256` hash of the archive with `shasum -a 256 `, compare it with the corresponding one in the manifest file, and ensure they match *exactly*. ## Verifying the Release Binaries Our release binaries are fully reproducible. Third parties are able to verify that the release binaries were produced properly without having to trust the release manager(s). See our [reproducible builds guide](https://github.com/lightningnetwork/lnd/tree/master/build/release) for how this can be achieved. The release binaries are compiled with `go1.13.6`, which is required by verifiers to arrive at the same ones. They include the following build tags: `autopilotrpc`, `signrpc`, `walletrpc`, `chainrpc`, `invoicesrpc`, `routerrpc`, and `watchtowerrpc`. Note that these are already included in the release script, so they do not need to be provided. Finally, you can also verify the _tag_ itself with the following command: ``` git verify-tag v0.9.0-beta ``` # Building the Contained Release Users are able to rebuild the target release themselves without having to fetch any of the dependencies. In order to do so, assuming that `vendor.tar.gz` and `lnd-source-v0.9.0-beta.tar.gz` are in the current directory, follow these steps: ``` tar -xvzf vendor.tar.gz tar -xvzf lnd-source-v0.9.0-beta.tar.gz GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.9.0-beta" ./cmd/lnd GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.9.0-beta" ./cmd/lncli ``` The `-mod=vendor` flag tells the `go build` command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory. Additionally, it's now possible to use the [enclosed `release.sh` script to bundle a release for a _specific_ system like so](https://github.com/lightningnetwork/lnd/pull/2191): ``` LNDBUILDSYS="linux-arm64 darwin-amd64" ./build/release/release.sh ``` ⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️ # Release Notes ## Macaroon Bakery 🍪 A new custom [macaroon bakery](https://github.com/lightningnetwork/lnd/pull/1160) is now available through gRPC (`BakeMacaroon`) or the command line (`lncli bakemacaroon`)! The bakery allows users to mint macaroons with their own custom set of read/write permissions for use cases where the three pre-defined default macaroons (admin, readonly and invoice) are not enough. The [documentation has also been updated](https://github.com/lightningnetwork/lnd/blob/master/macaroons/README.md#bakery) to describe the process. **Note:** The bakery requires a new permission that the existing `admin.macaroon` does not contain. Follow the steps described in [the documentation](https://github.com/lightningnetwork/lnd/blob/master/macaroons/README.md#upgrading-from-v080-beta-or-earlier) to re-generate the admin macaroon. ## New Payment Features ### Multi-Path Payments This release includes full support for receiving [Multi-Path Payments](https://github.com/lightningnetwork/lightning-rfc/pull/643) as defined in BOLT 04, which is the first flavor of AMP rolling out on the network. MPP allows an invoice to be settled by multiple concurrent HTLCs, each of which carries a portion of the total payment. With receiving support, MPP-aware senders can make better use of the payee's total inbound capacity in case there is no single channel that can support the payment amount. From a UX perspective, this allows wallet UIs to begin exploring a true, _unified_ receiving capacity since the sender is able to utilize all of the receiver's channels for a single payment. As more MPP-aware senders come on to the network, 0.9 nodes will be able to accept payments without upgrading their software. We are currently targeting support for sending MPP in 0.10. At a high-level, MPP is implemented via an additional TLV record in the final hop's onion blob, containing a random `payment_secret` and `total_amt_msat` being sent. [Each HTLC carries this additional record in its onion blob](https://github.com/lightningnetwork/lnd/pull/3442), allowing the receiver to identify and [settle an invoice with multiple HTLCs](https://github.com/lightningnetwork/lnd/pull/3415). Invoices will track all HTLCs in the set that are used to settle an invoices, as can be seen in this stripped down invoice settled by two HTLCs: ``` { "memo": "multi path payment", "r_preimage": "73d8d926c13579b0b1decf6b808a2bbf7db59073999c5d0a14d4dcc9c4225a49", "r_hash": "a3309015130da7cc28f529191a3d0aeb06edfedfd58dac1dfde8b1fbd4eadc91", "value_msat": "1000000", "amt_paid_msat": "1000000", "state": "SETTLED", "htlcs": [ { "chan_id": "571746046509056", "htlc_index": "4", "amt_msat": "500000", "accept_height": 525, "accept_time": "1579112858", "resolve_time": "1579112860", "expiry_height": 565, "state": "SETTLED", "custom_records": { }, "mpp_total_amt_msat": "1000000" }, { "chan_id": "571746046509056", "htlc_index": "5", "amt_msat": "500000", "accept_height": 525, "accept_time": "1579112654", "resolve_time": "1579112654", "expiry_height": 565, "state": "SETTLED", "custom_records": { }, "mpp_total_amt_msat": "1000000" } ], } ``` Note that `amt_msat` paid by each HTLC sums both to the `mpp_total_amt_msat` value included by the sender, as well as `amt_paid_msat` displaying the total amount received. The [rpc responses for `ListPayments`, `routerrpc.SendPayment`, and `routerrpc.TrackPayment` have also been extended with an `htlcs` field that exposes additional data about individual HTLC attempts](https://github.com/lightningnetwork/lnd/pull/3499). For now these will only show the last attempted HTLC, but will be populated with a full log of all HTLC attempts sent once MPP sending is fully integrated. This will allow us to display multiple successful HTLCs, as well as failed attempts. The latter will useful in allowing senders to inspect failed routes, and glean insights about why certain attempts failed. A stripped down example of the new payment responses: ``` { "payment_hash": "8eedb4bcd024f7f46fdf60b57e54dba60e19cb2c8934ec5b6fbb7f5efb58b29b", "payment_preimage": "cf6b808a2bbf7db59073973d8d926c13579b0b1de99c5d0a14d4dcc9c4225a49", "creation_time_ns": "1578620428000000000", "path": [ "03186ce6a4a186895ee44a9864ef025fde8fb587d98926bfc370e2c366597a3f8f", "0206e82064f547768dcef9776b80cabdf82638aac86948e4c77b9ea931ea75bab5" ], "value_msat": "1000", "fee_msat": "0", "status": "SUCCEEDED", "htlcs": [ { "status": "SUCCEEDED", "route": { "total_time_lock": 612315, "total_fees_msat": "0", "total_amt_msat": "1000", "hops": [ { "chan_id": "612270746636810592", "chan_capacity": "5000000", "fee_msat": "0", "amt_to_forward_msat": "1000", "expiry": 612171, "pub_key": "03186ce6a4a186895ee44a9864ef025fde8fb587d98926bfc370e2c366597a3f8f", "tlv_payload": true, "mpp_record": null, "custom_records": { } }, { "chan_id": "612270746636810592", "chan_capacity": "5000000", "expiry": 612171, "fee_msat": "0", "amt_to_forward_msat": "1000", "pub_key": "0206e82064f547768dcef9776b80cabdf82638aac86948e4c77b9ea931ea75bab5", "tlv_payload": true, "mpp_record": { "payment_addr": "f91731b1e2227cc26137d2a2c0029ceef41ac8c481c8d3cc61510bf76ff27cbe", "total_amt_msat": "1000" }, "custom_records": { } } ] }, } ] } ``` Here we can see that each HTLC attempt displays it's full route, including the fees paid and timelocks incurred at each hop. MPP payments can be identified by the presences of an `mpp_record` attached to the final hop, which containing the new fields described above. If this field is not present, it indicates that payment was made using the legacy payment type. Note: The `path` field has been deprecated in favor of extracting the pubkeys from the route directly. `creation_date` has also been deprecated and will replaced by `creation_time_ns`. As seen above, legacy `creation_date`s are converted into nanoseconds for display, but still only have one second resolution. #### Weakness Addressed by MPP MPP is more robust against active probing of the receiver, since the sender must know the invoice's `payment_secret`. This value serves as a one-time authentication token, further restricting the set of individuals who can settle a particular invoice. In particular, invoices can now only be settled by those who have seen it, rather than only needing to know the correct payment hash which is transmitted in clear over the network. MPP also offers increased safety to situations involving over payment of invoices, most notably zero-value invoices used for tipping or donations. With the legacy protocol, it is possible to snipe the difference between the invoice amount and total amount carried by the HTLC, allowing intermediaries to take the difference as profit in addition to fees earned. MPP payments resolve this by privately communicating the _sender's_ intended payment amount in the onion blob (via `total_amt_msat`) to the receiver. Since intermediaries will not know the `payment_secret`, sniping is prevented even if the intermediary can construct a valid HTLC with a smaller `total_amt_msat`. ### Single-Shot MPP Payments by Default Although 0.9 does not support sending true MPP payments, it does use the new MPP protocol as the default mechanism for single-shot payments (payments with only one HTLC). Doing so addresses the known weaknesses in the legacy payment mechanism discussed above, and seamlessly upgrades the overall privacy and security of the basic payment style used today. No action needs to be taken in order to benefit from this upgrade. Senders will [automatically detect the receiver's support via feature bits in BOLT 11 invoices, falling back to the legacy protocol for older nodes](https://github.com/lightningnetwork/lnd/pull/3679). As such, the new mechanism will be used when paying from 0.9 to another 0.9 node, or a newer release of C-Lightning or Eclair. Further, these improved payments are secure and can be used even if intermediaries are outdated, as it only requires support from the endpoints. ### Custom Onion-Tunneled TLV Payment Metadata Records It is now possible to [attach additional data](https://github.com/lightningnetwork/lnd/pull/3744) to a payment using custom records. The send calls on the RPC interface have a field `dest_custom_records` that takes the custom records to send. This field is also present on `QueryRoutes`. Custom records take up space in the fixed size onion packet, leaving fewer bytes to describe the route. The pathfinding algorithm needs to know how many bytes are left and limit the route length to that. Checks for [length](https://github.com/lightningnetwork/lnd/pull/3818) and [size](https://github.com/lightningnetwork/lnd/pull/3841) have been added. On the receiving end, the [custom records are stored](https://github.com/lightningnetwork/lnd/pull/3742) in the invoice database and exposed through the existing invoice query calls (`LookupInvoice`, `ListInvoices`, etc). This new feature allows any payment on the network to attach additional _meta-data_ along with the payment. As an example, a user could send a payment to an exchange to deposit funds, and attach their account ID along-side the payment, which the exchange then checks+verifies upon receipt before crediting their account with the deposit. This is a highly anticipated feature as it allows far a large class of new applications to be built on top of Lightning. For further ideas on how to leverage this new feature, [check out Laolu's talk on Advanced Lightning Applications from last last year](https://docs.google.com/presentation/d/1nCqBVPF63xi0apzDEfWNrj_IA0iNt1XYSINjHtmmqaI/edit?usp=sharing). Sending a payment with custom records attached will look something like the following on the command line: ``` { "payment_error": "", "payment_preimage": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de", "payment_route": { "total_time_lock": 1657728, "total_fees": "0", "total_amt": "1000", "hops": [ { "chan_id": "1589156041461923840", "chan_capacity": "16777215", "amt_to_forward": "1000", "fee": "0", "expiry": 1657728, "amt_to_forward_msat": "1000000", "fee_msat": "0", "pub_key": "0270685ca81a8e4d4d01beec5781f4cc924684072ae52c507f8ebe9daf0caaab7b", "tlv_payload": true, "mpp_record": null, "custom_records": { "5482373484": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de" } } ], "total_fees_msat": "0", "total_amt_msat": "1000000" }, "payment_hash": "8cf790cc128a0bb0552b3223d542adfba6a93c948f84e49dcd532309f5b85634" } ``` Notice the new `custom_records` field, as well as the `tlv_payload` boolean which specifies that this route used the new modern onion paylaoad format. On the receiver, end, the new output of `ListInvoices` looks something like: ``` { "memo": "", "r_preimage": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de", "r_hash": "8cf790cc128a0bb0552b3223d542adfba6a93c948f84e49dcd532309f5b85634", "htlcs": [ { "chan_id": "1589156041461923840", "htlc_index": "8", "amt_msat": "1000000", "accept_height": 1657685, "accept_time": "1578617436", "resolve_time": "1578617436", "expiry_height": 1657728, "state": "SETTLED", "custom_records": { "5482373484": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de" }, "mpp_total_amt_msat": "0" } ], } ``` Once again the `custom_records` will will expose a map from the integer type of the record to the raw bytes of the value. Note that custom records are only permitted with a type greater than `65536`, any records below this value are reversed for official protocol usage. For further details with respect to the new RPC-level changes to the payment sending/invoicing calls, see the [`lnd` API documentation which is now up-to-date as of `lnd` `v0.9.0`](https://api.lightning.community/). With the introduction of custom record sending and receiving, it became possible to attach arbitrary data to a payment. One obvious use case is attaching a human-readable message to a payment. Especially in the case of spontaneous keysend payments, this can give the receiver some context on the payment. For example: tipping. Usually people sending a tip would want to include some information on who they are or what the tip is for. For Lightning this may be even more desired than for other payment methods, because payments are anonymous by default. Custom records can also be set using the `lncli` command like so: ``` lncli sendpayment -d 0374e7fb33eafd74fe1acb6db7680bb4aa78e1c839a6f954e38abfad680f645ef7 -a 100 --keysend --data 323442=00,3234556=ffff080812 ``` To specify a string value, the standard command line tool xxd can be used (the example record id here is the 3-byte ascii string 'tip' converted to an integer): ``` --data 7629168=$(echo -n "Thank you!" | xxd -pu -c 10000) ``` (The `-c` parameter is to prevent `xxd` from inserting line breaks) Note: The available onion blob space of `1300` bytes is used for routing info and custom records. The bigger the size of the custom records, the fewer bytes remain for routing info and the shorter the maximum route length will be. ### New Payment Type: `keysend` One application of custom records is a spontaneous payment, also known as [keysend](https://github.com/lightningnetwork/lnd/pull/3795). In key send, a custom record is used to encode the payment preimage in the onion payload for the recipient of the payment. This allows them to pull the payment without prior knowledge of the preimage. Note that spontaneous payment is not yet defined in the Lightning spec. Therefore the current implementation should be considered experimental and is subjected to change. In order to send a `keysend` payment on the command line, a new flag `--keysend` needs to be specified, along with the destination (`--dest`) and the amount to send (`--amt`). Notice that we don't specify a payment hash anywhere! Instead, the sender will actually _encrypt_ the payment pre-image to the receiver, who will then decrypt the payload along with the rest of the normal per-hop routing information. A sample `keysend` executed at the command line resembles something liek the following: ``` 🏔 tlncli sendpayment --keysend --dest=0270685ca81a8e4d4d01beec5781f4cc924684072ae52c507f8ebe9daf0caaab7b --amt=1000 --final_cltv_delta=40 { "payment_error": "", "payment_preimage": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de", "payment_route": { "total_time_lock": 1657728, "total_fees": "0", "total_amt": "1000", "hops": [ { "chan_id": "1589156041461923840", "chan_capacity": "16777215", "amt_to_forward": "1000", "fee": "0", "expiry": 1657728, "amt_to_forward_msat": "1000000", "fee_msat": "0", "pub_key": "0270685ca81a8e4d4d01beec5781f4cc924684072ae52c507f8ebe9daf0caaab7b", "tlv_payload": true, "mpp_record": null, "custom_records": { "5482373484": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de" } } ], "total_fees_msat": "0", "total_amt_msat": "1000000" }, "payment_hash": "8cf790cc128a0bb0552b3223d542adfba6a93c948f84e49dcd532309f5b85634" } ``` The key area to examine is the following: ``` "custom_records": { "5482373484": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de" } ``` Here we use [lnd's keysend record](https://github.com/lightningnetwork/lnd/blob/6c8c99dae99e741d2817d444f8b11945ddd15e2e/record/experimental.go#L4) record to package the pre-image to the receiver. You can verify that the `payment_preimage` field and the `payment_hash` field correspond. If one wishes to perform a `keysend` _programmatically_ via the RPC API, [then the `DestCustomRecords` field will need to be set accordingly](https://api.lightning.community/#grpc-request-sendrequest-streaming). On the receiver side, by default all nodes will reject any incoming `keysend` payments. In order to enable receiving such payments, nodes will need to opt-in by starting `lnd` with a new flag: `--accept-keysend`. When examining an invoice, it's easy to determine if it was payed using a `keysend` or not. When we receive a new incoming `keysend` payment that's to be accepted, we'll insert a _new_ invoice into the database which looks something like the following: ``` { "memo": "", "r_preimage": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de", "r_hash": "8cf790cc128a0bb0552b3223d542adfba6a93c948f84e49dcd532309f5b85634", "value": "1000", "value_msat": "1000000", "settled": true, "creation_date": "1578617436", "settle_date": "1578617436", "payment_request": "", "description_hash": null, { "chan_id": "1589156041461923840", "htlc_index": "8", "amt_msat": "1000000", "accept_height": 1657685, "accept_time": "1578617436", "resolve_time": "1578617436", "expiry_height": 1657728, "state": "SETTLED", "custom_records": { "5482373484": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de" }, "mpp_total_amt_msat": "0" } ], "features": { "9": { "name": "tlv-onion", "is_required": false, "is_known": true } }, "is_keysend": true } ``` Notice that there's no `payment_request` value, and the set of new fields such as `custom_records` and information detailing the set of feature bits used to complete the payment (described in further detail below). ### First-Class Rebalancing via Circular Payments to Self We have added support to send a [payment to yourself](https://github.com/lightningnetwork/lnd/pull/3736). This allows rebalancing of channels without the need for external scripts. To control the incoming channel, a new [last hop](https://github.com/lightningnetwork/lnd/pull/3739) routing restriction was added. This new feature is a valuable tool in one's toolkit for node management as it allows you control the _incoming_ and _outgoing_ channels used to send a payment. Let's say I have a channel with ID `12345` and I sent to send funds _from_ that channel (rebalancing to be able to receive more into the channel) to another channel that I have with peer `02e1dfa` (rebalancing to be able to send more out of the channel). First, I'll create a new invoice on the command line for the amount I want to use for rebalancing purposes: ``` 🏔 lncli --network=testnet addinvoice --amt=1000 { "r_hash": "ac7c8d8f72b34d68b1ed53b1401908b4fa3a2b0c41a21206686289f437bd02fa", "payment_request": "lntb10u1p0p0nm3pp5437gmrmjkdxk3v0d2wc5qxggknar52cvgx3pypngv2ylgdaaqtaqdqqcqzpgrzjq027z73uyyl7fy8pkrpcn7x0el82pz3fw974p2052de4uz4j5lqqx943xgqqq9qqqqqqqqqqqqqqqqqqpusp5ljp309kulcyghcfs7usd9ed0fxm85tquuyxf36z5k6wj6cddyssq9qy9qsqqwsldznwmg3prdgq7ddul7jspgk3n6fvkqgjc94phjc78jhkzc9y9q8vsmknp02mqg8tglhlvuseqq2nnar338tnqsspu79qvtmae0gpujtztv", "add_index": "167" } ``` Now `lnd` is ready to execute the rebalancing attempt. To do this, we'll execute a new modified call to our trusty `payinvoice` command. Note that it's possible to do this _programmatically_+ as well using the RPC interface API. In any case, here's our command to _complete_ this rebalancing attempt: ``` lncli --network=testnet payinvoice --allow_self_payment --outgoing_chan_id=12345 --last_hop=02e1dfa --pay_req=lntb10u1p0p0nm3pp5437gmrmjkdxk3v0d2wc5qxggknar52cvgx3pypngv2ylgdaaqtaqdqqcqzpgrzjq027z73uyyl7fy8pkrpcn7x0el82pz3fw974p2052de4uz4j5lqqx943xgqqq9qqqqqqqqqqqqqqqqqqpusp5ljp309kulcyghcfs7usd9ed0fxm85tquuyxf36z5k6wj6cddyssq9qy9qsqqwsldznwmg3prdgq7ddul7jspgk3n6fvkqgjc94phjc78jhkzc9y9q8vsmknp02mqg8tglhlvuseqq2nnar338tnqsspu79qvtmae0gpujtztv ``` ### Local balance check Failure reporting in case none of the local channels has enough balance to execute the payment has been improved. We now return a specific [insufficient balance](https://github.com/lightningnetwork/lnd/pull/3749) payment result instead of only indicating that no route could be found. ### mSAT support Support for `mSAT` payments has been added on the RPC level for [payments](https://github.com/lightningnetwork/lnd/pull/3706) and [invoices](https://github.com/lightningnetwork/lnd/pull/3729). ## Privacy Enhancement This new release of `lnd` will properly initialize the starting mix-header Sphinx packets with random bytes in order to patch a recently discovered privacy leak that could potentially allow the receiver of a payment to ascertain the number of hops the payment traveled. See this [email by roasbeef for further details](https://lists.linuxfoundation.org/pipermail/lightning-dev/2019-November/002288.html). ## BOLT Compatibility ### Invalid Onion Payload Failure 0.9 now [returns the new `invalid_onion_payload` failure upon receiving a malformed onion TLV payload](https://github.com/lightningnetwork/lnd/pull/3470). ### Flat Features The legacy local and global feature bit namespaces have been [merged into a single namespace](https://github.com/lightningnetwork/lnd/pull/3685), simplifying the way feature bits are assigned and used within the protocol. Features previously advertised as global in `init` messages will remain for backwards compatibility, while new features will be appended to the existing local features field. The global features field will be deprecated entirely once this is widespread within the network. ### Validate Sorted Uncompressed Short Channel IDs 0.9 now [verifies that short channel IDs in `QueryShortChanIDs` and `ChannelRangeReply` are sent in ascending order for the uncompressed format](https://github.com/lightningnetwork/lnd/pull/3743). This had already implemented for the zlib encoding, but is now checked across both encodings. ### Add `payment_secret` to BOLT 11 Payment Requests Payment requests containing a `payment_secret` are properly encoded and decoded in 0.9. In addition, every newly created payment requests will encode a random `payment_secret` so that senders may use the improved MPP payment mechanism. https://github.com/lightningnetwork/lnd/pull/3788 ## Cross-Implementation Protocol Compatibility Fixes It was discovered that slight deviations existed between the various implementations w.r.t the way they implemented the gossip queries protocol feature. This version of `lnd` [has been patched](https://github.com/lightningnetwork/lnd/pull/3785) to ensure [that we'll properly send responses](https://github.com/lightningnetwork/lnd/pull/3836) as dictated by the spec to ensure better compatibility with c-lightning and various other implementations. ## Channel Funding+Management Extensions ### Funding Flow Abstraction [This new release of `lnd` contains a dramatically refactored funding workflow internally](https://github.com/lightningnetwork/lnd/pull/3659). This new refactoring allowed us to abstract away aspects such as _how_ the funding transaction is constructed. We have many new features planned around this cut out such as generic external channel funding driven by PSBT, internal merging of several funding transaction into a single, or specifying a cold commitment key (where all your funds are sent to on force close, only need for sweeping and not channel updates). [With this release, we're shipping a low-level call that allow _external_ crafting of a funding transaction, given extra-protocol negotiation](https://github.com/lightningnetwork/lnd/pull/3722). In other words, a new RPC call that allows two parties to agree on the structure of a funding transaction outside of the protocol, obtain the outpoint, then use that as part of the normal funding workflow. Such a flow is useful for a number of _advanced_ features such as the creation of *channel factories*. The new [`FundingShim`](https://api.lightning.community/#fundingshim) parameter allows a caller to initiate a new funding flow, with apriori knowledge of information typically generated dynamically during the funding workflow such as the outpoint of the final funding transaction and the keys used within the multi-sig output. [A responder will then use the new `FundingStateStep` method to register a matching funding intent to dispatch automatically based on an expected pending channel ID](https://api.lightning.community/#fundingstatestep). As mentioned above, we have a number of interesting features planned for this new API call, so stay tuned! ### Decoupled Min HTLC Settings The default minimum htlc amount channel open parameter is [lowered from 1000 msat to 1 msat](https://github.com/lightningnetwork/lnd/pull/3697). This parameter is immutable during the life of the channel. By changing it to the lowest possible value, maximum flexibility is achieved. The minimum htlc amount can still be constrained to a higher value through the forwarding policy. Updating the minimum forward amount is now possible via the RPC and `lncli`. ### Option Upfront Shutdown Support This release adds [support](https://github.com/lightningnetwork/lnd/pull/3655) for `option_upfront_shutdown` which enforces payout on cooperative close to an address that is set on channel open. `lnd` will disconnect from peers that attempt to close out to a script other than the one that they specified on startup. Further information about feature bit can be found in the [BOLT2 specification](https://github.com/lightningnetwork/lightning-rfc/blob/8e69306e0a93375a1bbb1a0099f7ce3025ae4c0f/02-peer-protocol.md#the-open_channel-message). Use of this feature provides a partial protection against node compromise because channels can be set to close out to addresses which are not controlled by `lnd`. An upfront shutdown address can be set with the `close_address` field in the lnrpc/lncli `OpenChannel` call. Currently, only seed controlled addresses can be set as the upfront shutdown address when we do not initiate a channel open because this step is non-interactive. `lnd` can be started with `--enable-upfront-shutdown` to set seed addresses for all channel opens and closes by default. ### Sweep Small Outputs Sweeper (responsible for sweeping close transaction outputs back into the wallet) has been enhanced so that it also [sweeps small outputs](https://github.com/lightningnetwork/lnd/pull/3814) that on their own do not reach the sweep transaction dust limit. It does so by attaching additional utxos from the wallet, if available. This change is a preparation for the upcoming [anchor commitment format](https://github.com/lightningnetwork/lightning-rfc/pull/688). This change also makes it easier to use the [`BumpFee` RPC](https://github.com/lightningnetwork/lnd/blob/03cf9a53e4cb63aba47b715faa9e3cf52fe2d09c/lnrpc/walletrpc/walletkit.proto#L277) (or `lncli wallet bumpfee`) call for generic CPFP usage. ### Close Address Channels can be closed out to a [custom address](https://github.com/lightningnetwork/lnd/pull/3702) using the `delivery_address` field in lncli/lnrpc's `CloseChannel` call. Note that this feature cannot be used if `option_upfront_shutdown`, described [above](###Option-Upfront-Shutdown-Support), was pre-set for the channel on opening. This feature allows users to close out a channel and make a payment within the same transaction. This can also be used with [`Lightning Loop`](https://blog.lightning.engineering/posts/2019/03/20/loop.html) to initiate a _Loop In_ from a channel close, or even to create a new channel by sending to a P2WSH multi-sig script using the new funding APIs described above. ### Autopilot External Score Trigger Autopilot will now [check whether it should open more channels](https://github.com/lightningnetwork/lnd/pull/3520) after the external scores list has been updates. ## Channel Fitness Tracking A [channelfitness subsytem](https://github.com/lightningnetwork/lnd/pull/3332) for tracking the health of the node's existing set of channels has been added. This subsystem tracks the online status of the remote peers that we have channels open with. These results are not currently persisted, so uptime is tracked from node restart rather than from channel open time. A lifetime value which indicates how long a peer has been monitored for is provided to allow calculation of uptime percentages. Uptime and lifetime are exposed as experimental fields on the lnrpc/lncli `ListChannels` call; both are expressed in seconds. **Note:** these fields are experimental and may be moved or changed in future versions on LND. ## Routing ### Mission control Several updates have been applied to mission control, the sub-system in `lnd` that is responsible for tracking the performance of nodes involved in past payments. The estimation of success probabilities for channels has been improved. Channels are no longer evaluated in isolation. A [new factor](https://github.com/lightningnetwork/lnd/pull/3462) was added that represents the overall performance of a node. This prevents pathfinding from getting stuck in trying every channel of a badly performing node. Probability estimation for local channels is [more accurate](https://github.com/lightningnetwork/lnd/pull/3686) now. This prevents unnecessary detours to reach a direct peer. Mission control now remembers more about what happened in the past. Previously it only kept the last payment attempt for a channel. With this release, this is extended to [tracking both the last failure and the last success](https://github.com/lightningnetwork/lnd/pull/3493). The goal is to improve payment success rates especially when small amount probes are performed frequently. Finally, mission control state reporting has changed slightly. Probabilities are now amount-dependent and are therefore no longer reported through `QueryMissionControl`. A new call [`QueryProbability`](https://github.com/lightningnetwork/lnd/pull/3556) takes over this function. ### Pathfinding Improvements The [routing algorithm acknowledges](https://github.com/lightningnetwork/lnd/pull/3558) that a pair of nodes may have multiple channels between them with distinct forwarding policies. This is [not a recommended practice](https://github.com/lightningnetwork/lightning-rfc/blob/master/04-onion-routing.md#recommendation), but needs to be dealt with nevertheless. `lnd` will craft a route that satisfies the most demanding (fee and time lock) channel of the channel set. Routing decisions are based on cost and reliability. We applied a small touch up in this area to [tie break on reliability](https://github.com/lightningnetwork/lnd/pull/3782) when costs are equal. ## RPC Enhancements ### Deeper Feature Bit Inspection The tried and true `lncli getinfo` call has been upgraded to show additional detail including the set of _feature bits_ that the node is configured to advertise on the network. A sample of the new format of the call looks something like: ``` { "version": "0.9.0-beta commit=queue/v1.0.2-222-g6c8c99dae99e741d2817d444f8b11945ddd15e2e", "identity_pubkey": "0270685ca81a8e4d4d01beec5781f4cc924684072ae52c507f8ebe9daf0caaab7b", "features": { "0": { "name": "data-loss-protect", "is_required": true, "is_known": true }, "5": { "name": "upfront-shutdown-script", "is_required": false, "is_known": true }, "7": { "name": "gossip-queries", "is_required": false, "is_known": true }, "9": { "name": "tlv-onion", "is_required": false, "is_known": true }, "13": { "name": "static-remote-key", "is_required": false, "is_known": true }, "15": { "name": "payment-addr", "is_required": false, "is_known": true }, "17": { "name": "multi-path-payments", "is_required": false, "is_known": true } } } ``` The same feature vectors format is also now exposed on a variety of other rpc responses, each with slightly difference semantics: - `GetNodeInfo` and `DescribeGraph`: displays the current feature vector advertised on a `node_announcement` within our routing table. - `DecodePayReq`: displays the feature vector decoded from an arbitrary payment request. - `ListInvoices`: displays the feature vectors advertised on our own invoices. - `ListPeers`: displays the set of features advertised by the remote party while exchanging `init` messages. ### Updates to Default gRPC Settings The receive buffer of the gRPC client in [`lncli`](https://github.com/lightningnetwork/lnd/pull/3730) and the [REST proxy](https://github.com/lightningnetwork/lnd/pull/3740) has been increased from 50MiB to 200MiB to avoid errors on very large response data (for example, thousands of transactions in `listchaintxns`). ### Uniform `lncli` Hex-Encoding The gRPC gateway library that is used to expose lnd's interface as a REST/JSON API on top of the normal gRPC API did not support request parameters of the type `[]byte` (or `bytes` in the gRPC proto language). The [update to version 1.8.6 of this library](https://github.com/lightningnetwork/lnd/pull/3650) fixes this and should now allow REST users to use all API calls that lnd offers. Fields declared as `bytes` in the proto file must be encoded as base64 when used over REST. Some [`string` fields in `rpc.proto`](https://github.com/lightningnetwork/lnd/pull/3661) that were only added as a workaround in earlier versions have been deprecated and will be removed in the next version. With this change, _all_ raw byte fields exposed on the `lncli` command responses will now all be _hex-encoded_, with no more usage of base64. This change was made to clear up confusion that users ran into when information was encoded in multiple ways across distinct calls. ### Updates to `QueryRoutes` [The `QueryRoutes` RPC call is now able to pin _which outgoing_ channel is used when path finding, pin a pubkey to target for the final hop, and also accept custom unrolled hop hints](https://github.com/lightningnetwork/lnd/pull/3911). All of these changes make the RPC even _more_ generally useable as it can be used for things like rebalancing pre-processing, advanced more advanced payment types, and fine-grained channel selection. ### New RPC Calls With the experimental `signer` RPC subserver you can now [sign and verify messages with custom keys](https://github.com/lightningnetwork/lnd/pull/3812) of the wallet (limited to the `m/1017'/` special purpose derivation branch). **Note:** This RPC requires a new permission, re-generate your `admin.macaron` and/or `signer.macaroon` to get the new permission. A new RPC to [derive shared Diffie-Hellman (ECDH) keys](https://github.com/lightningnetwork/lnd/pull/3812) against a node's identity public key has been added to the experimental `signer` subserver. A new RPC endpoint has been added to provide a subscription for [peer online and offline events](https://github.com/lightningnetwork/lnd/pull/3397). ## Mobile ### Building The mobile build is [now created](https://github.com/lightningnetwork/lnd/pull/3748) using the `github.com/lightninglabs/falafel` tool. ### API changes The `Start` library method [has gotten an additional callback](https://github.com/lightningnetwork/lnd/pull/3775). Now `lnd` will notify both when the wallet is ready to be unlocked, and when `lnd` is ready to receive commands after the wallet has been unlocked. ### Macaroons and TLS support With `falafel v0.7` the proper [macaroon and TLS certificate](https://github.com/lightningnetwork/lnd/pull/3783) is needed also when using the in-memory gRPC connections (which is done under the hood in the mobile build). This has no external facing changes, so it should work using the same APIs as before. This paves the way for using custom macaroons also on mobile, and moves the mobile API closer to the regular gRPC API. ## Docker The docker quick start files located in the `docker` directory have been made more user friendly by [adding a persistent volume to lnd](https://github.com/lightningnetwork/lnd/pull/2533). This allows the lnd container to be restarted without losing its user data. The build instructions in `docker/lnd/Dockerfile` now also allow for quicker rebuilds [by making use of the docker build cache](https://github.com/lightningnetwork/lnd/pull/2598). The main `Dockerfile` in the root directory [takes a branch, label or commit as a build argument](https://github.com/lightningnetwork/lnd/pull/2520) to allow for custom builds. The use of the `docker-compose` file [is now easier](https://github.com/lightningnetwork/lnd/pull/2699) and you can spin a `btcd` and `lnd` pair with just one command. **Note:** If you've used the `docker-compose.yml` file before, make sure you run `docker-compose build --no-cache` the first time you use to new version to make sure the changes are taken into account. ## Config Changes ### Default `unsafe-disconnect` Setting and Deprecation This option is deprecated and now defaults to true, [allowing users to disconnect peers with active channels via the `disconnectpeer` rpc](https://github.com/lightningnetwork/lnd/pull/3648). This option was originally added long ago for purposes of testing and off by default to prevent unintended consistency bugs. This extra caution has been unneeded for several releases and no longer required. Once `unsafe-disconnect` is fully removed, this will remain the default behavior. ## Staged Travis Builds The [travis build matrix now does preliminary sanity checks before entering a second phase that executes long-running tests](https://github.com/lightningnetwork/lnd/pull/3794). This results in less wasteful usage of travis instances, and tightens the feedback loop for developers contributing to the project. ## Bug fixes * Accidental modification of database migration code has caused issues in the past. In this release, all migration code was [isolated](https://github.com/lightningnetwork/lnd/pull/3633) to prevent this from happening again. * Several fixes have been applied to the [exchange of commitment signatures](https://github.com/lightningnetwork/lnd/pull/2927). Empty commit signatures are no longer sent. * Forwarding in `lnd` is "non-strict", meaning that any channel that is capable of carrying the payment may be utilized. The channel specified in the payload merely serves as a short-hand notation for the next node pub key. A [bug](https://github.com/lightningnetwork/lnd/pull/3547) was fixed where an incorrect failure message was returned if none of the channels qualified. * Expired invoices are now canceled correctly. This fixes the "[Invoices can be paid after expiration](https://github.com/lightningnetwork/lnd/pull/3694)" bug. * A bug in the chain notifier has been fixed that led to a panic when [paying a subscribed script hash multiple times](https://github.com/lightningnetwork/lnd/pull/3569) in different blocks. ## Changelog The full list of changes since `v0.8.2-beta` can be found here: * https://github.com/lightningnetwork/lnd/compare/v0.8.2-beta...v0.9.0-beta # Contributors (Alphabetical Order) Alex Bosworth Andras Banki-Horvath Andreas M. Antonopoulos Anton Kovalenko Arik Sosman bitromortac Bjarne Magnussen bluetegu Carla Kirk-Cohen Carson Mullins Conner Fromknecht Daniel McNally Dennis Reimann Elle Mouton Johan T. Halseth Joost Jager Juan Pablo Civile kiwiidb Lars Lehtonen Matheus Degiovani Eugene Siegel Olaoluwa Osuntokun Oliver Gugger Otto Suess Philipp Gillé Roei Erez Steven Roose Tomas Carnecky Vignesh Karthikeyan Wilmer Paulino Yancy Tibbens Yan Pritzker