• Expose wally_psbt_get_length
• Add wally_scriptpubkey_to_address
• Return written bytes in asset_pak_whitelistproof
• Elements bug fixes

Features

• Big rework of the bridge models as documented in https://github.com/LedgerHQ/ledger-live-common/pull/437
  • ETH sync is about 3 times more performant on average.
  • second account of some BTC-like heavy accounts will be enhanced, sometimes by a factor of 10. This was done with more paginated approach as well as not returning all inputs/outputs where there are a lot.
  • we are preparing our bridge API to be used in a paginated way.
  • balanceHistory field in account will allow better performance. It is already enabled and backed by libcore.
  • scanAccountsOnDevice was renamed to scanAccounts in context of the signature change
  • signAndBroadcast was split into signOperation and broadcast. With signature change and new events, including a way to obtain progress event for bitcoin operation signing (it sometimes takes a long time to stream to device).
• XTZ two more "legacy" derivation path as requested in https://github.com/LedgerHQ/ledger-live-desktop/issues/2559
• Remove bitpanda
• added custom error FirmwareNotRecognized when manager is not handling targetId instead of "HTTP 404"
• Manager V2 related: add isIncompleteState, improve formatSize, fix inferring apps for nano s 1.3.1
• disable the "flush" mecanism (sending a 0 apdu to reset device state) because not needed for incoming Manager v2

related libcore updates

• fixes XTZ. prepare support for BTC v3

Internal

Benchmark folder

this idea is to be refined and continue but we have introduced a way to compare performance of HEAD against past versions of live-common / libcore.

Improve the way we do tests

• Tests are mostly all running with Jest on the top level project now. With test coverage: https://codecov.io/github/LedgerHQ/ledger-live-common/
• We have a way to test scanAccounts and signOperation too

• fixes web3-subprovider (recent regression after upgrading ethereumjs-tx)

• hw-app-eth: updated the ERC20 list (see #449)
• hw-transport (all transports impacted): the error thrown when you access to a method while a previous call was not finished, used to be a TransportError. It's now a TransportRaceCondition error with a clearer message. This error is for developer to address: you need to prevent user to do an action twice (that might includes potentially blocking your UI when the device have a blocking action waiting for the user).

also internally:

• updated Jest

hw-app-btc: onDeviceStreaming

Introduce a new callback to be notified on progress events of APDUs streaming, typically for each input of a transaction.

onDeviceStreaming, if provided, will be called with an object { index, total, progress }. Progress will varies from 0.0 to 1.0 (included). index will vary from 0 to total (included).

onDeviceStreaming is not called at all if there is not more than X inputs. (today X = 3). In other words, it's only called when there is a significant waiting time to expect.

A typically integration you can do is: - initially display a spinner/loading state to the user - hook to onDeviceStreaming to display user some progress (e.progress*100).toFixed(0)+"%" - hook to onDeviceSignatureRequested to know when you need to ask user to confirm on device - hook to onDeviceSignatureGranted to know when user have confirmed on the device (and resume to a spinner)

• hw-app-btc: add a onDeviceStreaming callback. (see release notes of 5.6.3)
• hw-transport-mocker: add a way to automatically skip the APDU that doesn't match instead of erroring (useful in some circumstance, like some apdu were dropped and you want to replay with previous series of apdus)
• heavily test the mocker (mocker is used mostly for tests)

• internal update of some deps (babel, events,..)
• adding some internal errors for Ledger Live

• hw-app-eth: Fixes ERC20 data (some signature were incorrect in recent bump)

• hw-app-btc: bugfix when signTransaction is called with path="" (in context of coins like ZEC – recent regression)

• hw-app-btc: directly depends on sha.js and ripemd160 (to solve issues in context of Electron)

This marks the first major release in the v0.9.x series, as well as the first lnd release of the new decade! This release contains several new features, optimizations, new protocol-level features surfaced on the RPC layer, cross-implementation protocol compatibility fixes, and as usual a ton of bug fixes. Shouts out to all the contributors (over 30+ for this release!) as well as all those that reported bugs and tested, that helped make this new release happen.

Database Migrations

This release contains a single migration which modifies the on-disk representation of the invoices to be more flexible by utilizing the TLV serialization defined in the BOLT specifications. This change allows us to easily add more information to the invoices, which will be especially useful as we finalize our support for AMP within the daemon. The migration should look something like this upon initial start up:

2020-01-09 16:35:39.586 [INF] LTND: Version: 0.9.0-beta commit=v0.9.0-beta, build=development, logging=default
2020-01-09 16:35:39.587 [INF] LTND: Active chain: Bitcoin (network=simnet)
2020-01-09 16:35:39.589 [INF] CHDB: Checking for schema update: latest_version=12, db_version=11
2020-01-09 16:35:39.589 [INF] CHDB: Performing database schema migration
2020-01-09 16:35:39.589 [INF] CHDB: Applying migration #12
2020-01-09 16:35:39.589 [INF] CHDB: Migrating invoice bodies to TLV, adding payment addresses and feature vectors.
2020-01-09 16:35:39.589 [INF] CHDB: Migration to TLV invoice bodies, payment address, and features complete!

Verifying the Release

In order to verify the release, you'll need to have gpg or gpg2 installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already:

curl https://keybase.io/roasbeef/pgp_keys.asc | gpg --import

Once you have the required PGP keys, you can verify the release (assuming manifest-v0.9.0-beta.txt and manifest-v0.9.0-beta.txt.sig are in the current directory) with:

gpg --verify manifest-v0.9.0-beta.txt.sig

You should see the following if the verification was successful:

gpg: assuming signed data in 'manifest-v0.9.0-beta.txt'
gpg: Signature made Wed Jan 22 09:02:56 2020 PST
gpg:                using RSA key 4AB7F8DA6FAEBB3B70B1F903BC13F65E2DC84465
gpg: Good signature from &#34;Olaoluwa Osuntokun <laolu32@gmail.com>&#34; [ultimate]

That will verify the signature of the manifest file, which ensures integrity and authenticity of the archive you've downloaded locally containing the binaries. Next, depending on your operating system, you should then re-compute the sha256 hash of the archive with shasum -a 256 <filename>, compare it with the corresponding one in the manifest file, and ensure they match exactly.

Verifying the Release Binaries

Our release binaries are fully reproducible. Third parties are able to verify that the release binaries were produced properly without having to trust the release manager(s). See our reproducible builds guide for how this can be achieved. The release binaries are compiled with go1.13.6, which is required by verifiers to arrive at the same ones. They include the following build tags: autopilotrpc, signrpc, walletrpc, chainrpc, invoicesrpc, routerrpc, and watchtowerrpc. Note that these are already included in the release script, so they do not need to be provided.

Finally, you can also verify the tag itself with the following command:

git verify-tag v0.9.0-beta

Building the Contained Release

Users are able to rebuild the target release themselves without having to fetch any of the dependencies. In order to do so, assuming that vendor.tar.gz and lnd-source-v0.9.0-beta.tar.gz are in the current directory, follow these steps:

tar -xvzf vendor.tar.gz
tar -xvzf lnd-source-v0.9.0-beta.tar.gz
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.9.0-beta" ./cmd/lnd
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.9.0-beta" ./cmd/lncli

The -mod=vendor flag tells the go build command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.

Additionally, it's now possible to use the enclosed release.sh script to bundle a release for a specific system like so:

LNDBUILDSYS="linux-arm64 darwin-amd64" ./build/release/release.sh

⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️

Release Notes

Macaroon Bakery 🍪

A new custom macaroon bakery is now available through gRPC (BakeMacaroon) or the command line (lncli bakemacaroon)! The bakery allows users to mint macaroons with their own custom set of read/write permissions for use cases where the three pre-defined default macaroons (admin, readonly and invoice) are not enough. The documentation has also been updated to describe the process.

Note: The bakery requires a new permission that the existing admin.macaroon does not contain. Follow the steps described in the documentation to re-generate the admin macaroon.

New Payment Features

Multi-Path Payments

This release includes full support for receiving Multi-Path Payments as defined in BOLT 04, which is the first flavor of AMP rolling out on the network. MPP allows an invoice to be settled by multiple concurrent HTLCs, each of which carries a portion of the total payment.

With receiving support, MPP-aware senders can make better use of the payee's total inbound capacity in case there is no single channel that can support the payment amount. From a UX perspective, this allows wallet UIs to begin exploring a true, unified receiving capacity since the sender is able to utilize all of the receiver's channels for a single payment. As more MPP-aware senders come on to the network, 0.9 nodes will be able to accept payments without upgrading their software. We are currently targeting support for sending MPP in 0.10.

At a high-level, MPP is implemented via an additional TLV record in the final hop's onion blob, containing a random payment_secret and total_amt_msat being sent. Each HTLC carries this additional record in its onion blob, allowing the receiver to identify and settle an invoice with multiple HTLCs. Invoices will track all HTLCs in the set that are used to settle an invoices, as can be seen in this stripped down invoice settled by two HTLCs:

{                                                                  
    "memo": "multi path payment",                                             
    "r_preimage": "73d8d926c13579b0b1decf6b808a2bbf7db59073999c5d0a14d4dcc9c4225a49",
    "r_hash": "a3309015130da7cc28f529191a3d0aeb06edfedfd58dac1dfde8b1fbd4eadc91",
    "value_msat": "1000000",                                                                               
    "amt_paid_msat": "1000000",                                    
    "state": "SETTLED",                                            
    "htlcs": [                                                     
        {                                                          
            "chan_id": "571746046509056",                          
            "htlc_index": "4",                                     
            "amt_msat": "500000",                                 
            "accept_height": 525,                                  
            "accept_time": "1579112858",                           
            "resolve_time": "1579112860",                          
            "expiry_height": 565,                                  
            "state": "SETTLED",                                    
            "custom_records": {                                    
            },                                                     
            "mpp_total_amt_msat": "1000000"                        
        },
        {                                                          
            "chan_id": "571746046509056",                          
            "htlc_index": "5",                                     
            "amt_msat": "500000",                                 
            "accept_height": 525,                                  
            "accept_time": "1579112654",                           
            "resolve_time": "1579112654",                          
            "expiry_height": 565,                                  
            "state": "SETTLED",                                    
            "custom_records": {                                    
            },                                                     
            "mpp_total_amt_msat": "1000000"                        
        }                                                               
    ],                                                                                    
}

Note that amt_msat paid by each HTLC sums both to the mpp_total_amt_msat value included by the sender, as well as amt_paid_msat displaying the total amount received.

The rpc responses for ListPayments, routerrpc.SendPayment, and routerrpc.TrackPayment have also been extended with an htlcs field that exposes additional data about individual HTLC attempts. For now these will only show the last attempted HTLC, but will be populated with a full log of all HTLC attempts sent once MPP sending is fully integrated. This will allow us to display multiple successful HTLCs, as well as failed attempts. The latter will useful in allowing senders to inspect failed routes, and glean insights about why certain attempts failed.

A stripped down example of the new payment responses:

        {
            "payment_hash": "8eedb4bcd024f7f46fdf60b57e54dba60e19cb2c8934ec5b6fbb7f5efb58b29b",
            "payment_preimage": "cf6b808a2bbf7db59073973d8d926c13579b0b1de99c5d0a14d4dcc9c4225a49",
            "creation_time_ns": "1578620428000000000",
            "path": [
                "03186ce6a4a186895ee44a9864ef025fde8fb587d98926bfc370e2c366597a3f8f",
                "0206e82064f547768dcef9776b80cabdf82638aac86948e4c77b9ea931ea75bab5"
            ],
            "value_msat": "1000",
            "fee_msat": "0",
            "status": "SUCCEEDED",
            "htlcs": [
                {
                    "status": "SUCCEEDED",
                    "route": {
                        "total_time_lock": 612315,
                        "total_fees_msat": "0",
                        "total_amt_msat": "1000",
                        "hops": [
                            {
                                "chan_id": "612270746636810592",
                                "chan_capacity": "5000000",
                                "fee_msat": "0",
                                "amt_to_forward_msat": "1000",
                                "expiry": 612171,
                                "pub_key": "03186ce6a4a186895ee44a9864ef025fde8fb587d98926bfc370e2c366597a3f8f",
                                "tlv_payload": true,
                                "mpp_record": null,
                                "custom_records": {
                                }
                            },
                            {
                                "chan_id": "612270746636810592",
                                "chan_capacity": "5000000",
                                "expiry": 612171,
                                "fee_msat": "0",
                                "amt_to_forward_msat": "1000",
                                "pub_key": "0206e82064f547768dcef9776b80cabdf82638aac86948e4c77b9ea931ea75bab5",
                                "tlv_payload": true,
                                "mpp_record": {
                                    "payment_addr": "f91731b1e2227cc26137d2a2c0029ceef41ac8c481c8d3cc61510bf76ff27cbe",
                                    "total_amt_msat": "1000"
                                },
                                "custom_records": {
                                }
                            }
                        ]
                    },
                }
            ]
        }

Here we can see that each HTLC attempt displays it's full route, including the fees paid and timelocks incurred at each hop. MPP payments can be identified by the presences of an mpp_record attached to the final hop, which containing the new fields described above. If this field is not present, it indicates that payment was made using the legacy payment type.

Note: The path field has been deprecated in favor of extracting the pubkeys from the route directly. creation_date has also been deprecated and will replaced by creation_time_ns. As seen above, legacy creation_dates are converted into nanoseconds for display, but still only have one second resolution.

Weakness Addressed by MPP

MPP is more robust against active probing of the receiver, since the sender must know the invoice's payment_secret. This value serves as a one-time authentication token, further restricting the set of individuals who can settle a particular invoice. In particular, invoices can now only be settled by those who have seen it, rather than only needing to know the correct payment hash which is transmitted in clear over the network.

MPP also offers increased safety to situations involving over payment of invoices, most notably zero-value invoices used for tipping or donations. With the legacy protocol, it is possible to snipe the difference between the invoice amount and total amount carried by the HTLC, allowing intermediaries to take the difference as profit in addition to fees earned. MPP payments resolve this by privately communicating the sender's intended payment amount in the onion blob (via total_amt_msat) to the receiver. Since intermediaries will not know the payment_secret, sniping is prevented even if the intermediary can construct a valid HTLC with a smaller total_amt_msat.

Single-Shot MPP Payments by Default

Although 0.9 does not support sending true MPP payments, it does use the new MPP protocol as the default mechanism for single-shot payments (payments with only one HTLC). Doing so addresses the known weaknesses in the legacy payment mechanism discussed above, and seamlessly upgrades the overall privacy and security of the basic payment style used today.

No action needs to be taken in order to benefit from this upgrade. Senders will automatically detect the receiver's support via feature bits in BOLT 11 invoices, falling back to the legacy protocol for older nodes. As such, the new mechanism will be used when paying from 0.9 to another 0.9 node, or a newer release of C-Lightning or Eclair. Further, these improved payments are secure and can be used even if intermediaries are outdated, as it only requires support from the endpoints.

Custom Onion-Tunneled TLV Payment Metadata Records

It is now possible to attach additional data to a payment using custom records. The send calls on the RPC interface have a field dest_custom_records that takes the custom records to send. This field is also present on QueryRoutes. Custom records take up space in the fixed size onion packet, leaving fewer bytes to describe the route. The pathfinding algorithm needs to know how many bytes are left and limit the route length to that. Checks for length and size have been added.

On the receiving end, the custom records are stored in the invoice database and exposed through the existing invoice query calls (LookupInvoice, ListInvoices, etc). This new feature allows any payment on the network to attach additional meta-data along with the payment. As an example, a user could send a payment to an exchange to deposit funds, and attach their account ID along-side the payment, which the exchange then checks+verifies upon receipt before crediting their account with the deposit. This is a highly anticipated feature as it allows far a large class of new applications to be built on top of Lightning. For further ideas on how to leverage this new feature, check out Laolu's talk on Advanced Lightning Applications from last last year.

Sending a payment with custom records attached will look something like the following on the command line:

{
    "payment_error": "",
    "payment_preimage": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de",
    "payment_route": {
        "total_time_lock": 1657728,
        "total_fees": "0",
        "total_amt": "1000",
        "hops": [
            {
                "chan_id": "1589156041461923840",
                "chan_capacity": "16777215",
                "amt_to_forward": "1000",
                "fee": "0",
                "expiry": 1657728,
                "amt_to_forward_msat": "1000000",
                "fee_msat": "0",
                "pub_key": "0270685ca81a8e4d4d01beec5781f4cc924684072ae52c507f8ebe9daf0caaab7b",
                "tlv_payload": true,
                "mpp_record": null,
                "custom_records": {
                    "5482373484": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de"
                }
            }
        ],
        "total_fees_msat": "0",
        "total_amt_msat": "1000000"
    },
    "payment_hash": "8cf790cc128a0bb0552b3223d542adfba6a93c948f84e49dcd532309f5b85634"
}

Notice the new custom_records field, as well as the tlv_payload boolean which specifies that this route used the new modern onion paylaoad format.

On the receiver, end, the new output of ListInvoices looks something like:

        {
            "memo": "",
            "r_preimage": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de",
            "r_hash": "8cf790cc128a0bb0552b3223d542adfba6a93c948f84e49dcd532309f5b85634",
            <snip>
            &#34;htlcs&#34;: [
                {
                    &#34;chan_id&#34;: &#34;1589156041461923840&#34;,
                    &#34;htlc_index&#34;: &#34;8&#34;,
                    &#34;amt_msat&#34;: &#34;1000000&#34;,
                    &#34;accept_height&#34;: 1657685,
                    &#34;accept_time&#34;: &#34;1578617436&#34;,
                    &#34;resolve_time&#34;: &#34;1578617436&#34;,
                    &#34;expiry_height&#34;: 1657728,
                    &#34;state&#34;: &#34;SETTLED&#34;,
                    &#34;custom_records&#34;: {
                        &#34;5482373484&#34;: &#34;5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de&#34;
                    },
                    &#34;mpp_total_amt_msat&#34;: &#34;0&#34;
                }
            ],
          <snip>
        }

Once again the custom_records will will expose a map from the integer type of the record to the raw bytes of the value. Note that custom records are only permitted with a type greater than 65536, any records below this value are reversed for official protocol usage. For further details with respect to the new RPC-level changes to the payment sending/invoicing calls, see the lnd API documentation which is now up-to-date as of lnd v0.9.0.

With the introduction of custom record sending and receiving, it became possible to attach arbitrary data to a payment. One obvious use case is attaching a human-readable message to a payment. Especially in the case of spontaneous keysend payments, this can give the receiver some context on the payment.

For example: tipping. Usually people sending a tip would want to include some information on who they are or what the tip is for. For Lightning this may be even more desired than for other payment methods, because payments are anonymous by default.

Custom records can also be set using the lncli command like so:

lncli sendpayment -d 0374e7fb33eafd74fe1acb6db7680bb4aa78e1c839a6f954e38abfad680f645ef7 -a 100 --keysend --data 323442=00,3234556=ffff080812

To specify a string value, the standard command line tool xxd can be used (the example record id here is the 3-byte ascii string 'tip' converted to an integer):

--data 7629168=$(echo -n "Thank you!" | xxd -pu -c 10000)

(The -c parameter is to prevent xxd from inserting line breaks)

Note: The available onion blob space of 1300 bytes is used for routing info and custom records. The bigger the size of the custom records, the fewer bytes remain for routing info and the shorter the maximum route length will be.

New Payment Type: keysend

One application of custom records is a spontaneous payment, also known as keysend. In key send, a custom record is used to encode the payment preimage in the onion payload for the recipient of the payment. This allows them to pull the payment without prior knowledge of the preimage.

Note that spontaneous payment is not yet defined in the Lightning spec. Therefore the current implementation should be considered experimental and is subjected to change.

In order to send a keysend payment on the command line, a new flag --keysend needs to be specified, along with the destination (--dest) and the amount to send (--amt). Notice that we don't specify a payment hash anywhere! Instead, the sender will actually encrypt the payment pre-image to the receiver, who will then decrypt the payload along with the rest of the normal per-hop routing information.

A sample keysend executed at the command line resembles something liek the following:

🏔 tlncli sendpayment --keysend --dest=0270685ca81a8e4d4d01beec5781f4cc924684072ae52c507f8ebe9daf0caaab7b --amt=1000 --final_cltv_delta=40
{
    "payment_error": "",
    "payment_preimage": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de",
    "payment_route": {
        "total_time_lock": 1657728,
        "total_fees": "0",
        "total_amt": "1000",
        "hops": [
            {
                "chan_id": "1589156041461923840",
                "chan_capacity": "16777215",
                "amt_to_forward": "1000",
                "fee": "0",
                "expiry": 1657728,
                "amt_to_forward_msat": "1000000",
                "fee_msat": "0",
                "pub_key": "0270685ca81a8e4d4d01beec5781f4cc924684072ae52c507f8ebe9daf0caaab7b",
                "tlv_payload": true,
                "mpp_record": null,
                "custom_records": {
                    "5482373484": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de"
                }
            }
        ],
        "total_fees_msat": "0",
        "total_amt_msat": "1000000"
    },
    "payment_hash": "8cf790cc128a0bb0552b3223d542adfba6a93c948f84e49dcd532309f5b85634"
}

The key area to examine is the following:

                "custom_records": {
                    "5482373484": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de"
                }

Here we use lnd's keysend record record to package the pre-image to the receiver. You can verify that the payment_preimage field and the payment_hash field correspond. If one wishes to perform a keysend programmatically via the RPC API, then the DestCustomRecords field will need to be set accordingly.

On the receiver side, by default all nodes will reject any incoming keysend payments. In order to enable receiving such payments, nodes will need to opt-in by starting lnd with a new flag: --accept-keysend. When examining an invoice, it's easy to determine if it was payed using a keysend or not. When we receive a new incoming keysend payment that's to be accepted, we'll insert a new invoice into the database which looks something like the following:

        {
            "memo": "",
            "r_preimage": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de",
            "r_hash": "8cf790cc128a0bb0552b3223d542adfba6a93c948f84e49dcd532309f5b85634",
            "value": "1000",
            "value_msat": "1000000",
            "settled": true,
            "creation_date": "1578617436",
            "settle_date": "1578617436",
            "payment_request": "",
            "description_hash": null,
            <snip>
                {
                    &#34;chan_id&#34;: &#34;1589156041461923840&#34;,
                    &#34;htlc_index&#34;: &#34;8&#34;,
                    &#34;amt_msat&#34;: &#34;1000000&#34;,
                    &#34;accept_height&#34;: 1657685,
                    &#34;accept_time&#34;: &#34;1578617436&#34;,
                    &#34;resolve_time&#34;: &#34;1578617436&#34;,
                    &#34;expiry_height&#34;: 1657728,
                    &#34;state&#34;: &#34;SETTLED&#34;,
                    &#34;custom_records&#34;: {
                        &#34;5482373484&#34;: &#34;5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de&#34;
                    },
                    &#34;mpp_total_amt_msat&#34;: &#34;0&#34;
                }
            ],
            &#34;features&#34;: {
                &#34;9&#34;: {
                    &#34;name&#34;: &#34;tlv-onion&#34;,
                    &#34;is_required&#34;: false,
                    &#34;is_known&#34;: true
                }
            },
            &#34;is_keysend&#34;: true
        }

Notice that there's no payment_request value, and the set of new fields such as custom_records and information detailing the set of feature bits used to complete the payment (described in further detail below).

First-Class Rebalancing via Circular Payments to Self

We have added support to send a payment to yourself. This allows rebalancing of channels without the need for external scripts. To control the incoming channel, a new last hop routing restriction was added.

This new feature is a valuable tool in one's toolkit for node management as it allows you control the incoming and outgoing channels used to send a payment. Let's say I have a channel with ID 12345 and I sent to send funds from that channel (rebalancing to be able to receive more into the channel) to another channel that I have with peer 02e1dfa (rebalancing to be able to send more out of the channel). First, I'll create a new invoice on the command line for the amount I want to use for rebalancing purposes:

🏔 lncli --network=testnet addinvoice --amt=1000
{
    "r_hash": "ac7c8d8f72b34d68b1ed53b1401908b4fa3a2b0c41a21206686289f437bd02fa",
    "payment_request": "lntb10u1p0p0nm3pp5437gmrmjkdxk3v0d2wc5qxggknar52cvgx3pypngv2ylgdaaqtaqdqqcqzpgrzjq027z73uyyl7fy8pkrpcn7x0el82pz3fw974p2052de4uz4j5lqqx943xgqqq9qqqqqqqqqqqqqqqqqqpusp5ljp309kulcyghcfs7usd9ed0fxm85tquuyxf36z5k6wj6cddyssq9qy9qsqqwsldznwmg3prdgq7ddul7jspgk3n6fvkqgjc94phjc78jhkzc9y9q8vsmknp02mqg8tglhlvuseqq2nnar338tnqsspu79qvtmae0gpujtztv",
    "add_index": "167"
}

Now lnd is ready to execute the rebalancing attempt. To do this, we'll execute a new modified call to our trusty payinvoice command. Note that it's possible to do this programmatically+ as well using the RPC interface API. In any case, here's our command to complete this rebalancing attempt:

lncli --network=testnet payinvoice --allow_self_payment --outgoing_chan_id=12345 --last_hop=02e1dfa --pay_req=lntb10u1p0p0nm3pp5437gmrmjkdxk3v0d2wc5qxggknar52cvgx3pypngv2ylgdaaqtaqdqqcqzpgrzjq027z73uyyl7fy8pkrpcn7x0el82pz3fw974p2052de4uz4j5lqqx943xgqqq9qqqqqqqqqqqqqqqqqqpusp5ljp309kulcyghcfs7usd9ed0fxm85tquuyxf36z5k6wj6cddyssq9qy9qsqqwsldznwmg3prdgq7ddul7jspgk3n6fvkqgjc94phjc78jhkzc9y9q8vsmknp02mqg8tglhlvuseqq2nnar338tnqsspu79qvtmae0gpujtztv

Local balance check

Failure reporting in case none of the local channels has enough balance to execute the payment has been improved. We now return a specific insufficient balance payment result instead of only indicating that no route could be found.

mSAT support

Support for mSAT payments has been added on the RPC level for payments and invoices.

Privacy Enhancement

This new release of lnd will properly initialize the starting mix-header Sphinx packets with random bytes in order to patch a recently discovered privacy leak that could potentially allow the receiver of a payment to ascertain the number of hops the payment traveled. See this email by roasbeef for further details.

BOLT Compatibility

Invalid Onion Payload Failure

0.9 now returns the new invalid_onion_payload failure upon receiving a malformed onion TLV payload.

Flat Features

The legacy local and global feature bit namespaces have been merged into a single namespace, simplifying the way feature bits are assigned and used within the protocol. Features previously advertised as global in init messages will remain for backwards compatibility, while new features will be appended to the existing local features field. The global features field will be deprecated entirely once this is widespread within the network.

Validate Sorted Uncompressed Short Channel IDs

0.9 now verifies that short channel IDs in QueryShortChanIDs and ChannelRangeReply are sent in ascending order for the uncompressed format. This had already implemented for the zlib encoding, but is now checked across both encodings.

Add payment_secret to BOLT 11 Payment Requests

Payment requests containing a payment_secret are properly encoded and decoded in 0.9. In addition, every newly created payment requests will encode a random payment_secret so that senders may use the improved MPP payment mechanism.

https://github.com/lightningnetwork/lnd/pull/3788

Cross-Implementation Protocol Compatibility Fixes

It was discovered that slight deviations existed between the various implementations w.r.t the way they implemented the gossip queries protocol feature. This version of lnd has been patched to ensure that we'll properly send responses as dictated by the spec to ensure better compatibility with c-lightning and various other implementations.

Channel Funding+Management Extensions

Funding Flow Abstraction

This new release of lnd contains a dramatically refactored funding workflow internally. This new refactoring allowed us to abstract away aspects such as how the funding transaction is constructed. We have many new features planned around this cut out such as generic external channel funding driven by PSBT, internal merging of several funding transaction into a single, or specifying a cold commitment key (where all your funds are sent to on force close, only need for sweeping and not channel updates).

With this release, we're shipping a low-level call that allow external crafting of a funding transaction, given extra-protocol negotiation. In other words, a new RPC call that allows two parties to agree on the structure of a funding transaction outside of the protocol, obtain the outpoint, then use that as part of the normal funding workflow. Such a flow is useful for a number of advanced features such as the creation of channel factories.

The new FundingShim parameter allows a caller to initiate a new funding flow, with apriori knowledge of information typically generated dynamically during the funding workflow such as the outpoint of the final funding transaction and the keys used within the multi-sig output. A responder will then use the new FundingStateStep method to register a matching funding intent to dispatch automatically based on an expected pending channel ID.

As mentioned above, we have a number of interesting features planned for this new API call, so stay tuned!

Decoupled Min HTLC Settings

The default minimum htlc amount channel open parameter is lowered from 1000 msat to 1 msat. This parameter is immutable during the life of the channel. By changing it to the lowest possible value, maximum flexibility is achieved. The minimum htlc amount can still be constrained to a higher value through the forwarding policy. Updating the minimum forward amount is now possible via the RPC and lncli.

Option Upfront Shutdown Support

This release adds support for option_upfront_shutdown which enforces payout on cooperative close to an address that is set on channel open. lnd will disconnect from peers that attempt to close out to a script other than the one that they specified on startup. Further information about feature bit can be found in the BOLT2 specification. Use of this feature provides a partial protection against node compromise because channels can be set to close out to addresses which are not controlled by lnd.

An upfront shutdown address can be set with the close_address field in the lnrpc/lncli OpenChannel call. Currently, only seed controlled addresses can be set as the upfront shutdown address when we do not initiate a channel open because this step is non-interactive. lnd can be started with --enable-upfront-shutdown to set seed addresses for all channel opens and closes by default.

Sweep Small Outputs

Sweeper (responsible for sweeping close transaction outputs back into the wallet) has been enhanced so that it also sweeps small outputs that on their own do not reach the sweep transaction dust limit. It does so by attaching additional utxos from the wallet, if available. This change is a preparation for the upcoming anchor commitment format.

This change also makes it easier to use the BumpFee RPC (or lncli wallet bumpfee) call for generic CPFP usage.

Close Address

Channels can be closed out to a custom address using the delivery_address field in lncli/lnrpc's CloseChannel call. Note that this feature cannot be used if option_upfront_shutdown, described above, was pre-set for the channel on opening.

This feature allows users to close out a channel and make a payment within the same transaction. This can also be used with Lightning Loop to initiate a Loop In from a channel close, or even to create a new channel by sending to a P2WSH multi-sig script using the new funding APIs described above.

Autopilot External Score Trigger

Autopilot will now check whether it should open more channels after the external scores list has been updates.

Channel Fitness Tracking

A channelfitness subsytem for tracking the health of the node's existing set of channels has been added. This subsystem tracks the online status of the remote peers that we have channels open with. These results are not currently persisted, so uptime is tracked from node restart rather than from channel open time. A lifetime value which indicates how long a peer has been monitored for is provided to allow calculation of uptime percentages. Uptime and lifetime are exposed as experimental fields on the lnrpc/lncli ListChannels call; both are expressed in seconds. Note: these fields are experimental and may be moved or changed in future versions on LND.

Routing

Mission control

Several updates have been applied to mission control, the sub-system in lnd that is responsible for tracking the performance of nodes involved in past payments.

The estimation of success probabilities for channels has been improved. Channels are no longer evaluated in isolation. A new factor was added that represents the overall performance of a node. This prevents pathfinding from getting stuck in trying every channel of a badly performing node.

Probability estimation for local channels is more accurate now. This prevents unnecessary detours to reach a direct peer.

Mission control now remembers more about what happened in the past. Previously it only kept the last payment attempt for a channel. With this release, this is extended to tracking both the last failure and the last success. The goal is to improve payment success rates especially when small amount probes are performed frequently.

Finally, mission control state reporting has changed slightly. Probabilities are now amount-dependent and are therefore no longer reported through QueryMissionControl. A new call QueryProbability takes over this function.

Pathfinding Improvements

The routing algorithm acknowledges that a pair of nodes may have multiple channels between them with distinct forwarding policies. This is not a recommended practice, but needs to be dealt with nevertheless. lnd will craft a route that satisfies the most demanding (fee and time lock) channel of the channel set.

Routing decisions are based on cost and reliability. We applied a small touch up in this area to tie break on reliability when costs are equal.

RPC Enhancements

Deeper Feature Bit Inspection

The tried and true lncli getinfo call has been upgraded to show additional detail including the set of feature bits that the node is configured to advertise on the network. A sample of the new format of the call looks something like:

{
    "version": "0.9.0-beta commit=queue/v1.0.2-222-g6c8c99dae99e741d2817d444f8b11945ddd15e2e",
    "identity_pubkey": "0270685ca81a8e4d4d01beec5781f4cc924684072ae52c507f8ebe9daf0caaab7b",
     <snip>
    &#34;features&#34;: {
        &#34;0&#34;: {
            &#34;name&#34;: &#34;data-loss-protect&#34;,
            &#34;is_required&#34;: true,
            &#34;is_known&#34;: true
        },
        &#34;5&#34;: {
            &#34;name&#34;: &#34;upfront-shutdown-script&#34;,
            &#34;is_required&#34;: false,
            &#34;is_known&#34;: true
        },
        &#34;7&#34;: {
            &#34;name&#34;: &#34;gossip-queries&#34;,
            &#34;is_required&#34;: false,
            &#34;is_known&#34;: true
        },
        &#34;9&#34;: {
            &#34;name&#34;: &#34;tlv-onion&#34;,
            &#34;is_required&#34;: false,
            &#34;is_known&#34;: true
        },
        &#34;13&#34;: {
            &#34;name&#34;: &#34;static-remote-key&#34;,
            &#34;is_required&#34;: false,
            &#34;is_known&#34;: true
        },
        &#34;15&#34;: {
            &#34;name&#34;: &#34;payment-addr&#34;,
            &#34;is_required&#34;: false,
            &#34;is_known&#34;: true
        },
        &#34;17&#34;: {
            &#34;name&#34;: &#34;multi-path-payments&#34;,
            &#34;is_required&#34;: false,
            &#34;is_known&#34;: true
        }
    }
}

The same feature vectors format is also now exposed on a variety of other rpc responses, each with slightly difference semantics: - GetNodeInfo and DescribeGraph: displays the current feature vector advertised on a node_announcement within our routing table. - DecodePayReq: displays the feature vector decoded from an arbitrary payment request. - ListInvoices: displays the feature vectors advertised on our own invoices. - ListPeers: displays the set of features advertised by the remote party while exchanging init messages.

Updates to Default gRPC Settings

The receive buffer of the gRPC client in lncli and the REST proxy has been increased from 50MiB to 200MiB to avoid errors on very large response data (for example, thousands of transactions in listchaintxns).

Uniform lncli Hex-Encoding

The gRPC gateway library that is used to expose lnd's interface as a REST/JSON API on top of the normal gRPC API did not support request parameters of the type []byte (or bytes in the gRPC proto language). The update to version 1.8.6 of this library fixes this and should now allow REST users to use all API calls that lnd offers. Fields declared as bytes in the proto file must be encoded as base64 when used over REST. Some string fields in rpc.proto that were only added as a workaround in earlier versions have been deprecated and will be removed in the next version.

With this change, all raw byte fields exposed on the lncli command responses will now all be hex-encoded, with no more usage of base64. This change was made to clear up confusion that users ran into when information was encoded in multiple ways across distinct calls.

Updates to QueryRoutes

The QueryRoutes RPC call is now able to pin which outgoing channel is used when path finding, pin a pubkey to target for the final hop, and also accept custom unrolled hop hints. All of these changes make the RPC even more generally useable as it can be used for things like rebalancing pre-processing, advanced more advanced payment types, and fine-grained channel selection.

New RPC Calls

With the experimental signer RPC subserver you can now sign and verify messages with custom keys of the wallet (limited to the m/1017'/ special purpose derivation branch).

Note: This RPC requires a new permission, re-generate your admin.macaron and/or signer.macaroon to get the new permission.

A new RPC to derive shared Diffie-Hellman (ECDH) keys against a node's identity public key has been added to the experimental signer subserver.

A new RPC endpoint has been added to provide a subscription for peer online and offline events.

Mobile

Building

The mobile build is now created using the github.com/lightninglabs/falafel tool.

API changes

The Start library method has gotten an additional callback. Now lnd will notify both when the wallet is ready to be unlocked, and when lnd is ready to receive commands after the wallet has been unlocked.

Macaroons and TLS support

With falafel v0.7 the proper macaroon and TLS certificate is needed also when using the in-memory gRPC connections (which is done under the hood in the mobile build). This has no external facing changes, so it should work using the same APIs as before. This paves the way for using custom macaroons also on mobile, and moves the mobile API closer to the regular gRPC API.

Docker

The docker quick start files located in the docker directory have been made more user friendly by adding a persistent volume to lnd. This allows the lnd container to be restarted without losing its user data. The build instructions in docker/lnd/Dockerfile now also allow for quicker rebuilds by making use of the docker build cache.

The main Dockerfile in the root directory takes a branch, label or commit as a build argument to allow for custom builds.

The use of the docker-compose file is now easier and you can spin a btcd and lnd pair with just one command. Note: If you've used the docker-compose.yml file before, make sure you run docker-compose build --no-cache the first time you use to new version to make sure the changes are taken into account.

Config Changes

Default unsafe-disconnect Setting and Deprecation

This option is deprecated and now defaults to true, allowing users to disconnect peers with active channels via the disconnectpeer rpc. This option was originally added long ago for purposes of testing and off by default to prevent unintended consistency bugs. This extra caution has been unneeded for several releases and no longer required. Once unsafe-disconnect is fully removed, this will remain the default behavior.

Staged Travis Builds

The travis build matrix now does preliminary sanity checks before entering a second phase that executes long-running tests. This results in less wasteful usage of travis instances, and tightens the feedback loop for developers contributing to the project.

Bug fixes

• Accidental modification of database migration code has caused issues in the past. In this release, all migration code was isolated to prevent this from happening again.

• Several fixes have been applied to the exchange of commitment signatures. Empty commit signatures are no longer sent.

• Forwarding in lnd is "non-strict", meaning that any channel that is capable of carrying the payment may be utilized. The channel specified in the payload merely serves as a short-hand notation for the next node pub key. A bug was fixed where an incorrect failure message was returned if none of the channels qualified.

• Expired invoices are now canceled correctly. This fixes the "Invoices can be paid after expiration" bug.

• A bug in the chain notifier has been fixed that led to a panic when paying a subscribed script hash multiple times in different blocks.

Changelog

The full list of changes since v0.8.2-beta can be found here:

• https://github.com/lightningnetwork/lnd/compare/v0.8.2-beta...v0.9.0-beta

Contributors (Alphabetical Order)

Alex Bosworth Andras Banki-Horvath Andreas M. Antonopoulos Anton Kovalenko Arik Sosman bitromortac Bjarne Magnussen bluetegu Carla Kirk-Cohen Carson Mullins Conner Fromknecht Daniel McNally Dennis Reimann Elle Mouton Johan T. Halseth Joost Jager Juan Pablo Civile kiwiidb Lars Lehtonen Matheus Degiovani Eugene Siegel Olaoluwa Osuntokun Oliver Gugger Otto Suess Philipp Gillé Roei Erez Steven Roose Tomas Carnecky Vignesh Karthikeyan Wilmer Paulino Yancy Tibbens Yan Pritzker/laolu32@gmail.com