lnd v0.10.0-beta

This marks the first major release in the `v0.10.0` series! This release contains several new features including support for sender-side MPP payment splitting, PSBT channel funding, and much more. Notably, this release also includes a new experimental commitment type (anchor commitments). As usual, this release also contains a number of optimizations and bug fixes. # Database Migrations This release contains a single migration which modifies the structure of the on-disk representation of payments for the sake of the new MPP payment splitting feature. This migration should look something like this upon initial start up: ``` 2020-04-15 18:19:18.113 [INF] LTND: Version: 0.10.0-beta. commit=v0.10.0-beta, build=development, logging=default 2020-04-15 18:19:18.376 [INF] LTND: Opening the main database, this might take a few minutes.. 2020-04-15 18:19:18.380 [INF] CHDB: Checking for schema update: latest_version=13, db_version=12 2020-04-15 18:19:18.381 [INF] CHDB: Performing database schema migration 2020-04-15 18:19:18.381 [INF] CHDB: Applying migration #13 2020-04-15 18:19:18.381 [INF] CHDB: Migrating payments to mpp structure 2020-03-15 18:19:18.391 [INF] LTND: Database now open (time_to_open=15.651683ms) ``` # Verifying the Release In order to verify the release, you'll need to have `gpg` or `gpg2` installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already: ``` curl https://keybase.io/roasbeef/pgp_keys.asc | gpg --import ``` Once you have the required PGP keys, you can verify the release (assuming `manifest-v0.9.2-beta.txt` and `manifest-v0.10.0-beta.txt.sig` are in the current directory) with: ``` gpg --verify manifest-v0.10.0-beta.txt.sig ``` You should see the following if the verification was successful: ``` gpg: assuming signed data in 'manifest-v0.10.0-beta.txt' gpg: Signature made Wed Apr 29 11:02:38 2020 PDT gpg: using RSA key 4AB7F8DA6FAEBB3B70B1F903BC13F65E2DC84465 gpg: Good signature from "Olaoluwa Osuntokun " [ultimate] ``` That will verify the signature of the manifest file, which ensures integrity and authenticity of the archive you've downloaded locally containing the binaries. Next, depending on your operating system, you should then re-compute the `sha256` hash of the archive with `shasum -a 256 `, compare it with the corresponding one in the manifest file, and ensure they match *exactly*. ## Verifying the Release Binaries Our release binaries are fully reproducible. Third parties are able to verify that the release binaries were produced properly without having to trust the release manager(s). See our [reproducible builds guide](https://github.com/lightningnetwork/lnd/tree/master/build/release) for how this can be achieved. The release binaries are compiled with `go1.13.10`, which is required by verifiers to arrive at the same ones. They include the following build tags: `autopilotrpc`, `signrpc`, `walletrpc`, `chainrpc`, `invoicesrpc`, `routerrpc`, and `watchtowerrpc`. Note that these are already included in the release script, so they do not need to be provided. The `make release` command can be used to ensure one rebuilds with all the same flags used for the release. If one wishes to build for only a single platform, then `make release sys= tag=` can be used. Finally, you can also verify the _tag_ itself with the following command: ``` git verify-tag v0.10.0-beta ``` # Building the Contained Release Users are able to rebuild the target release themselves without having to fetch any of the dependencies. In order to do so, assuming that `vendor.tar.gz` and `lnd-source-v0.10.0-beta.tar.gz` are in the current directory, follow these steps: ``` tar -xvzf vendor.tar.gz tar -xvzf lnd-source-v0.10.0-beta.tar.gz GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.10.0-beta" ./cmd/lnd GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.10.0-beta" ./cmd/lncli ``` The `-mod=vendor` flag tells the `go build` command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory. Additionally, it's now possible to use the [enclosed `release.sh` script to bundle a release for a _specific_ system like so](https://github.com/lightningnetwork/lnd/pull/2191): ``` make release sys="linux-arm64 darwin-amd64" ``` ⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️ # Release Notes ## Architectural Changes As is today, the default database for `lnd` is `bbolt`, an embedded key-value store based on a B+Tree. In the future, we'd like to enable users to be able to swap out `bbolt` for any arbitrary strongly consistent database. In this release, we've made a step towards this goal by introducing a new [KV-store database abstraction](https://github.com/lightningnetwork/lnd/pull/3833). This new interface allows any database implementation which can adhere to the interface to be dropped into `lnd`. In the upcoming versions of `lnd`, the first new database [we plan to support is `etcd`](https://github.com/lightningnetwork/lnd/pull/4015), a strongly consistent distributed key-value store which is used as the basis of projects such as Kubernetes. `etcd` actually uses `bbolt` underneath the hood, so it won't be a drastic departure. [Once this issue to further abstract the initialization of `lnd` is complete](https://github.com/lightningnetwork/lnd/issues/3944), then users will be able to "build their own lnd", attaching a database implementation _at runtime_ which adhere to the the primary interface. ## Wallet Bug Fixes A bug was fixed in `btcwallet` that would at times (non-deterministically) result in _change addresses_ not being picked up doing a full wallet recovery instance. A bug related to map iteration order would at times cause a change address to be placed a "non-default scope" (where we store all the keys for contracts: multi-sig keys, etc), rather than the main "default scope". The end result is that due to this bug, it's possible that prior instances of full wallet recovery (seed import) _missed_ some funds on chain. If you've ever recovered an existing `lnd` wallet, then _ceased using it_, we **strongly recommend** that you attempt the seed import process again, as there may be funds that weren't picked up the first time around. ## Payments ### Multi-Path Payments Sending Support Receiving multi-path payments was already possible since version 0.9. Today we are pleased to announce that this release completes the circle by adding support for [multi-path payment sending](https://github.com/lightningnetwork/lnd/pull/3967). Preparatory changes for the send functionality that were implemented during the development process are [database support for multiple htlcs per payment](https://github.com/lightningnetwork/lnd/pull/4000) and a payment loop that is [aware of partial sends](https://github.com/lightningnetwork/lnd/pull/3970). The splitting algorithm that is employed is a variation of "divide and conquer". First, `lnd` will try to complete the payment in a single shot. If that is not possible - because channel capacities aren't sufficient or other constraints like the fee limit are hit - an attempt is made for half the payment amount. After that htlc has been launched, a second path is searched for to pay the remaining amount (the other half). If further failures happen, amounts are repeatedly split until a minimum amount limit is reached or the payment succeeds. We opted for this relatively simple strategy to begin with. It is expected to address various basic liquidity problems that arose in the past, but it is unlikely to always execute payments in a way that is optimal. The field of multi-path planning is still mostly uncharted and contains lots of challenges. Therefore we hope to collect valuable feedback with this release and use that to drive further development. Multi-path payments can only be initiated through the `routerrpc.SendPayment` rpc. Payments will be single-shot by default unless the new parameter `max_shards` is set to a value greater than one. `max_shards` controls the maximum number of paths that will be used for the payment. The following `lncli` command can be used to make a multi-path payment: `lncli payinvoice --max_shards 5 ` Additionally, the payment limit has now be removed for calls to the _new_ MPP-enabled payment API. This allows users to now send payments that fully utilize all their outgoing payment bandwidth. ### Payment tracking Improvements have been made to provide more fine-grained information about the status of payments. A list of per-payment htlcs was already added in a prior release. This has now been [extended](https://github.com/lightningnetwork/lnd/pull/4000) with the exact failure reason for failed htlcs. Furthermore the payment status stream that is returned by `routerrpc.SendPayment` and `routerrpc.TrackPayment` now also [contains intermediate state updates](https://github.com/lightningnetwork/lnd/pull/3972). An example of this is the launch of an additional htlc. This will result in a state update on the stream. The proto message on these update streams has been aligned to match the main rpc `Payment` message. *This is a breaking change in the sub-server API.* ## Invoices ### Lifted Invoice Limit As part of the `mpp` changes, [the invoice limit has been lifted from 4.2 million satoshis](https://github.com/lightningnetwork/lnd/pull/4075). With this change, receivers will now be able to create invoices that can be used in tandem with `mpp` at the sender, to fully utilize the channel bandwidth at _both_ the sender and receiver to complete payments. ## Channel Commitments & Funding ### PSBT Funding It is now possible to [open channels by using a PSBT](https://github.com/lightningnetwork/lnd/pull/4079). This allows users to fund multiple channels in one on-chain transaction and/or open channels directly from a hardware wallet without the need for a transaction to `lnd`'s wallet first. An example of how to use this new feature in combination with a `bitcoind` [node can be found here](https://github.com/lightningnetwork/lnd/blob/master/docs/psbt.md). NOTE: The normal funding dialog between the initiator's node and the remote node is paused after the multisig keys for the funding transaction have been exchanged. It is therefore possible that the remote node will not wait indefinitely for the flow to continue and cancel the funding reservation after some time. `lnd` nodes will cancel after 10 minutes of inactivity for example. Future versions of `lnd` will expand the PSBT capabilities in order to allow generalized usage of the transaction packet format for all transaction crafting within the daemon. ### Anchor commitment format An new commitment format that includes [anchor outputs](https://github.com/lightningnetwork/lightning-rfc/pull/688) has been [added](https://github.com/lightningnetwork/lnd/pull/3821). Anchor outputs allow users to [increase the fee of their commitment transaction](https://github.com/lightningnetwork/lnd/pull/3758) after it has been published by making use of CPFP. The anchor output is [force-swept](https://github.com/lightningnetwork/lnd/pull/3809) in a child transaction that may be negatively yielding. It allows timely confirmation even if the miner fee of the commitment tx itself is insufficient. To do so, the `BumpFee` rpc can be used to increase the fee of the total package. For channels in the `waiting_close_channels stage`, the following `lncli` command can be issued. Note that `lnd` must be built with the `walletrpc` build tag for this command to be available. `lncli wallet bumpclosefee 2b66cf489298b5b4a9305127413e4124e385588178e4b0fc4cec2d28c277b17d:0 --sat_per_byte 15` The specified fee rate is the fee rate of just the child transaction. It needs to be chosen such that the fee rate of the total package including the commitment transaction is bumped to the desired level. To open a payment channel with anchors, both sides of the channel need to run `lnd` with `--protocol.anchors`. All newly opened channels between those peers will then automatically be of the anchor type. To facilitate identification, the channel type is now exposed on [`ListChannels`](https://github.com/lightningnetwork/lnd/pull/4068) and [`PendingChannels`](https://github.com/lightningnetwork/lnd/pull/4129). Be aware that anchor channels are experimental. They don't yet have the reliability track record of the regular commitment format and therefore carry more risk. The specification hasn't been finalized yet, so there is also still a chance of breaking changes in future releases. In that case, *lnd will likely require anchor channels to be closed before upgrading*. ## RPC Changes Because `lncli` now uses the `jsonpb` JSON marshaler to print RPC responses, the `json_name` hints in the proto files [are no longer necessary and have been removed](https://github.com/lightningnetwork/lnd/pull/3996). The proto files can now be formatted with the `make rpc-format` command, if the `clang-format` binary is installed. More importantly, [Travis CI now makes sure the proto files were compiled using the correct version](https://github.com/lightningnetwork/lnd/pull/4042) of the protobuf compiler. `routerrpc` is [no longer conditionally compiled](https://github.com/lightningnetwork/lnd/pull/4128) and from now on available in any build. `lncli` is also switched over to use the payment rpcs of `routerrpc`. The asynchronous `SendPayment` call on the main rpc has been deprecated. The `--reversed` flag for the `lncli listinvoices` [command has been removed in favor of a new flag: `--paginate-forwards`](https://github.com/lightningnetwork/lnd/pull/4003) in order to make the functionality easier to use. [The `ListPayments` endpoint now supports _pagination_](https://github.com/lightningnetwork/lnd/pull/3960). The exposed API is similar to the existing pagination support for `ForwardingHistory`. On the command-line, users can set the `index_offset` and `max_payments` values to seek around within their past payment history. ## `bitcoind` Support [A new config option has been added to allow the user to select which fee estimation modifier](https://github.com/lightningnetwork/lnd/pull/4078) (the confidence parameter) we use when requesting fee estimates from `bitcoind`. users can now select from `CONSERVATIVE` and `ECONOMICAL` using the `--bitcoind.estimatemode` config setting. ## Documentation An [Operational Safety Guidelines document](https://github.com/lightningnetwork/lnd/pull/3963) has been added that advises users on how to safely operate a mainnet `lnd` node. ## Tor [We now support authenticating to tor via the `HASHEDPASSWORD` authentication mechanism](https://github.com/lightningnetwork/lnd/pull/4048). This allow users to delpoy `lnd` with `tor` enabled, in configruation where `lnd` may not have access to the filesystem that the `tor` daemon uses. ## Watchtowers Watchtowers now have support for [automatically setting up a Tor hidden service](https://github.com/lightningnetwork/lnd/pull/4087). This supports both v2 and v3 hidden services and can either be specified on the command line or via config file. Instructions on how to do this can be found [here](https://github.com/lightningnetwork/lnd/blob/master/docs/watchtower.md#tor-hidden-services). ## Testing+Security `lnd` has a new `fuzz` package which contains fuzzing harnesses for the `lnwire`, `wtwire`, and `brontide` packages. The harnesses for `lnwire` and `wtwire` check that (de)serialization works as expected. The `brontide` harnesses test the validity of various states of the initial brontide handshake. Documentation on how to get started with fuzzing `lnd` can be found [here](https://github.com/lightningnetwork/lnd/blob/master/docs/fuzz.md). ## Miscellaneous To make automated unlocking of `lnd` nodes easier, the `lncli unlock` [now accepts the password from the standard input](https://github.com/lightningnetwork/lnd/pull/4066), if the `--stdin` flag is passed. NOTE: This is considered to be unsafe/dangerous if the password is located in a file that can be read by another user. The `--stdin` flag should only be used in combination with some sort of password manager or secrets vault. ## Bug fixes * The `channel.backup` file is now also [updated while a channel is still pending](https://github.com/lightningnetwork/lnd/pull/3993). Previously a channel was only added to the SCB file once it was fully confirmed on-chain. * The JSON marshaler used by the REST gateway by default omitted "falsey" values (values that evaluate to `false` in JSON/JavaScript, like `0`, `""`, `false`). This was especially confusing/incorrect with enums. because the whole field was omitted if the first enum option was set (because that defaulted to `0` in the underlying gRPC transport). This [behavior was fixed in the REST marshaler](https://github.com/lightningnetwork/lnd/pull/3965). * A bug that would at times [cause persistently high CPU usage with certain configurations using `bitcoind`](https://github.com/lightningnetwork/lnd/issues/3370) has been resolved. * Channel updates received from the remote party are now [stored on disk](https://github.com/lightningnetwork/lnd/pull/3872) until we've produced a signature that includes these updates. Previously the updates were only stored in memory and dropped across restarts. This could lead to the production of an invalid signature and channel force closure. * A current limitation of the Lightning protocol is that it is possible for the channel initiator to reach a balance that is so close to the reserve, that it is no longer possible to send or receive htlcs with a non-dust value. The internal available channel balance calculation [has been refined](https://github.com/lightningnetwork/lnd/pull/3691) to take this possibility into account and report a lower balance if necessary to prevent the channel from reaching a degraded state. * Node-level failures are now [penalized harder](https://github.com/lightningnetwork/lnd/pull/3945) in mission control. This prevents routes that are unlikely to succeed from being explored and thereby speeds up the payment process. * An erroroneous log message [has been nixed](https://github.com/lightningnetwork/lnd/pull/4107). Previously, this message would be printed upon each successful payment and caused some confusion for users when nothing was actually wrong. The full list of changes since `v0.9.2-beta` can be found here: * https://github.com/lightningnetwork/lnd/compare/v0.9.2-beta...v0.10.0-beta # Contributors (Alphabetical Order) Andras Banki-Horvath bitromortac Carla Kirk-Cohen Carsten Otto chokoboko Conner Fromknecht Craig Hammell Danny Paz Ediz Turcan elkanatovey ErikEk Eugene Siegel fguisso Hampus Sjöberg Johan T. Halseth John Newbery Joost Jager MrManPew Olaoluwa Osuntokun Oliver Gugger Roei Erez Tomas Carnecky Wilmer Paulino Yaacov Akiba Slama yancy yyforyongyu