lnd 0.11.0-beta.rc1

This marks the first major release in the `v0.11.0` series! # Verifying the Release In order to verify the release, you'll need to have `gpg` or `gpg2` installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already: ``` curl https://keybase.io/bitconner/pgp_keys.asc | gpg --import ``` Once you have the required PGP keys, you can verify the release (assuming `manifest-v0.11.0-beta.rc1.txt` and `manifest-v0.11.0-beta.rc1.txt.sig` are in the current directory) with: ``` gpg --verify manifest-v0.11.0-beta.rc1.txt.sig ``` You should see the following if the verification was successful: ``` gpg: assuming signed data in 'manifest-v0.11.0-beta.rc1.txt' gpg: Signature made Wed Jul 29 21:21:50 2020 PDT gpg: using RSA key 9C8D61868A7C492003B2744EE7D737B67FA592C7 gpg: Good signature from "Conner Fromknecht " [ultimate] ``` That will verify the signature of the manifest file, which ensures integrity and authenticity of the archive you've downloaded locally containing the binaries. Next, depending on your operating system, you should then re-compute the `sha256` hash of the archive with `shasum -a 256 `, compare it with the corresponding one in the manifest file, and ensure they match *exactly*. ## Verifying the Release Binaries Our release binaries are fully reproducible. Third parties are able to verify that the release binaries were produced properly without having to trust the release manager(s). See our [reproducible builds guide](https://github.com/lightningnetwork/lnd/tree/master/build/release) for how this can be achieved. The release binaries are compiled with `go1.14.6`, which is required by verifiers to arrive at the same ones. They include the following build tags: `autopilotrpc`, `signrpc`, `walletrpc`, `chainrpc`, `invoicesrpc`, `routerrpc`, and `watchtowerrpc`. Note that these are already included in the release script, so they do not need to be provided. The `make release` command can be used to ensure one rebuilds with all the same flags used for the release. If one wishes to build for only a single platform, then `make release sys= tag=` can be used. Finally, you can also verify the _tag_ itself with the following command: ``` git verify-tag v0.11.0-beta.rc1 ``` # Building the Contained Release Users are able to rebuild the target release themselves without having to fetch any of the dependencies. In order to do so, assuming that `vendor.tar.gz` and `lnd-source-v0.11.0-beta.rc1.tar.gz` are in the current directory, follow these steps: ``` tar -xvzf vendor.tar.gz tar -xvzf lnd-source-v0.11.0-beta.rc1.tar.gz GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.11.0-beta.rc1" ./cmd/lnd GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.11.0-beta.rc1" ./cmd/lncli ``` The `-mod=vendor` flag tells the `go build` command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory. Additionally, it's now possible to use the [enclosed `release.sh` script to bundle a release for a _specific_ system like so](https://github.com/lightningnetwork/lnd/pull/2191): ``` make release sys="linux-arm64 darwin-amd64" ``` ⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️ # Release Notes WIP, release will be up shortly! ## Wumbo Channel Support [This version of `lnd` supports accepting and creating wumbo channels!](https://github.com/lightningnetwork/lnd/pull/4429) A wumbo channel is a channel whose total capcity is greater than `0.16 BTC` (the existing channel limit). Activating wumbo channel features is _opt in_ meaning users will need to set a name flag (`--protocol.wumbo-channels`) in order to use it. Most users can likely get by w/o wumbo channels, but larger nodes or exchanges/services may really benefit from the ability to manage a fewer set of larger channels. As is, there's no limit w.r.t the size of a channel that a node will accept. However, nodes may want to limit the size of channel they'll accept in order to limt their exposure. This can be done by using a [`ChannelAcceptor` call back](https://api.lightning.community/#channelacceptor) to _reject_ all in-bound funding requests greater than a certain value. [To coincide with this feature, the existing payment limits have been lifted in the payment related RPC calls](https://github.com/lightningnetwork/lnd/pull/4335). The max payment is now bounded by what the reciever can receive, and what the sender can route over using MPP. ## New Autopilot Heuristic [In this version of `lnd`, we've added a new autopilot heuristic that is more robust and resistant to manipulation compared to the existing preferential attachment heuristic](https://github.com/lightningnetwork/lnd/pull/4384). This new heuristic is based on the concept [betweenness centrality](https://en.wikipedia.org/wiki/Betweenness_centrality). The betweenness centrality metric is a essentially a measure of the number of shortest paths that pass through a particular node/vertex. The new heuristic uses this value rather than the relative number of channels a node has to make attachment (channel creation) decisions. This new heuristic should be more effective for both clients and routing nodes as as it's more indicative of a nodes potential as a routing node than simply the number of channels it has. This new mode can be activated by passing a new argument: `--autopilot.heuristic=top_centrality:1.0` to lnd. Note that it's possible to blend the old and new heuristics by specifying a another linear combination of heuristics with their total weight summing up to zero. The current heuristic still maintains the stochastic nature of the existing attachment heuristics. In later releases, we aim to improve on this, by doing a mini simulation to identify the best `N` new channels a node can add to maximize the gain of its betweenness centrality metrics. Once this is complete, then this metric will actually attempt to actively better position a potential routing node in the network, rather than try to connect to the node w/ the "best" value for a given metric. ## Experimental Support for new Storage Backend: `etcd` [In this release of `etcd`, we've added experimental support for `etcd` as a new storage backend for `lnd`](https://github.com/lightningnetwork/lnd/pull/4015). [`etcd`](https://etcd.io/) differs from `bbolt` (the current default storage backend) in that it's a replicated database which can be run in a clustered manner. The main advantage over `bbolt` is that the replicated nature eliminates the current single point of failure that is `channel.db`. When activated, `lnd` will store data that can be reconstructed easily locally in `bbolt` (such as the channel graph), while critical data such as the current state of a channel will be replicated and stored in `etcd`. As this mode is still experimental, in order to test out this new DB mode, users will need to compile `lnd` with the new `kvdb_etcd` build tag: `make install tags=kvdb_etcd`. [Before running `etcd`, users should check out the new documentation for information concerning the recommended `etcd` configuration set up](https://github.com/lightningnetwork/lnd/blob/master/docs/etcd.md). Future releases will further optimize our usage of the new database backend, eventually adding support to run `lnd` in a _clustered_ mode by by using `etcd`'s leader election APIs. ## Protocol/Spec Compliance ### Sporadic Force Close Issue Resolution This release fixes three distinct cases of erroneous force closes. All were related to the code that handles restarting channels, so nodes would regularly hit this if one of them flapped. Nodes should see increased stability across the network with this release. - https://github.com/lightningnetwork/lnd/pull/4430 - https://github.com/lightningnetwork/lnd/pull/4431 - https://github.com/lightningnetwork/lnd/pull/4432 ### `ChannelAnnouncement` Feature Bit Decoding [We'll now properly include the feature bits of a `ChannelAnnouncement` in the wire messages we send out](https://github.com/lightningnetwork/lnd/pull/4342). ### Short Channel ID Decoding [A bug has been fixed that would otherwise cuase `lnd` incorrectly decode a zero value short channel ID](https://github.com/lightningnetwork/lnd/pull/4391). ## Support for Dynamic IPs and Domain Resolution for Node Addrs [In this version of `lnd`, a new flag has been added to allow users that either have a dynamic IP or want to advertise the resolved IP address of a domain name: `--external-hosts`](https://github.com/lightningnetwork/lnd/pull/4281). This new flag can be set to a series of external hosts. If this flag is set, then `lnd` will peridically attempt to resolve the target host, advetising the new IP address. We'll also keep track of the prior IPs reported by that domain name, and will send out a new advertisement with the most up to date IP addresses if we detect a change. ## RPC Enhancement and Bug Fixes ### New External Coin Selection API [In this release, we've added a new set of calls to the `WalletKit` sub-server that allows clients to perform custom coin selection](https://github.com/lightningnetwork/lnd/pull/4303). Along-side the existing `SignRPC` sub-server calls, [this set of APIs allows clients to manually construct, sign and broadcast custom transactions](https://api.lightning.community/#service-walletkit). This may also be combined with the new PSBT APIs to carry out complex multi-party protocols. Three new RPC calls have been added: * `ListUnspent`: a copy of the existing `ListUnspent` RPC in `LightningClient`, which has been marked as deprecated in favor of using the `walletrpc` one. * `LeaseOutput`: lease an output for a predetermined amount of time bound to a unique ID to ensure no other applications can lease the same output. * `ReleaseOutput`: if the lease for an output is no longer needed, it can be returned to the set of spendable UTXOs manually through this RPC, or it will be done automatically by the wallet once the lease expires. ### New HTLC Interception API [This new release of `lnd` contains a new RPC API which is essentially the forwarded HTLC analgogue of the existing "hodl" invoice API](https://api.lightning.community/#htlcinterceptor). [This new API allows for _custom_ forward policies to be implemented](https://github.com/lightningnetwork/lnd/pull/4018). With the new API, for each forwarded HTLC, a call-back implemented as a streaming RPC will be called. With that call back, users can either signal they want the HTLC to immediately be settled, held, or instantly failed. This new API allows for a number of cool use cases such as notifying off-line clients of a new incoming payment (HTLC mailbox) and also opening a channel to a client in order to satisfy an incoming HTLC forwaard event. This new API also allows user to examine any custom records which may have been dropped off to intermediate nodes by the sender. This intermedaite node custom records have no "official" protocl use, but some interesting meta-protocol are possible with its inclusion. ### Hodl Keysend Payments [With this new version of lnd, we've added a new option to `lnd` to hold received keysend payments in the accepted state for a configured period of time](https://github.com/lightningnetwork/lnd/pull/4167). During this time, an application can inspect the payment parameters and decide whether to cancel or settle this payment. Example: a keysend payment with an embedded order comes in. The payment is held and an external application checks that the paid amount is sufficient for the ordered goods. If not, the payment is canceled without the need to refund anything. If the amount is sufficient, the payment is settled and the order processed. This PR makes development of applications like [tlvshop.com](https://tlvshop.com/) possible without forking `lnd`. Users can use the existing `SettleInvoice` call to manually settle inbound keysend payments if the `--keysend-hold-time` flag is set. ### Updated Close Summary The `CloseChannelSummary` returned by the `ClosedChannels` endpoint has been updated to [include a list of on chain actions](https://github.com/lightningnetwork/lnd/pull/4157) that were taken to resolve channels that were force closed. The on chain actions taken depend on the type of commitment the channel had, and the party that force closed. * Commit sweep: Present if our node force closed the channel, represents sweeping of our own balance, which is behind a timelock for local force closes, back to our own wallet. * Incoming HTLC: Incoming htlcs offer to pay your node funds in exchange for reveal of a preimage. If your node had knowledge of the preimage, these outputs will reflect as having been claimed, which means your node swept the funds to itself. If your node force closed, this claim will have two on chain resolutions, a success transaction followed by a sweep. Otherwise, the remote party will have timed out the htlc. * Outgoing HTLC: Outgoing htlcs are offers that you make to pay a node in exchange for a preimage. If the remote party has knowledge of the preimage, they htlc will reflect as having been claimed, and the funds are swept by the remote party. If they do not provide the preimage before the timeout, our node will sweep the funds back to itself, and the htlc will reflect as having been timed out. If your node force closed, this timeout will have two on chain resolutions, a timeout transaction followed by a sweep. * Anchor: Present if the channel was created using the experimental anchors commitment format. These outputs can either be swept back to the wallet by our node, or claimed by another party if we did not sweep it within 16 blocks. ### New List Sweeps API A new `ListSweeps` endpoint that [provides a list of all the on chain transactions that lnd has published to sweep funds back to its own wallet](https://github.com/lightningnetwork/lnd/pull/4139) has been added to the walletrpc subserver. By default, the endpoint returns a list of transaction IDs. The `verbose` option can be used to provide full transaction details, identical to those provided in the existing `GetTransactions` endpoint. ### Height Parameters for GetTransactions The `GetTransactions` endpoint was [updated to take a start and end height parameter](https://github.com/lightningnetwork/lnd/pull/4139), both values are inclusive. If the start height provided is greater than the end height, transactions will be listed in reverse order (from highest to lowest block). To include unconfirmed transactions in your query, the end height should be set to -1, and a list of transactions from start height to chain tip will be returned. This option is set by default to maintain backwards compatibility with the old API. ### Transaction Labelling On chain transactions broadcast by LND [can now be labelled with strings](https://github.com/lightningnetwork/lnd/pull/4213) up to 500 characters long. These labels can be added on transaction creation when using the various on chain sending APIs, or [retrospectively using the `LabelTransaction` endpoint](https://github.com/lightningnetwork/lnd/pull/4228) in the walletkit. If a transaction already has a label, you will need to explicitly set an overwrite flag.These labels are visible in the output of `GetTransactions`. ## Payments Indexing Payments in lnd are uniquely identified by a payment index which is auto-incremented as we dispatch payment attempts. Previously, calls to `ListPayments` allowed pagination using these indexes, but required that we fetch all payments from the database and sift through them to find payments lying in the correct range. This caused performance issues for nodes with large numbers of payments. This release adds [an index by sequence number](https://github.com/lightningnetwork/lnd/pull/4261) which allows efficient lookup of payments in a range. The pagination API itself has not changed, but node operators with a lot of payments can expect a performance increase. ## Multi-Channel Outgoing Payment Restriction [The payment RPCs now support sticking to a set of specified channels during path finding](https://github.com/lightningnetwork/lnd/pull/4257). This allows for more fine-grained path contorl, as if one want sto rebalance a specific set of channels, they can use this along with the existing allow/deny lists to force lnd to take a specific route. ### REST API With a series of pull requests the REST API has been overhauled quite notably. First of all, [REST is now enabled on all (experimental) subservers](https://github.com/lightningnetwork/lnd/pull/4251). This allows users of the REST API to make use of the new MPP enabled payment calls ([`Router.SendPaymentV2`](https://github.com/lightningnetwork/lnd/blob/master/lnrpc/routerrpc/router.proto#L17)) for example. Then, to make the many server-streaming RPC calls available to REST users, a [WebSocket proxy was added](https://github.com/lightningnetwork/lnd/pull/4141) that transforms streaming responses from single line JSON strings on a keep-alive HTTP response into proper WebSocket messages. Our [online API documentation](https://api.lightning.community) now also shows a [JavaScript example of how to use the WebSockets](https://api.lightning.community?javascript#v2-router-send). To make our RPC protobuf definitions easier to use and compile by other projects, we removed the Google annotations from our `rpc.proto` file and instead [extracted all REST definitions into the file `lnrpc/rest-annotations.yaml`](https://github.com/lightningnetwork/lnd/blob/master/lnrpc/rest-annotations.yaml). ### Bug Fixes [We'll now properly parse both possible `KeyDescriptor` fields for all calls to the `signrpc` sub-server](https://github.com/lightningnetwork/lnd/pull/4356). [While the server hasnt' yet fully started, we'll now omit uptime information in `ListChannels` as we wouldn't have connected out to any peers by then.](https://github.com/lightningnetwork/lnd/pull/4176) [Pagination for the ListInvoices was updated to return invoices when called with the reversed flag set, and the last index offset was greater than the number of invoices on disk.](https://github.com/lightningnetwork/lnd/pull/4261/commits/eea871b5831b01c77c27d4f37e99f67217a3f2d8) Previously, this endpoint would return no invoices if the API was called with the reversed flag and did not have the precise number of invoices as its offset. ## `htlcswitch` Improvements [We'll now log each time an HTLC is succesfully forwarded on the least verbose logging level (info)](https://github.com/lightningnetwork/lnd/pull/3018). This allows users to monitor their logs to detect sueccesfully forwarded HTLCs. [Payments initiated by the routing will now be directly handed to the link, rather than going through the switch first](https://github.com/lightningnetwork/lnd/pull/4183). This results in lower latency for MPP payments, and also ensures that a link is able to actually carry the HTLC before the router moves onto the next payment shard. ## AMP Progress As a stepping stone towards enabling Atomic Multi-path Payments, `0.11.0` [now indexes all newly created invoices by `payment_addr`](https://github.com/lightningnetwork/lnd/pull/4285) in addition to the typical `payment_hash` index. Unlike MPP, the `payment_hash` of AMP HTLCs is not known to the sender at the time of invoice creation. Additionally, the `payment_hash` of each HTLC is expected to be unique, which creates an n-to-1 mapping of `payment_hash` to invoice. In the future, `payment_addr` will serve as a universal identifier for both MPP and AMP invoices, as it provides a consistent 1-to-1 mapping for both payment types. The remaining pieces of the AMP saga are expected to appear in a later release, at which point users will be able to query and/or interact with invoices via RPC using `payment_addr` as the primary identifier. As of now, the plan is to continue to support RPC interactions via `payment_hash` for MPP invoices as is done today. ## Build process Writing software that compiles and runs on all major operating systems and hardware architectures has become increasingly involved. Our Continuous Integration platform Travis only lets us run a certain number of parallel tasks which limited what we could automatically compile and test on each new commit. [Offloading some of those tasks to run as GitHub Workflows](https://github.com/lightningnetwork/lnd/pull/4220) has allowed us to add cross-compilation checks for all major operating systems. [Travis now only runs a very basic sanity check and then the integration tests](https://github.com/lightningnetwork/lnd/pull/4482). Splitting the work between GitHub and Travis also freed up two build slots where we now run [integration tests on ARM](https://github.com/lightningnetwork/lnd/pull/4408) and [on Windows](https://github.com/lightningnetwork/lnd/pull/4475). A [number of small improvements to our integration tests in general](https://github.com/lightningnetwork/lnd/pull/4344) reduced the flakiness of those tests significantly. Production binaries built from source via `make install` are now stripped of DWARF symbols and are [noticeably smaller](https://github.com/lightningnetwork/lnd/pull/4262). ## Full-Node Compatability Updates [This new version of `lnd` now properly supports `bitcoind v0.20`](https://github.com/lightningnetwork/lnd/pull/4366). [The latest `master` branch of `btcd` is now also supported as well](https://github.com/lightningnetwork/lnd/pull/4291). ## Safety Improvements ### SCB Recovery This release of `lnd` fixes an existing bug that would at times replace the on-disk SCB state with fresh information, if `lnd` was started in SCB recovery mode (a clean slate) with an _existing_ SCB file on disk. [With this new version of `lnd`, we'll now always combine our in memory state with what's on disk, ensuring that we never override the information on disk](https://github.com/lightningnetwork/lnd/pull/4379). [We'll also no longer insert a recovered SCB channel into the graph database](https://github.com/lightningnetwork/lnd/pull/4370). This resolves a number of edge cases, and also makes the recovery process much simpler. ### PSBT Channel Funding The recently introduced PSBT channel funding flow was found to be risky when used to open multiple channels in one batch transaction. It can happen that the funding transaction is published too early and some of the channel funds get stuck. To mitigate that risk, we [added the `--no-publish` flag to the PSBT funding flow](https://github.com/lightningnetwork/lnd/pull/4455). ### On-Chain Contract Handling [We'll now properly clean up local state within the database after a local channel breach](https://github.com/lightningnetwork/lnd/pull/3892). - Add `--dry-run-migration` config option [#2668](https://github.com/lightningnetwork/lnd/pull/2668), which will prevent `lnd` from committing migrations on startup. This provides two useful features: - Avoids unexpected migrations that would otherwise prevent downgrading to a prior database version. - Allows users to aid in testing migrations for correctness and to guage performance on real-world databases. ## Refactoring This release sees the addition of the [`lnwallet/chancloser`](https://github.com/lightningnetwork/lnd/pull/4365) and [`peer`](https://github.com/lightningnetwork/lnd/pull/4347) packages. This modularization will help with testing now that these components are properly isolated and can be imported into other packages. ## Bug fixes - Fixed sporadic `tls: internal error: failed to clone hash` error that [was caused by a faulty library import](https://github.com/lightningnetwork/lnd/pull/4300). - Added a sanity check to the Tor configuration options that [disallows setting Tor socks and control port to the same value](https://github.com/lightningnetwork/lnd/pull/4207). - Fixes several `subscribeinvoice` bugs [#4298](https://github.com/lightningnetwork/lnd/pull/4298): - Backlogs are now delivered properly even if no invoices have been settled. - Two separate race conditions that may have lead to duplicate notifications are now resolved. - Fixed properly [annotating certain classes of closed channels in the database](https://github.com/lightningnetwork/lnd/pull/4361) - [Fixed a panic that coul darise if a notification of the `ChainNotifier` sub-server is cancelled as a new block arrives](https://github.com/lightningnetwork/lnd/pull/4355) - [We'll now log the full raw TX hex of any broadcasted funding transactions](https://github.com/lightningnetwork/lnd/pull/4341) - [A bug has been fixed to prevent a panic when accepting an incompatible output index in the wallet](https://github.com/lightningnetwork/lnd/pull/4302) - [A bug that could arise when a client switches between multiple towers has been fixed](https://github.com/lightningnetwork/lnd/pull/4274) - [We'll now ignore reconnection for unadvertised inbound peers over Tor](https://github.com/lightningnetwork/lnd/pull/4194) - [Bad features will no longer cause path finding to fail early](https://github.com/lightningnetwork/lnd/pull/4327) The full list of changes since `v0.10.0-beta` can be found here: * https://github.com/lightningnetwork/lnd/compare/v0.10.0-beta-branch...v0.11.0-beta.rc1 # Contributors (Alphabetical Order) Andreas M. Antonopoulos András Bánki-Horváth Anthony Ronning Bjarne Magnussen Brian Mancini CandleHater Carla Kirk-Cohen Conner Fromknecht cryptagoras ErikEk Eugene Siegel fguisso Gabriel Comte Jake Sylvestre Jasper Rädisch Johan T. Halseth Joost Jager Markus Fix Matheus Degiovani Olaoluwa Osuntokun Oliver Gugger Roei Erez Sebastian Falbesoner vctt94 Wilmer Paulino yyforyongyu