This release marks the first major release in the 0.13 series, and the second major release of the year! This release includes a number of compelling additions including: first-class pruning support, AMP sending+receiving support, arbitrary pubkey/xpub import w/ PSBT transaction crafting, clustered lnd using etcd failover, and much more!
Database Migrations
The lnd database is migrated to store all wire messages with an additional TLV field. See details below.
Verifying the Release
In order to verify the release, you'll need to have gpg or gpg2 installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already:
$ curl https://keybase.io/bitconner/pgp_keys.asc | gpg --import
$ curl https://keybase.io/roasbeef/pgp_keys.asc | gpg --import
Once you have the required PGP keys, you can verify the release (assuming manifest-roasbeef-v0.13.0-beta.sig and manifest-v0.13.0-beta.txt are in the current directory) with:
$ gpg --verify manifest-roasbeef-v0.13.0-beta.sig manifest-v0.13.0-beta..txt
You should see the following if the verification was successful:
gpg: Signature made Wed Sep 30 17:35:20 2020 PDT
gpg: using RSA key 4AB7F8DA6FAEBB3B70B1F903BC13F65E2DC84465
gpg: Good signature from "Olaoluwa Osuntokun <laolu32@gmail.com>" [ultimate]
That will verify the signature of the manifest file, which ensures integrity and authenticity of the archive you've downloaded locally containing the binaries. Next, depending on your operating system, you should then re-compute the sha256 hash of the archive with shasum -a 256 <filename>, compare it with the corresponding one in the manifest file, and ensure they match exactly.
Verifying the Release Timestamp
From this new version onward, in addition time-stamping the git tag with OpenTimestamps, we'll also now timestamp the manifest file along with its signature. Two new files are now included along with the rest of our release artifacts: manifest-roasbeef-v0.13.0-beta.txt.asc.ots.
Assuming you have the OpenTimestamps client installed locally, the timestamps can be verified with the following commands:
$ ots verify manifest-roasbeef-v0.13.0-beta.sig.ots -f manifest-roasbeef-v0.13.0-beta.sig
Alternatively, the open timestamps website can be used to verify timestamps if one doesn't have a bitcoind instance accessible locally. These timestamps should give users confidence in the integrity of this release even after the key that signed the release expires.
Verifying the Release Binaries
Our release binaries are fully reproducible. Third parties are able to verify that the release binaries were produced properly without having to trust the release manager(s). See our reproducible builds guide for how this can be achieved.
The release binaries are compiled with go1.16.3, which is required by verifiers to arrive at the same ones.
They include the following build tags: autopilotrpc, signrpc, walletrpc, chainrpc, invoicesrpc, routerrpc, and watchtowerrpc. Note that these are already included in the release script, so they do not need to be provided.
The make release command can be used to ensure one rebuilds with all the same flags used for the release. If one wishes to build for only a single platform, then make release sys=<os-arch> tag=<tag> can be used.
Finally, you can also verify the tag itself with the following command:
$ git verify-tag v0.13.0-beta
gpg: Signature made Thu 17 Jun 2021 02:25:51 AM UTC using RSA key ID 9B280306
gpg: Good signature from "Olaoluwa Osuntokun <laolu32@gmail.com>"
Verifying the Docker Images
To verify the lnd and lncli binaries inside the docker images against the signed, reproducible release binaries, there is a verification script in the image that can be called (before starting the container for example):
$ docker pull lightninglabs/lnd:v0.13.0-beta
$ docker run --rm --entrypoint="" lightninglabs/lnd:v0.13.0-beta /verify-install.sh
$ OK=$?
$ if [ "$OK" -ne "0" ]; then echo "Verification failed!"; exit 1; done
$ docker run lightninglabs/lnd [command-line options]
Building the Contained Release
Users are able to rebuild the target release themselves without having to fetch any of the dependencies. In order to do so, assuming
that vendor.tar.gz and lnd-source-v0.13.0-beta.tar.gz are in the current directory, follow these steps:
$ tar -xvzf vendor.tar.gz
$ tar -xvzf lnd-source-v0.13.0-beta.tar.gz
$ GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.13.0-beta" ./cmd/lnd
$ GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.13.0-beta" ./cmd/lncli
The -mod=vendor flag tells the go build command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.
Additionally, it's now possible to use the enclosed release.sh script to bundle a release for a specific system like so:
$ make release sys="linux-arm64 darwin-amd64"
⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️
Release Notes
Database and wire changes
Wire Message TLV Support
The wire messages sent between LN peers have the ability to carry additional data, using the TLV format. This allows attaching data and protocol extensions to messages in a non-breaking way, paving the way for future feature upgrades to the protocol.
In this release all stored messages in the lnd database are migrated to a format supporting these TLV extensions, and message parsing now always read out these fields and keep them for future handling.
Protocol Updates
Anchor Output Channels
The spec compliant anchor channel format introduced in v0.12 is now the default channel type if both nodes support it when opening a new channel. You can read more about it in the v0.12.0 release notes, and it can be disabled by providing the --protocol.no-anchors flag at startup.
Since a node having channels using this format must keep on-chain funds around in case unilateral fee bumping is needed, we reserve 10k sats per channel for this purpose. In this release we cap this at 100k sats, and in addition avoid reserving this value for private channels.
Finally a change to breach handling has been made, to mitigate a theoretical attack the channel peer can perform by pinning HTLC outputs on a breached commitment transaction. If we suspect such pinning is taking place, lnd will now attempt to sweep the breached commitment outputs separately.
P2P Gossip Handling, Hardening & Optimizations
Ephemeral & Persistent Gossip Reject Caches
Those that run larger lnd instances may have noticed a cyclic nature of gossip traffic that would lead to many announcements being rejected, only to be processed hours later. This burst of traffic typically causes high CPU and memory usage, along with a large batch of blocks fetched from the node backend.
During this release cycle we dug into the issue and found that the culprit was actually a new variant of zombie channel churn: fully closed channels (funding UTXO spent) being continually offered by nodes that aren't (for some reason) pruning their channel graph.
In this release, we'll now add channels that fail full validation irrevocably to the existing zombie index cache. Expanding the usage of this cache means that new lnd nodes will only validate those spent channels once (as they should), then never request them again during the historical sync spot checks that happen periodically.
Related to the above fix, a bug has been fixed that would cause us to continually re-validate a channel announcement that we had already rejected.
Dependent Gossip Processing
ChannelUpdate and NodeAnnouncement gossip messages depend on its ChannelAnnouncement being processed successfully. Throughout our investigative efforts above, we also noticed that lnd would process these messages when their corresponding ChannelAnnouncement's validation failed, which has now been fixed.
Improved Gossip Rate Limiting
lnd v0.12.1-beta featured a new gossip rate limiting heuristic in which keep-alive updates were throttled to allow one per day, while non-keep-alive updates were throttled to allow one per block. The latter heuristic for non-keep-alive updates proved to be inconsistent, especially for our auto enable/disable channel behavior, as blocks are not guaranteed to arrive at a constant rate. To mitigate this, we've moved towards a token bucket based approach to allow by default one update per minute with a maximum burst tolerance of 10 updates. These defaults may change as the structure of the network does, but they can also be changed through two new config options: gossip.channel-update-interval and gossip.max-channel-update-burst.
Routing Optimizations & New Payment Types
Atomic Multi-Path (AMP) Payments
This version of lnd introduces initial support for AMP payments, which is a generalized version of the experimental keysend protocol added in lnd-v0.9.0-beta. While keysend payments must be fulfilled with a single HTLC, AMP offers the ability to perform multi-path spontaneous payments and leverage channel liquidity more effectively. It also offers an initial incarnation of AMP invoices, which helps to facilitate spontaneous payments to private nodes, and serve as a basis for recurring payments down the road.
Receiving Spontaneous Payments
Users can set the accept-amp=1 configuration option to automatically accept incoming, spontaneous AMP payments. Without setting this option, all spontaneous AMP payments will be rejected. Note that this setting is not required to accept one-time payments to an AMP Invoice.
Sending Spontaneous Payments
Users can send spontaneous AMP payments to nodes with accept-amp=1 using lncli, routerrpc.SendToRouteV2, or routerrpc.SendPaymentV2.
When paying with lncli: by including the --amp flag, payments only need to specify the amount and pubkey of the destiation, e.g. lncli sendpayment --amt 100 --dest 0247a5b3b2f6794da6ab930974e3396fd9973aa7dc333ea8dd3b59fb71a70165be --amp. This will only work for nodes in public channel graph, to pay private nodes see below regarding AMP Invoices.
When paying withrouterrpc.SendPaymentV2: when manually specifiying arguments, setting the Amp flag to true and omitting the PaymentHash will cause lnd to initiate an AMP payment. This behavior can also be forced by providing an AMP Invoice in the PaymentRequest field.
When paying with routerrpc.SendToRouteV2: users can force an AMP payment by setting AmpRecord field to a well crafted lnrpc.AmpRecord on the route's final lnrpc.Hop.
AMP Invoices
In this version of lnd, we introduce a new concept of AMP invoices. Having invoices at all may seem counterintuitive to the spontaneous use case of AMP, however AMP invoices will provided two necessary use cases:
- Offering the ability to pay private nodes via AMP, by providing the sender with hop hints to reach the desination.
- In the future, the ability to make recurring payments to an invoice, e.g. subscriptions, deposits/withdrawals, etc.
Full-blown recurring invoices are not implemented in this release, but will be available at a later point. For now an AMP invoice can only be settled by one payment. That said, this release does include support for simulating recurring payments, making the existing AMP invoices "pseudo-reusable".
In order to do so, users can make payments via routerrpc.SendPaymentV2 by passing in the (AMP) PaymentRequest, and providing an randomly-generated PaymentAddr. The external PaymentAddr will causelnd to ignore the payment address provided in the invoice, and make a spontaneous payment using the rest of the parameters in the payment request. Note that the receiver must have accept-amp=1 set, otherwise these subsequent payments will be ignored.
The default expiry for AMP invoices is set 30 days. Users may wish to increase this duration using the Expiry field to further amortize the setup costs of obtaining a new invoice.
MPP+AMP Payment Splitting on by Default
In this version of lnd, MPP/AMP payment splitting is now activated by default. Prior versions of lnd required a caller to manually specify that they wanted their payment to possibly be split via the MaxShards RPC/CLI flag. The default number of attempted splits is now 16 (we'll attempt to split up to 16 times by default). In addition, a new MaxShardAmt flag has been added that allows the caller to control how large the largest split can be. This new flag can effectively be used to force lnd to adhere to a given maximum payment unit expressed in satoshis, which means lnd may split more frequently than before.
Strict Zombie Pruning
Typically lnd will mark a channel as a zombie when both of its edges aren't re-advertised within a two week period. This is typically referred to as a channel's heartbeat update. Analysis by Conner Fromknecht demonstrated that the channel graph itself could be shrunk by ~20% if a stricter pruning heuristic was adopted: pruning a channel once only a single edge fails the heartbeat check. The latest version of lnd has implemented this stricter pruning variant in an opt-in manner.
This new graph pruning variant is on by default for neutrino nodes as it was common for them to never actually receive a "double disable" of a channel's edges, meaning it would never actually remove certain closed channels from its channel graph. Routing nodes and nodes that frequently send outbound payments can activate the new pruning variant with a new flag:
--routing.strictgraphpruning. The new logic also requires that party that originally failed the heartbeat check to the be one that re-publishes the channel update in order for a prior zombie channel to be resurrected.
New M1 Release Target
This is the first release that includes binaries for the recently released Apple Silicon (M1) architecture! roasbeef has switched to using an M1 machine as his primary development machine over the past few months and hasn't reported any unsafe behavior. We encourage those switching to run lnd on an M1 machine to reach out to us if any abnormal behavior is detected.
For the 0.14 release, we aim to start to remove architectures for binaries included with the final packaged release that are either in the bottom tier of downloads, or have no downloads at all.
P2P Networking
A bug that wouldn't allow a node to listen on an IPv6 REST port when Tor is active has been fixed.
Wallet Enhancements
This release features several improvements that will enable lnd (in a future release!) to operate without the access of private keys within its process.
Watch-Only Key Import
The wallet is now able to import BIP-0049 and BIP-0084 accounts/public keys as watch-only through xpubs! This feature was made possible by the introduction of watch-only accounts to btcwallet, our internal wallet. This functionality is exposed through two new RPCs within the WalletKit sub-server: ImportAccount and ImportPublicKey.
Events (deposits/spends) for imported keys or keys derived from an imported account will only be detected by lnd if they happen after the import. Rescans to detect past events will be supported in a future release, so make sure to use a new unused wallet when using this functionality.
A new document was added that explains the account import and PSBT signing process.
ECDH Revocation Seed
The revocation seed is used to generate revocation secrets during the lifetime of a channel. The seed-generation code has been updated to no longer handle raw private keys, using ECDH to achieve this.
Auto-unlock wallet from password file
In automated or unattended setups such as cluster/container environments, unlocking the wallet through RPC presents a series of challenges. Usually, the password is present as a file somewhere in the container already anyway so writing an extra custom script for unlocking is error prone and tedious. Instead the new --wallet-unlock-password-file flag was added that can be used to read the password for an existing wallet from a file (or device or named pipe) to auto-unlock it.
Random Coin Selection
By default lnd's internal wallet uses a coin selection algorithm that always picks the largest UTXOs available first when crafting a transaction. This can lead to a wallet state that has many small and uneconomical UTXOs. To counteract that, a new --coin-selection-strategy flag was added that can be set to random instead of the default larges value to enable random coin selection.
Backend Enhancements & Optimizations
Daemon-Level Block Cache
lnd now maintains a global daemon-level block cache to prevent redundant GetBlock calls across all of its sub-systems. This cache can store up to 20MB worth of block data by default, and can be changed through the new config option blockcachesize.
bitcoind Pruned Node Support
lnd now supports connecting to pruned bitcoind backends! A new sub-system has been introduced that maintains connections to bitcoin peers to retrieve pruned blocks from and caches them. These peers are sourced from the peer list of the connected bitcoind instance, or from the getnodeaddress RPC otherwise. The number of connections (4 by default) can be configured through the new config option bitcoind.pruned-node-max-peers.
Neutrino Optimizations
The memory footprint used throughout the initial block header sync has been reduced by 50%.
The set of filter header checkpoints for the mainnet and testnet chains have been updated.
Neutrino Channel Validation
Compact filters retrieved from peers to determine whether the corresponding block contains any relevant transactions are now validated by checking whether they were constructed correctly.
lnd instances backed by Neutrino now have channel validation disabled by default. Channel validation includes obtaining the block a channel was included in and verifying the channel output remains unspent. This is an intensive process for light-clients as they don't maintain a block index and UTXO set. The routing.assumechanvalid config option, which disables channel validation, has now been deprecated. Channel validation can be re-enabled for Neutrino-backed lnd nodes with the new config option neutrino.validatechannels.
Neutrino Configuration
neutrino.broadcasttimeout allows users to specify how long they wish their broadcast attempts to last before giving up.
neutrino.persistfilters allows users to persist compact filters retrieved from the network. This will result in less bandwidth consumption at the expense of storage.
neutrino.validatechannels can be activated to enable channel validation. Note that light-clients are not well-equipped to perform this type of validation without SPV channel proofs (keep an eye out for this in future releases!).
In lnd v0.12.0 we saw a massive enhancement to graph sync, since batch writing of gossip messages to the database was introduced. However, this also included local updates made, which could cause these to become slightly delayed. In this release the local updates will now get special treatment, and are added immediately.
Clustered LND configuration with leader elected primary
So far lnd has mainly been used in single node setups with local databases. This configuration makes it challenging to build a highly available lighting service, since the node itself has to be online and users have to build ad-hoc solutions for database replication and re-deployment in case of failure or datacenter migration.
With the recent addtion of the etcd kvdb driver for lnd we've made incremental steps toward an out-of-the box HA solution.
With this release we add experimental support for leader elected lnd cluster, where apart from the graph, all important state can reside in a replicated etcd database. This allows us to use etcd's leader election facilities to form a cluster of multiple lnd nodes, where one of them is the primary node accessing the DB, while the rest are waiting in line to become the leader if a failover happens (eg. primary is decomissioned, crash, network partition, etc). In such configuration all the nodes in the cluster share the same lightning identity but only the active leader has full control while the rest are dormant.
RPC
Unification of WalletUnlocker and Lightning services
This release brings a massive overhaul to the way the wallet is unlocked on the backend, while staying backwards compatible.
Earlier version of lnd would bring up a gRPC server that exposed the WalletUnlocker service, which would in turn be stopped and restarted to bring up the Lightning service after successful unlock. This caused issues with some clients that didn't expect the available services to change during runtime, and it also made it hard to determine the state of the wallet without string matching on the error received from calling the RPC server.
In lnd v0.13.0, the two gRPC services (and subserver services) are available at all times such that we avoid tearing down the gRPC server at runtime. The RPC endpoints are gated by the wallet state (see below) such that only RPCs from the WalletUnlocker service are available as long as the wallet remains locked.
If you relied on the RPC errors or availability of the different services to determine the wallet state, you should change to use the new State service.
New State Service
A new service is now exposed on the lnd gRPC server: State. This can be used to get the state of the wallet at all times, and should be used to programatically drive the unlocking process.
The possible wallet states are [NON_EXISTING, LOCKED, UNLOCKED, RPC_ACTIVE] and the state can be subscribed to using the SubscribeState RPC, or fetched using lncli state.
Mission Control
Mission Control's parameters can now read and set while lnd is running using the GetMissionControlConfig and SetMissionControlConfig endpoints. This allows nodes to experiment with these parameters without restarting lnd.
An experimental XImportMissionControl API has been added to allow nodes to import state from other lnd nodes' QueryMissionControl output. This allows new nodes to start include routing insights that another node has learned, rather than starting with no information about the network. This imported information is not currently persisted. Note that sharing your QueryMissionControl output with another party may leak information about your payment history.
Bi-directional Stream Support for REST WebSockets
A fix to the WebSocket proxy now allows all RPCs to be used over REST/WebSockets, including the client-streaming ones (specifically lnrpc.Lightning.ChannelAcceptor, lnrpc.Lightning.SendPayment and routerrpc.Router.HtlcInterceptor).
An example for using the channel acceptor over REST was added to the documentation.
Option to delete failed payments
In earlier versions of lnd you could delete all finished payments from the database using the DeleteAllPayments RPC. Two new flags failed_payments_only and failed_htlcs_only have been added to this RPC, which can be used to delete only payments (HTLCs) that failed.
LeaseOutput has been extended to allow custom lease expirations and a new ListLeases RPC has been added to allow users to determine which outputs are currently being leased and for how long.
As part of our watch-only key import efforts, three new RPCs have been added to the WalletKit sub-server to interact with them:
ImportAccount imports an BIP-0049/BIP-0084 account as watch-only through its extended public key (xpub).ImportPublicKey imports a BIP-0049/BIP-0084 public key into a default "imported" account.ListAccounts exposes information for accounts (watch-only or not) existing within the wallet.
Several existing RPCs were also modified (in a backwards-compatible manner) to either accept an account parameter in their requests or break down their responses by accounts.
GetTransactions and ListUnspent can now filter through transactions and outputs belonging to a specific account.FundPSBT/FinalizePSBT can now fund/finalize PSBTs from a specific account.NewAddress can now generate addresses for a specific account.WalletBalance now returns an additional breakdown of the total wallet balance by accounts.
Longer Default Invoice Expiries
Over the past few months, we've heard from multiple sources that the default invoice expiries are too short. As a result, the default expiry for MPP invoices has been raised from one hour to one day.
Surfacing INVALID_ONION_PAYLOAD failures
Payments that fail with INVALID_ONION_PAYLOAD are now properly reported via SendToRoute and SendPaymentV2 as lnrpc.INVALID_ONION_PAYLOAD. Prior to 0.13 these were reported as lnrpc.Failure_UNKNOWN_FAILURE.
pprof Security
lnd will no longer make its pprof port accessible to the world by default, and will instead listen on localhost if a port is specified. Exposing the pprof port is still left as an option for certain use cases.
Mobile
Simplified API using the State service
Using the mobile bindings one would earlier need to wait for the unlockerReady callback to fire to unlock the wallet, and then the second callbackrpcReady before the lnd RPCs were ready to be used. This was a bit awkward, and not very well supported with React Native.
This release moves back to a single rpcReady callback, which together with the new State service can be used to determine the state of lnd at all times.
To build lnd with these new bindings for mobile, falafel v0.9.0 should be used.
Avoid killing the app on lnd failure
Instead of killing the application in case lnd fails to start, we now return an error back over the API.
Delayed Zombie Pruning
We delay graph zombie pruning on startup by 30 sec, to avoid blocking on this operation. This shortens the delay from unlocking to lnd is ready, which especially should improve the experience on mobile.
Bug Fixes
- A regression in the channel state-machine that would lead to de-sync and force close has been fixed. Channels between upgraded nodes should not experience incorrect force closes.
- The failure message used by the HTLC Interceptor to reject HTLCs has been updated to include a channel update with the temporary channel failure error code.
- A bug that could cause lnd to fail starting up if a new channel was in the process of being opened has been fixed.
- In case a channel announcment for a local channel was received after channel closure,
neutrino backed nodes could end up re-adding the channel to the graph. This has been fixed by ignoring all local announcements coming from our peers.
- A bug in the payment state machine which would result in a payment getting stuck if it received a terminal payment error shortly before dispatching another shard has been addressed.
- The
SubscribeSingleInvoice API has been updated to terminate the stream of events once the invoice reaches a terminal state.
- During a block storm or integration tests the router subsystem can lag behind on its best known block due to the pruning work it has to complete for each block. Previously the
synced_to_chain flag of the GetInfo RPC would only look at the best height of the wallet vs. its chain backend. To make sure the router is also up to date, the flag now only turns true if the router subsystem is synced to the same height as the wallet
- Some terminals don't allow more than 4096 characters to be pasted if running in interactive/TTY mode. This lead to large PSBTs being cut off during channel open. This was fixed by allowing the PSBT to be read from a file in addition to the terminal.
- The
GetNodeInfo RPC has been updated to output the channel policies for the requested node in the expected order. Previously, its ordering was inconsistent with DescribeGraph and GetChanInfo.
- The
num_pending_backups stat exposed by configured watchtower clients through the WatchtowerClient sub-server is now properly decremented after a backup has been accepted by a watchtower.
Contributors (Alphabetical Order)
Alex Bosworth
Andras Banki-Horvath
Anton Kovalenko
Bjarne Magnussen
Carla Kirk-Cohen
Conner Fromknecht
Elle Mouton
Elliott Jin
Eugene Siegel
GameXG
Hampus Sjöberg
Jake Sylvestre
Johan T. Halseth
Jonathan Underwood
Joost Jager
Juan Pablo Civile
Martin Habovštiak
Olaoluwa Osuntokun
Oliver Gugger
Pierre Rochard
Roei Erez
Tom Kirkpatrick
Umar Bolatov
Vlad Stan
Wilmer Paulino
Yaacov Akiba Slama
rockstardev
whythat
yuki-js
Yong Yu/laolu32@gmail.com/laolu32@gmail.com