This release marks the first major release in the v0.12.x series! As this is a major release several new features are included in this release including: anchor commitment types are now the default, anchor commitment support for watchtowers, new arguments to auto compact the database as well as drop the wtxmgr state, generic wallet PSBT crafting+signing, and much more! As usual, this release contains several important bug fixes, so we recommend all users update.
Database Migrations
A migration to initialize a top-level peers bucket is included in this release. The bucket is used to track flap counts for peers that we have channels open with across restarts. These values are used to rate-limit the amount of memory that lnd uses to track peers online state, ensuring that we do not store large volumes of uptime information for peers that are constantly changing online state.
This release contains a migration to initialize a top-level-bucket for an outpoint index. There is also a subsequent migration that populates this index with an outpoint's status. This will cut down on expensive bbolt transactions throughout the codebase. The migration process should look something like this upon initial start up:
2020-12-21 10:45:07.256 [INF] LTND: Version: 0.12.0-beta commit=v0.12.0-beta, build=production, logging=default
2020-12-21 10:45:07.257 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2020-12-21 10:45:07.257 [INF] LTND: Opening the main database, this might take a few minutes...
2020-12-21 10:45:07.257 [INF] LTND: Opening bbolt database, sync_freelist=false, auto_compact=false
2020-12-21 10:45:07.304 [INF] CHDB: Checking for schema update: latest_version=20, db_version=17
2020-12-21 10:45:07.304 [INF] CHDB: Performing database schema migration
2020-12-21 10:45:07.304 [INF] CHDB: Applying migration #18
2020-12-21 10:45:07.304 [INF] CHDB: Creating top-level bucket: "peers-bucket" ...
2020-12-21 10:45:07.305 [INF] CHDB: Created top-level bucket: "peers-bucket"
2020-12-21 10:45:07.305 [INF] CHDB: Applying migration #19
2020-12-21 10:45:07.305 [INF] CHDB: Creating top-level bucket: "outpoint-bucket" ...
2020-12-21 10:45:07.305 [INF] CHDB: Created top-level bucket: "outpoint-bucket"
2020-12-21 10:45:07.305 [INF] CHDB: Applying migration #20
2020-12-21 10:45:07.324 [INF] LTND: Database now open (time_to_open=67.71764ms)!
Verifying the Release
In order to verify the release, you'll need to have gpg or gpg2 installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already:
curl https://keybase.io/bitconner/pgp_keys.asc | gpg --import
curl https://keybase.io/roasbeef/pgp_keys.asc | gpg --import
Once you have the required PGP keys, you can verify the release (assuming manifest-roasbeef-v0.12.0-beta.rc6.txt.asc is in the current directory) with:
gpg --verify manifest-roasbeef-v0.12.0-beta.rc6.txt.asc
You should see the following if the verification was successful:
gpg: Signature made Wed Sep 30 17:35:20 2020 PDT
gpg: using RSA key 4AB7F8DA6FAEBB3B70B1F903BC13F65E2DC84465
gpg: Good signature from "Olaoluwa Osuntokun <laolu32@gmail.com>" [ultimate]
That will verify the signature of the manifest file, which ensures integrity and authenticity of the archive you've downloaded locally containing the binaries. Next, depending on your operating system, you should then re-compute the sha256 hash of the archive with shasum -a 256 <filename>, compare it with the corresponding one in the manifest file, and ensure they match exactly.
Verifying the Release Timestamp
From this new version onwards, in addition time-stamping the git tag with OpenTimeStamps, we'll also now timestamp the manifest file along with its signature. Two new files are now included along with the rest of our release artifacts: manifest-roasbeef-v0.12.0-beta.rc6.txt.asc.ots.
Assuming you have the opentimestamps client installed locally, the timestamps can be verified with the following commands:
ots verify manifest-roasbeef-v0.12.0-beta.rc6.txt.asc.ots -f manifest-roasbeef-v0.12.0-beta.rc6.txt.asc
Alternatively, the open timestamps website can be used to verify timestamps if one doesn't have a bitcoind instance accessible locally.
These timestamps should give users confidence in the integrity of this release even after the key that signed the release expires.
Verifying the Release Binaries
Our release binaries are fully reproducible. Third parties are able to verify that the release binaries were produced properly without having to trust the release manager(s). See our reproducible builds guide for how this can be achieved.
The release binaries are compiled with go1.15.6, which is required by verifiers to arrive at the same ones.
They include the following build tags: autopilotrpc, signrpc, walletrpc, chainrpc, invoicesrpc, routerrpc, and watchtowerrpc. Note that these are already included in the release script, so they do not need to be provided.
The make release command can be used to ensure one rebuilds with all the same flags used for the release. If one wishes to build for only a single platform, then make release sys=<os-arch> tag=<tag> can be used.
Finally, you can also verify the tag itself with the following command:
$ git verify-tag v0.12.0-beta.rc6
gpg: Signature made Tue Sep 15 18:55:00 2020 PDT
gpg: using RSA key 4AB7F8DA6FAEBB3B70B1F903BC13F65E2DC84465
gpg: Good signature from "Olaoluwa Osuntokun <laolu32@gmail.com>" [ultimate]
Verifying the Docker Images
To verify the lnd and lncli binaries inside the docker images against the signed, reproducible release binaries, there is a verification script in the image that can be called (before starting the container for example):
$ docker pull lightninglabs/lnd:v0.12.0-beta.rc6
$ docker run --rm --entrypoint="" lightninglabs/lnd:v0.12.0-beta.rc6 /verify-install.sh
$ OK=$?
$ if [ "$OK" -ne "0" ]; then echo "Verification failed!"; exit 1; done
$ docker run lightninglabs/lnd [command-line options]
Building the Contained Release
Users are able to rebuild the target release themselves without having to fetch any of the dependencies. In order to do so, assuming
that vendor.tar.gz and lnd-source-v0.12.0-beta.rc6.tar.gz are in the current directory, follow these steps:
tar -xvzf vendor.tar.gz
tar -xvzf lnd-source-v0.12.0-beta.rc6.tar.gz
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.12.0-beta.rc6" ./cmd/lnd
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.12.0-beta.rc6" ./cmd/lncli
The -mod=vendor flag tells the go build command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.
Additionally, it's now possible to use the enclosed release.sh script to bundle a release for a specific system like so:
make release sys="linux-arm64 darwin-amd64"
⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️
Release Notes
New Default Autopilot Heuristic
In this version of lnd, the default heursitic for autopilot has been changed from preferential attachment, to a version that will attempt to optimize for the betweeness centrality of the node. At a high level, this change means that rather than trying to connect (stochastically) to the nodes that have the most channels, lnd will instead attempt to connect to the nodes that appear most often in the shortest paths within the network. This change will serve to step as a stepping stone to further diffuse the graph to make it more resilient.
Pathfinding Improvements
lnd will now properly penalize attempts of larger "wumbo" sized payments proportionally. This will serve to ensure that clients with less active failure information are able to properly prune the search space by increasing the attempt cost for larger payments. New flags has been added to allow users to configure the attempt cost for this value (attemptcost and attemptcostppm). We encourage users taht frequently send larger payments to tweak these parameters to find what works best, and ideally communicate this information back to the maintainers of lnd so we can better tune the current default value.
Graph Download Optimizations
lnd will now batch all insertion operations related to channel graph which should greatly speed up initial graph download. Initial becnhmarks show this change resluting in a 3x speed increase, with further gains likely being seen on mobile and more constrained platforms.
Peer to Peer Updates
A new flag has been added to lnd to enforce a global connection timeout when trying to connect out to new peers. Setting a lower value for this new command line option (timeout) will mean that lnd will give up on unrechable peers much sooner than before, which can be useful when attempting to connect to a set of addresses to open chnnel to a peer.
Automatic Database Compaction
The most important data of any lnd node is stored in its channel database (channel.db). The database library currently used for this DB is bbolt which by design does not give back free space to the file system, even if data is deleted from the DB. This can lead to large DB files and slow startup times. Compaction is the process of creating a fresh copy of a bbolt database that only contains data and no "reserved free space". This process also de-fragments and validates the integrity of the data.
Automatic compaction of the channel.db can now be turned on with the flag --db.bolt.auto-compact. By default this will compact on startup, if the last compaction was more than a week ago. The flag --db.bolt.auto-compact-min-age can also be set to 0 to force compaction on every startup, independent of how long ago it happened last.
Protocol Upgrades
Anchor Output Channels
lnd will now open the new channel type dubbed "anchor channels" by default if both peers support it. You signal support by setting the --protocol.anchors flag at lnd startup. This is a channel type that has been available to advanced users since lnd v0.10, but it has seen a few updates that makes it even safer and useful in high fee scenarios, and it is now in line with a proposed BOLT change.
The anchor channel type is a new type of channel that is much safer in high fee scenarios, as it allows bumping the fees after the channel has been force closed, instead of making the peers agreeing on a future close fee. This is also a nice UX improvement, as less of the channel capacity needs to go towards the commitment fee reserve, and can instead be used for payments. In addition it allows bundling multiple HTLC transactions together into one, potentially saving on chain fees in force close scenarios.
The commitment transaction still needs to be signed up front with a fee that ensures its mempool acceptance, and this fee now defaults to 10 sat/vbyte. This can be tuned by the --max-commit-fee-rate-anchors flag, but this should be used with caution.
Note that one has to have on-chain funds available in the wallet for fee bumping channel closes for anchor channels. Because of this, a small portion of the wallet balance will be reserved for this purpose, and some on-chain actions that used to be allowed can now be rejected if you have anchor channels open.
Static Remote Key Feature Bit Required
This new version of lnd now requires channels that use a static remote key, AKA "tweakless commitments". This change improves safety and security for users as now when a channel is force closed by the remote party, the funds will go directly to a user control key. Prior versions of lnd have supported this channel type, but lnd will now only allow this type of channel when making channels with new peer.
Lnd will waive this requirement in the case where it still has legacy channels with a peer. This ensures that lnd can still connect to nodes it has existing channels with, even if they do not understand the feature bit.
Improved End to End Payment Security
The MPP protocol upgrade included a so called "payment address" that improves end-to-end payment security by requiring the sender to include a special nonce in the onion payload specify by the receiver. As intermediate nodes can't guess this secret ahead of time, and it's encrypted in the onion only to the finally receiver, they thwarts a large class of probing and de-anonymization attacks. This new release of lnd will now require this feature bit set in any new invoices it creates, which means all payments that don't include this new payment secret will be rejected.
PSBT Signing
The internal wallet can now create and sign PSBTs. In combination with the ListUnspent RPC this allows RPC users to implement full coin control.
This feature also takes us one step closer to the goal of supporting watch-only on-chain wallets in lnd where an online node would only have public keys to track the UTXOs and would delegate the signing to a non-networked lnd node that has the private keys, all through using PSBTs.
Read more about the possible use cases and dive into the examples in our PSBT documentation.
Build System
Leveraging the power of GitHub Workflows, we now automatically build and push docker images of all our releases to Docker Hub. This includes images for amd64 and arm64.
The distinction between the production Dockerfile and the development dev.Dockerfile were made more clear in the documentation.
The release binaries for all OS/architectures are now also built by a GitHub Workflow. The deterministic build system introduced in a previous release allows us to independently build and sign the binaries locally. Signatures of more than just one developer will be added to releases in the future.
The experiential build tag has been removed for the assumechanvalid flag that is used to prevent long rescans for neutrino nodes.
Continuous Integration
Our continuous integration pipeline, most notably our integration tests, has received a number of improvements and bug fixes making them considerably faster and somewhat more stable:
- An integration test suite running against a bitcoind with the TX index disabled was added.
- The ~70 integration tests are now split into 4 parts and run in parallel reducing the execution time by ~50%.
- Log files are only uploaded to termbin.com and file.io for failed runs and the bitcoind binaries are extracted from a docker image instead of being downloaded, shaving off a few more minutes from the total itest execution time.
- The test harness for the btcd node used as the mining node were improved to fix port collisions which resulted in flaky tests.
- A check was added that forces new command line flags to also be documented in the sample-lnd.conf file.
- A new make target for itest flake hunting was added.
- New make targets for running fuzz tests were added.
- Build tags were removed from the integration test files, allowing the linter to check those as well.
- The zpay32 package's Decode and Encode functions now have corresponding fuzz tests in the fuzz package.
- The brontide fuzz tests have been fixed.
- Fuzz testing has been optimized to instruct gofuzz to always mutate the input.
Performance improvements were made to the contract court subsystem which is responsible for closing out channels on chain and taking on-chain actions required to fully resolve the channel. The number of database transactions required to start up the subsystem has been reduced from one per channel to a single transaction, which reduces startup time. Improvements to the way the subsystem consumes new blocks from its backing bitcoin node have also improved the memory footprint of the system.
Extended Health Checks
A new optional healthcheck has been added to insturct lnd to restart itself in order to refresh an expired RPC TLS cert. This change is useful in containerized contexts such as k8s, where an auto restarting lnd is able to propagate any auth changes in a decoupled manner upon restart.
htlcswitch Enhancements
Database contention has been reduced in the link by batching removal of forwarding packages. The removal timer has also been increased from 1 minute to 1 hour.
A bug has been fixed in our non-strict forwarding randomization to ensure we explciitly randomize our link sleection rather than relying on the undefined ordering of map interation in the Go spec.
Gossip Enhancements
During the development of this release, an increase in high disk usage was reported by several users throughout the network due to channel update spam. To minimize the effects to disks, lnd will now rate limit channel updates in two ways. Keep-alive updates, those which only increase their timestamps to signal liveliness, are now limited to one per rebroadcast interval (current default of 24H). Non-keep-alive updates are now limited to one per block per direction.
As required by the specification, lnd now enforces all channels for a block must fit within a single reply. It would previously not enforce this as it would allow an overlap of blocks across multiple replies, which could lead other implementations to send us a error due to the overlap and cause connect/disconnect cycles.
lnd will no longer accept premature channel announcements/updates and wait for their maturity.
Peer Flap Rate Tracking
An update to the channel fitness subsystem has introduced tracking of the number of times lnd is connected and disconnected from each of its peers. This information is surfaced in the output of the ListPeers API.
The flap rate we have recorded for peers is also used to rate limit the amount of data lnd will store to track the peer’s uptime. If a peer has a high flap rate, lnd will reduce the amount of data it stores in memory, resulting in more aggregated uptime information. This change is intended to protect against constantly flapping peers, and will have little effect on peers that are consistently online with the occasional restart. To ensure that we do not permanently punish a peer for a period of instability long in the past, the flap rate we track for peers is exponentially cooled down over time.
RPC Enchancements & Bug Fixes
The best_header_timestamp field included in the GetInfo's RPC response will now be set to what the backend reports while it is syncing. This will restore the ability for higher-level applications to determine their sync progress.
Watchtower Address Removal
The last address of a registered watchtower can no longer be removed to prevent a potential panic.
lnd now allows all RPC calls that craft and send transactions to spend unconfirmed coins.
This change the following RPCs:
Lightning.SendCoinsLightning.SendManyWalletKit.SendOutputs
We've added two new parameters for these methods, following the same format as used for Lightning.OpenChannel RPC:
min_confs (default=1)spend_unconfirmed (default=false)
Macaroon Root ID Key Rotation
lnd now supports root ID key rotation. This allows the baker (creator) of a set of macaroons to invalidate them all by deleting and regenerating the root key used to generate the macaroons. This feature is a useful security tool, as if an application/system that uses lnd's macaroons in a fine grained manner is compromised, the admin is able to revoke all generated macaroons.
Several new calls have been added to allow users to take advantage of this feature, namely:
* The lncli bakemacaroon call now accepts a new parameter: root_key_id. This new field is an integer that can be used to rotate root ID keys.
* A new lncli listmacaroonids command has been added to allow callers to monitor all their existing allocated root IDs.
* A new lncli deletemacaroonid call has been added which implements macaroon revocation by allowing the caller to delete a specified root key ID.
New Verbose Output for ChannelBalance
The lncli channebalance call now returns much more information than before in order to give users more insight w.r.t exactly how their funds are distributed off-chain. An output of the new call resmbles the following:
⛰lncli channelbalance
{
"balance": "27476201",
"pending_open_balance": "0",
"local_balance": {
"sat": "27476201",
"msat": "27476201135"
},
"remote_balance": {
"sat": "109137173",
"msat": "109137173865"
},
"unsettled_local_balance": {
"sat": "0",
"msat": "0"
},
"unsettled_remote_balance": {
"sat": "0",
"msat": "0"
},
"pending_open_local_balance": {
"sat": "0",
"msat": "0"
},
"pending_open_remote_balance": {
"sat": "1783362",
"msat": "1783362000"
}
}
Note that the first two fields (balance and pending_open_balance) are now deprecated and will be removed in the future. Callers should use the new fields that return both sat and msat instead.
Raw Key Support for SharedKeyRequest
The DeriveSharedKey now accepts a raw public key in addition to key locator.
The ListChannels call will now return additional information about the set of linked HTLCs in a channel. Namely, we'll now return:
* The htlc_index of the HTLC within the channel
* The forwarding_channel, or the channel that forwarded the HTLC to the targte channel
* The forwarding_htlc_index, or the HTLC index on the forwarded channel.
Automated Let's Encrypt Certificates
A new series of command line flags have been added to lnd which allows users to automatically obtain and renew a Let's Encrypt Certificate for the RPC interface of their lnd node. With this change, in certain configurations, callers will be able to hit an lnd now without having to manually store and update the tls.cert locally.
New flags added to the lnd command line and lnd.conf:
--letsencryptport: The port on which lnd will listen for Let's Encrypt challenges. Let's Encrypt will always try to contact on port 80. Often non-root processes are not allowed to bind to ports lower than 1024. This configuration option allows a different port to be used, but must be used in combination with port forwarding from port 80.--letsencryptdir: The directory to store Let's Encrypt certificates within. By default this is .lnd/letsencrypt.--letsencryptdomain: Request a Let's Encrypt certificate for the domain specified using this flag.
When lncli cannot find a tls.cert file, it will assume the server has a valid (Let's Encrypt) certificate. It is important to pass the domain name as a command line flag to lncli:
lncli --rpcserver my.domain.org:10009
This is necessary as well when connecting to localhost.
Custom Routing Hints for AddHoldInvoice
The AddHoldInvoice RPC call now allows the users to specify their own custom routing hints.
Allow No RPC Auth on Private Addresses
A new config evaluation has been added to allow user to instruct lnd that it should allow RPC requests with no authentiation only if lnd is listening on a private address. This makes certain Docker based configurations more user friendly, as any dependent containers no longer need to obtain and update lnd's RPC authentication information. Assuming lnd is only listening on a non-public private interface, then the --no-macaroons config option is now valid.
New Channel Acceptor Parameters
Additional fields have been added to the ChannelAcceptor API, which allow custom setting of custom errors for the remote peer, an upfront shutdown address for the channel (if supported by the peer), and more. Note that the error provided will be sent to the peer verbatim, so should not contain sensitive information.
Maximum Local CSV
When opening a channel, the remote party can specify the CSV delay for your funds. This value determines the amount of time that your balance will be unavailable in the case where your force close the channel. A max_local_csv parameter has been added to allow setting of custom limitations on this value. For outgoing channels, this can be set using the max_local_csv field in the OpenChannel request. The maxlocaldelay config value can be used to set a default maximum value for all channels.
Disable TLS for REST
It is now possible to disable TLS for REST RPC using --no-rest-tls.
Refactoring
This release sees the removal of several components from the main lnd package:
- fundingmanager.go and tests are moved to the funding package.
- chainregistry.go and chainparams.go have been moved to the chainreg package.
- mock.go has been removed in favor of the lntest/mock package.
- A global variable activeNetParams has been removed.
The peer package's dependency on brontide has been removed.
Miscellaneous
The DNS servers to use for initial peer bootstrapping can now be specified to overwrite the hard coded default values.
All supported command line flags are now also properly documented in the sample-lnd.conf file.
A new flag has been added to instruct lnd to timeout early if it can't obtain the file lock on bolt DB.
Multi node management
Hosting nodes on non-trusted (cloud) hardware was made safer by adding a stateless initialization mode that instructs lnd to not store any unencrypted macaroons on the host's file system. Instead, the admin macaroon is returned in the response of the wallet creation request and must be stored by the caller.
To support the stateless initialization mode mentioned above on the client side as well, configuration profiles for lncli can now be created. Those profiles make it easy to interact with multiple nodes from the same client machine. For additional security the macaroons stored in the profiles can optionally be encrypted with a password.
Recovery
Forcing the on-chain wallet to rescan its state from chain was made easier by adding the --reset-wallet-transactions flag to lnd that replaces the functionality previously only available in the external tool dropwtxmgr.
Individual subsystem log levels
A change that makes it possible to set the log level for individual subsystems was merged. One can now specify a global log level, and subsystem log levels that will override the global setting: --debuglevel=debug,PEER=info,SRVR=trace.
Bug fixes
The full list of changes since v0.11.0-beta can be found here:
Contributors (Alphabetical Order)
Alex Bosworth
András Bánki-Horváth
Ben Woosley
Bjarne Magnussen
Calvin Zachman
Carla Kirk-Cohen
Carsten Otto
Conner Fromknecht
Dan Janosik
Daniel Babbev
Dominik Spicher
Eugene Siegel
Federico Bond
Glen Cooper
githorray
Graham Krizek
Hampus Sjöberg
Johan T. Halseth
Joost Jager
Juan Pablo Civile
Jules Lamur
Kartik Shah
Marty Jones
Matheus Degiovani
Mayank Chhabra
MrManPew
Olaoluwa Osuntokun
Oliver Gugger
positiveblue
Roei Erez
Tom Kirkpatrick
Torkel Rogstad
Wilmer Paulino
Yaacov Akiba Slama
Yan Pritzker
yyforyongyu /laolu32@gmail.com/laolu32@gmail.com