Updating Offline Nodes
This section explains how to upgrade existing software or install new packages onto an offline machine (also called air gapped machine)
Background
Administrators of Linux machines may sometimes be required to update the Operating system of servers that are not connected to the internet. These servers may not have any internet access for a variety of reasons including:
They are at locatiions that have extremely slow and unusable or even no internet connections.
They were purposely disconnected from the internet for security reasons
These servers are called offline or air gapped machines.
In Cardano, stake pool operators are required to maintain an offline machine. For security reasons, this machine should never be connected to the internet.
On a regular basis (sometimes even daily), Operating System updates are released. These may include bug fixes, security updates and new features or enhancements.
Online machines that are Internet connected can benefit from these updates whenever the operator does regular maintenance or sometimes automatically for some updates that can be auto installed.
Offline machines that are not internet connected receive none of these updates. Nevertheless, these machines can be used for months or even years without having any updates as long as there is no need to update them.
For Cardano Stake Pool Operators, there was no real requirement to update the Operating System (OS) of their offline machine until version 1.35 required the installation of a new package (libsecp256k1). While only one new package was required, installing this package required other OS packages to be installed beforehand. These other packages were dependent on other packages to also be installed first and the cycle repeats.
Some operators are able to quickly resolve the issue by connecting their offline machine to the internet. This is not a valid solution in my opinion as it can compromise security— defeating the purpose of having an offline machine in the first place.
For this reason, I created this section to assist anyone needing to update an offline machine.
NOTE: I have received information on how to update libsecp256k1 by just copying certain files. If this is the only thing youLu want to do, please skip ahead to the end of this article.
Options Available
As I was researching this article, I came across various other articles— many of which were written to make the reader think that their solution is the only way to update an offline machine.
Unfortunately, I soon faced issues when I tried them out. Further research and troubleshooting led me to four of the better ways to update an offline machine. They are:
Manual Installation
Apt-Offline
Package Managers (e.g. Synaptic)
Rebuild
Of course there are other ways including creating your own package repository. However, these are the methods that I recommend as they are easier and more practical to use. Each method has their advantages and disadvantages which I will document below.
Option 1: Manual Installation
If you are only installing one software with no dependencies, then this may be the easiest option for you.
Unfortunately, I learned the hard way that almost all packages have dependencies which themselves have other dependencies so I’m almost sure you are better off with one of the other options. Nevertheless, for completion, the steps to do a manual update is listed below:
On an internet connected machine, go to the official location for downloading packages for the distribution you are using. For ubuntu it is https://packages.ubuntu.com
Look for the package that you need and download it
Your browser may stop the download due to security reasons. Just override the restriction by selecting the option to keep the package
Repeat the download for all dependencies and their dependencies as required
Copy all files to your offline machine
On the offline machine, install all packages using ”dpkg -i package-name”, taking care to install the dependencies first before the main package files.
If you think the above steps are tedious and time consuming, you are absolutely correct. That is why I recommend that you use other options below instead of this one.
Option 2: Apt-Offline
If you only want to update existing packages in your offline machine, the apt-offline command may be the best solution for you. It can quickly make a list of upgradeable software. You may also want to install new packages and their dependencies.
To use apt-offline, do this:
1. Install apt-offline on your offline machine using the manual method if it is not yet installed
Follow the steps in the previous section to install apt-offline
2. On the offline machine, create a signature file for the file you want to install e.g. pkg
sudo apt-offline set pkg-offline.sig --install-packages pkg
3. Copy the sig file to the online machine and download the packages:
apt-offline get pkg-offline.sig --bundle pkg-install.zip
4. Copy the file bundle to the offline machine and install it:
sudo apt-offline install pkg-install.zip
The package ”pkg” and all of its dependencies will be installed.
A number of experts recommend this option. Personally, I still find it tedious when there are a large number of files that need to be updated/installed.
Option 3: Package Managers (e.g. Synaptic)
Because of the limitations of the prior options, I highly recommend to use a package manager like synaptic instead. Synaptic can create download scripts for new installs, reinstallation, removal, upgradeable packages and can even fix broken dependencies. In practice however, I notice that it fails on about a third of packages and I have to manually install dependencies for these packages that failed. Still, this is my preferred method as of this writing.
The steps to use synaptic are below:
1. From your offline machine mark the packages
Install synaptic on your offline machine using one of the prior methods
Mark the new packages you want to install
NOTE: At the very least, I recommend you mark git, autotools-dev, autoconf, libtool and make for installation. Feel free to add to this list as you see fit.
Mark existing packages you want to upgrade (optional but recommended)
Go to File - Generate package download script
Select the location and enter the name of the script to be created
2. From your online machine get the packages
Copy the package download script to your online machine
Run the script to download the packages
3. From your offline machine, Install the packages
Copy the downloaded packages to your offline machine
Run synaptic
Install the packages using File - Add downloaded packages
Select the location where the packages are located (e.g. in a USB stick)
Click on Apply and wait for the process to complete
4. Install failed packages
If you have any falied updates, open a new terminal window
Install failed dependencies using ”dpkg -i package-name”, taking care to install the dependencies first before the main package files
Your update is now completed.
Option 4: Rebuild
WARNING: You may lose valuable data and actual money with this option if you do not know what you are doing. Please make sure you understand and know what you are doing before attempting this method. This section and actually the entire SPOT Check site is meant only for educational sharing. We are not responsible for any loss you may incur as a result of following any of these steps. As long as this is clear, you may proceed.
In some instances, especially if your offline machine is giving you system problems, the best way to do an update is to rebuild your offline machine. This is the most time consuming and dangerous option in this list so you should not use this regularly. Neverthesess, this option can give you a much cleaner and stable machine. Sometimes this may even be the best option especially if your offline machine has not been updated for years. You can do this easily and safely if you have a spare machine lying around as you can build a new offline machine without deleting anything from your old machine.
If however you need to rebuild onto the same machine, the process is the same but you need to be very careful as you may forget to backup certain important files. Please take this into consideration before attempting this method. If you are not sure where your important files are or are not confident you can rebuild your air gapped machine, please do not try this method at all.
With that out of the way, the steps to rebuild are below. Do all steps from your offline machine.
1. Backup important files
NOTE: Make sure all of your important files like your cold keys, scripts, compiled programs, etc are included in the backup list above.
2. Check if all important files have been copied
Make sure you can see all important files from the file listing when you run the above command
Copy mybackup.tgz to usb
3. Reinstall Linux
Follow the instructions from the distribution of Linux that you are planning to use
4. Install new packages
The initial installation of Linux will only install required and recommended packages. If you need other packages, connect your offline machine to the internet and install them now.
NOTE: At the very least, I recommend you install synaptic, git, autotools-dev, autoconf, libtool and make. Feel free to add to this list as you see fit.
5. Update Linux
Make sure you have the latest version of all packages:
5. Disconnect from the Internet
You are now finished with all updates
Remove any network cables and disable all wifi adapters
6. Restore important files
WARNING: Only restore your backup to an empty, newly created offline machine as your config files will be overwritten if you restore to an existing machine. Only proceed if you know what you are doing.
Copy mybackup.tgz to the newly rebuilt offline machine and restore the files:
Installing Libsecp256k1 Offline
The reason I created this section is to provide guidance on specific steps to install Libsecp256k1 on an offline machine. This is a required prerequisite for Cardano version 1.35 and above.
I initially used synaptic to do this but I have since been made aware of a mire straightforward method using file copying. For completion, I will put both methods here:
Synaptic Method
From your offline machine, follow Option 3 above and use synaptic to install the following packages to your offline machine:
git
autotools-dev
autoconf
libtool
make
Download the Libsecp256k1 files onto your online machine
Copy the sec.tgz tar file to your offline machine
Extract the Libsecp256k1 files onto your offline machine
Install the files
You have now completed installation of Libsecp256k1
File Copy Method
Use this method instead of the Synaptic method for a quicker and easier way to install Libsecp256k1 without compiling anything.
Big thanks to Paradoxical Sphere for sharing this easy, straightforward copy method. The entire section below is almost exclusively from him with minor tweaks and addition from myself:
1. On the air-gapped computer, add the following lines at the end of the ~/.bashrc file:
export LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"
export PKG_CONFIG_PATH="/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH"
2. To reload the ~/.bashrc file, type:
source ~/.bashrc
3. Using removable media, copy the following files from a computer that has libsecp256k1 installed to the corresponding air-gapped computer directories, making sure to recreate any needed directories and file ownership and permissions match:
/usr/local/lib/libsecp256k1.a
/usr/local/lib/libsecp256k1.la
/usr/local/lib/libsecp256k1.so.0.0.0
/usr/local/lib/pkgconfig/libsecp256k1.pc
4. To create symbolic links, type the following commands in a terminal window on the air-gapped computer:
cd /usr/local/lib
sudo ln -s libsecp256k1.so.0.0.0 libsecp256k1.so
sudo ln -s libsecp256k1.so.0.0.0 libsecp256k1.so.0
5. Confirm that in step 4 you created the following symbolic links in the /usr/local/lib folder:
lrwxrwxrwx root root libsecp256k1.so -> libsecp256k1.so.0.0.0
lrwxrwxrwx root root libsecp256k1.so.0 -> libsecp256k1.so.0.0.0
6. On the air-gapped, offline computer, type:
sudo ldconfig
Troubleshooting
Your installation may be different from mine so keep an eye out for errors regarding missing files or dependencies. To fix this, do the following:
Take note of the names of the missing packages
Install the missing packages using synaptic (option 3 above)
In extreme cases, note that you may need to repeat the above steps a few times if you see new dependency errors.
Last updated