Updating Offline Nodes

This section explains how to upgrade existing software or install new packages onto an offline machine (also called air gapped machine)

Background

Administrators of Linux machines may sometimes be required to update the Operating system of servers that are not connected to the internet. These servers may not have any internet access for a variety of reasons including:
  • They are at locatiions that have extremely slow and unusable or even no internet connections.
  • They were purposely disconnected from the internet for security reasons
These servers are called offline or air gapped machines.
In Cardano, stake pool operators are required to maintain an offline machine. For security reasons, this machine should never be connected to the internet.
On a regular basis (sometimes even daily), Operating System updates are released. These may include bug fixes, security updates and new features or enhancements.
  • Online machines that are Internet connected can benefit from these updates whenever the operator does regular maintenance or sometimes automatically for some updates that can be auto installed.
  • Offline machines that are not internet connected receive none of these updates. Nevertheless, these machines can be used for months or even years without having any updates as long as there is no need to update them.
For Cardano Stake Pool Operators, there was no real requirement to update the Operating System (OS) of their offline machine until version 1.35 required the installation of a new package (libsecp256k1). While only one new package was required, installing this package required other OS packages to be installed beforehand. These other packages were dependent on other packages to also be installed first and the cycle repeats.
Some operators are able to quickly resolve the issue by connecting their offline machine to the internet. This is not a valid solution in my opinion as it can compromise security— defeating the purpose of having an offline machine in the first place.
For this reason, I created this section to assist anyone needing to update an offline machine.
NOTE: I have received information on how to update libsecp256k1 by just copying certain files. If this is the only thing youLu want to do, please skip ahead to the end of this article.

Options Available

As I was researching this article, I came across various other articles— many of which were written to make the reader think that their solution is the only way to update an offline machine.
Unfortunately, I soon faced issues when I tried them out. Further research and troubleshooting led me to four of the better ways to update an offline machine. They are:
  1. 1.
    Manual Installation
  2. 2.
    Apt-Offline
  3. 3.
    Package Managers (e.g. Synaptic)
  4. 4.
    Rebuild
Of course there are other ways including creating your own package repository. However, these are the methods that I recommend as they are easier and more practical to use. Each method has their advantages and disadvantages which I will document below.

Option 1: Manual Installation

If you are only installing one software with no dependencies, then this may be the easiest option for you.
Unfortunately, I learned the hard way that almost all packages have dependencies which themselves have other dependencies so I’m almost sure you are better off with one of the other options. Nevertheless, for completion, the steps to do a manual update is listed below:
  1. 1.
    On an internet connected machine, go to the official location for downloading packages for the distribution you are using. For ubuntu it is https://packages.ubuntu.com
  2. 2.
    Look for the package that you need and download it
  3. 3.
    Your browser may stop the download due to security reasons. Just override the restriction by selecting the option to keep the package
  4. 4.
    Repeat the download for all dependencies and their dependencies as required
  5. 5.
    Copy all files to your offline machine
  6. 6.
    On the offline machine, install all packages using ”dpkg -i package-name”, taking care to install the dependencies first before the main package files.
If you think the above steps are tedious and time consuming, you are absolutely correct. That is why I recommend that you use other options below instead of this one.

Option 2: Apt-Offline

If you only want to update existing packages in your offline machine, the apt-offline command may be the best solution for you. It can quickly make a list of upgradeable software. You may also want to install new packages and their dependencies.
To use apt-offline, do this:

1. Install apt-offline on your offline machine using the manual method if it is not yet installed

Follow the steps in the previous section to install apt-offline

2. On the offline machine, create a signature file for the file you want to install e.g. pkg

sudo apt-offline set pkg-offline.sig --install-packages pkg

3. Copy the sig file to the online machine and download the packages:

apt-offline get pkg-offline.sig --bundle pkg-install.zip

4. Copy the file bundle to the offline machine and install it:

sudo apt-offline install pkg-install.zip
The package ”pkg” and all of its dependencies will be installed.
A number of experts recommend this option. Personally, I still find it tedious when there are a large number of files that need to be updated/installed.

Option 3: Package Managers (e.g. Synaptic)

Because of the limitations of the prior options, I highly recommend to use a package manager like synaptic instead. Synaptic can create download scripts for new installs, reinstallation, removal, upgradeable packages and can even fix broken dependencies. In practice however, I notice that it fails on about a third of packages and I have to manually install dependencies for these packages that failed. Still, this is my preferred method as of this writing.
The steps to use synaptic are below:

1. From your offline machine mark the packages

  • Install synaptic on your offline machine using one of the prior methods
  • Mark the new packages you want to install
NOTE: At the very least, I recommend you mark git, autotools-dev, autoconf, libtool and make for installation. Feel free to add to this list as you see fit.
  • Mark existing packages you want to upgrade (optional but recommended)
  • Go to File - Generate package download script
  • Select the location and enter the name of the script to be created

2. From your online machine get the packages

  • Copy the package download script to your online machine
  • Run the script to download the packages

3. From your offline machine, Install the packages

  • Copy the downloaded packages to your offline machine
  • Run synaptic
  • Install the packages using File - Add downloaded packages
  • Select the location where the packages are located (e.g. in a USB stick)
  • Click on Apply and wait for the process to complete

4. Install failed packages

  • If you have any falied updates, open a new terminal window
  • Install failed dependencies using ”dpkg -i package-name”, taking care to install the dependencies first before the main package files
Your update is now completed.

Option 4: Rebuild

WARNING: You may lose valuable data and actual money with this option if you do not know what you are doing. Please make sure you understand and know what you are doing before attempting this method. This section and actually the entire SPOT Check site is meant only for educational sharing. We are not responsible for any loss you may incur as a result of following any of these steps. As long as this is clear, you may proceed.
In some instances, especially if your offline machine is giving you system problems, the best way to do an update is to rebuild your offline machine. This is the most time consuming and dangerous option in this list so you should not use this regularly. Neverthesess, this option can give you a much cleaner and stable machine. Sometimes this may even be the best option especially if your offline machine has not been updated for years. You can do this easily and safely if you have a spare machine lying around as you can build a new offline machine without deleting anything from your old machine.
If however you need to rebuild onto the same machine, the process is the same but you need to be very careful as you may forget to backup certain important files. Please take this into consideration before attempting this method. If you are not sure where your important files are or are not confident you can rebuild your air gapped machine, please do not try this method at all.
With that out of the way, the steps to rebuild are below. Do all steps from your offline machine.

1. Backup important files

sudo tar cvzf mybackup.tgz /home /usr/local/bin /OtherDirsYouNeed
NOTE: Make sure all of your important files like your cold keys, scripts, compiled programs, etc are included in the backup list above.

2. Check if all important files have been copied

sudo tar tvzf mybackup.tgz
Make sure you can see all important files from the file listing when you run the above command
Copy mybackup.tgz to usb

3. Reinstall Linux

Follow the instructions from the distribution of Linux that you are planning to use

4. Install new packages

The initial installation of Linux will only install required and recommended packages. If you need other packages, connect your offline machine to the internet and install them now.
sudo apt install package-name
NOTE: At the very least, I recommend you install synaptic, git, autotools-dev, autoconf, libtool and make. Feel free to add to this list as you see fit.

5. Update Linux

Make sure you have the latest version of all packages:
sudo apt update
sudo apt upgrade -y

5. Disconnect from the Internet

You are now finished with all updates
Remove any network cables and disable all wifi adapters

6. Restore important files

WARNING: Only restore your backup to an empty, newly created offline machine as your config files will be overwritten if you restore to an existing machine. Only proceed if you know what you are doing.
Copy mybackup.tgz to the newly rebuilt offline machine and restore the files:
tar xvzf mybackup.tgz

Installing Libsecp256k1 Offline

The reason I created this section is to provide guidance on specific steps to install Libsecp256k1 on an offline machine. This is a required prerequisite for Cardano version 1.35 and above.
I initially used synaptic to do this but I have since been made aware of a mire straightforward method using file copying. For completion, I will put both methods here:

Synaptic Method

  • From your offline machine, follow Option 3 above and use synaptic to install the following packages to your offline machine:
    • git
    • autotools-dev
    • autoconf
    • libtool
    • make
  • Download the Libsecp256k1 files onto your online machine
cd $HOME/git
git clone https://github.com/bitcoin-core/secp256k1
tar cvzf sec.tgz secp256k1
  • Copy the sec.tgz tar file to your offline machine
  • Extract the Libsecp256k1 files onto your offline machine
mkdir $HOME/git
cd $HOME/git
tar xvzf sec.tgz
  • Install the files
cd secp256k1
git checkout ac83be33
./autogen.sh
./configure --enable-module-schnorrsig --enable-experimental
make
make check
sudo make install
sudo ldconfig
You have now completed installation of Libsecp256k1

File Copy Method

Use this method instead of the Synaptic method for a quicker and easier way to install Libsecp256k1 without compiling anything.
Big thanks to Paradoxical Sphere for sharing this easy, straightforward copy method. The entire section below is almost exclusively from him with minor tweaks and addition from myself:
1. On the air-gapped computer, add the following lines at the end of the ~/.bashrc file:
export LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"
export PKG_CONFIG_PATH="/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH"
2. To reload the ~/.bashrc file, type:
source ~/.bashrc
3. Using removable media, copy the following files from a computer that has libsecp256k1 installed to the corresponding air-gapped computer directories, making sure to recreate any needed directories and file ownership and permissions match:
/usr/local/lib/libsecp256k1.a
/usr/local/lib/libsecp256k1.la
/usr/local/lib/libsecp256k1.so.0.0.0
/usr/local/lib/pkgconfig/libsecp256k1.pc
4. To create symbolic links, type the following commands in a terminal window on the air-gapped computer:
cd /usr/local/lib
sudo ln -s libsecp256k1.so.0.0.0 libsecp256k1.so
sudo ln -s libsecp256k1.so.0.0.0 libsecp256k1.so.0
5. Confirm that in step 4 you created the following symbolic links in the /usr/local/lib folder:
lrwxrwxrwx root root libsecp256k1.so -> libsecp256k1.so.0.0.0
lrwxrwxrwx root root libsecp256k1.so.0 -> libsecp256k1.so.0.0.0
6. On the air-gapped, offline computer, type:
sudo ldconfig

Troubleshooting

Your installation may be different from mine so keep an eye out for errors regarding missing files or dependencies. To fix this, do the following:
  • Take note of the names of the missing packages
  • Install the missing packages using synaptic (option 3 above)
In extreme cases, note that you may need to repeat the above steps a few times if you see new dependency errors.
​