I have home server that runs Ubuntu. I was updating Samba on it recently and I realised that the Ubuntu version was well out of date – it was still on 18.04 LTS when the latest release is 24.04 LTS. I had been updating the packages but other than that I’d just left it along. I decided it was time to do a bit of housekeeping and bring the server up to the latest edition. I could then forget about it for a while again.
Starting off, I checked the system was up to date –
10:49:john@newpiggy:/home/john sudo apt-get update
Hit:1 http://gb.archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://gb.archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:3 http://gb.archive.ubuntu.com/ubuntu bionic-backports InRelease
Hit:4 http://gb.archive.ubuntu.com/ubuntu bionic-security InRelease
Hit:5 http://ppa.launchpad.net/ethereum/ethereum/ubuntu bionic InRelease
Get:6 https://download.docker.com/linux/ubuntu bionic InRelease [64.4 kB]
Hit:7 http://ppa.launchpad.net/linuxuprising/java/ubuntu bionic InRelease
Ign:8 https://overviewer.org/debian ./ InRelease
Hit:9 https://overviewer.org/debian ./ Release
Fetched 64.4 kB in 1s (107 kB/s)
Reading package lists... Done
0:50:john@newpiggy:/home/john sudo apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following package was automatically installed and is no longer required:
bootnode
Use 'sudo apt autoremove' to remove it.
The following packages have been kept back:
base-files docker-ce-cli grub-common grub-pc grub-pc-bin grub2-common linux-generic linux-headers-generic linux-image-generic netplan.io python3-update-manager sosreport ubuntu-advantage-tools ubuntu-server
update-manager-core
0 upgraded, 0 newly installed, 0 to remove and 15 not upgraded.All good so far – time to take the plunge
10:50:john@newpiggy:/home/john sudo do-release-upgrade
Checking for a new Ubuntu release
Please install all available updates for your release before upgrading.This was peculiar. My update/upgrade commands above were saying my system was up to date, but do-release-upgrade was saying “all available updates” were not installed.
Much head scratching ensued.
Knowledge Woes – I was wrong
I’d entered this process thinking that apt-get update/upgrade would update all the packages on my system and bring my system fully up to date. I was half right
The command apt-get update will update the local index files stored on the system. From the man page for apt-get:
update
update is used to resynchronize the package index files from their sources. The indexes of available packages are fetched from the location(s) specified in /etc/apt/sources.list. For example, when using a Debian archive, this command retrieves and scans the Packages.gz files, so that information about new and updated packages is available. An update should always be performed before an upgrade or dist-upgrade. Please be aware that the overall progress meter will be incorrect as the size of the package files cannot be known in advance.
So it hints at what I am missing here – if I look at the man section for dist-upgrade
dist-upgrade
dist-upgrade in addition to performing the function of upgrade, also intelligently handles changing dependencies with new versions of packages; apt-get has a “smart” conflict resolution system, and it will attempt to upgrade the most important packages at the expense of less important ones if necessary. The dist-upgrade command may therefore remove some packages. The /etc/apt/sources.list file contains a list of locations from which to retrieve desired package files. See also apt_preferences(5) for a mechanism for overriding the general settings for individual packages.
When a package is supplied as an argument, the package will be installed prior to the upgrade action.
the dist-upgrade option is obviously far more comprehensive, as it update the packages where dependencies have changed. It will add new packages if needed, and remove old ones too. It is an inherently more risky process as the dependencies may fail, or the may introduce new and unstable combinations of packages.
For me though, I had to run it to get my system into an upgradable state.
10:50:john@newpiggy:/home/john sudo do-release-upgrade
Reading cache
Checking package manager
Continue running under SSH?
This session appears to be running under ssh. It is not recommended
to perform a upgrade over ssh currently because in case of failure it
is harder to recover.
If you continue, an additional ssh daemon will be started at port
'1022'.
Do you want to continue? Python Woes
So, having solved the first problem I was immediately presented with another
Can not upgrade
Your python3 install is corrupted. Please fix the '/usr/bin/python3' symlink.This was a bit perplexing as my python versions were working fine.
john@newpiggy:/home/john python
Python 3.10.7 (main, May 11 2023, 16:30:32) [GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>So I was using pyenv to manage my python installations. I was hoping this was the issue, to check pyenv was managing the python runtime
john@newpiggy:/home/john which python
/home/john/.pyenv/shims/pythonI edited my .zshrc script and commented out the pyenv initialisation block.
# export PYENV_ROOT="$HOME/.pyenv"
# export PATH="$PYENV_ROOT/bin:$PATH"
# if command -v pyenv 1>/dev/null 2>&1; then
# eval "$(pyenv init -)"
# filog off/on, and running the upgrade again
Can not upgrade
Your python3 install is corrupted. Please fix the '/usr/bin/python3' symlink.Looking on the web this seems to be a fairly common problem. There were numerous avenues to go down, but eventually one worked
sudo ln -fs /usr/bin/python3.6 /usr/bin/python3This seemed to stop the problem. I was able to get past the error and see the screen scrolling with line after line of packages being updated and upgraded.
Grub Woes
So thinking I was on the home run, I let the system run – I came back to the following ominous screen –
This was a bit confusing because /dev/sda was part of a RAID array and I wasn’t booting off it. But nothing can scare me more than grub; probably because it can render your system completely unreachable, and because I never spent any time learning about it so its all a bit too much like magic to me. I understand what it does in principle, and I have a passing knowledge on how it works, but beyond that if I get presented with a screen like that above I groan.
Logging in on a second ssh connection I ran fdisk -l
Disk /dev/sdc: 465.76 GiB, 500107862016 bytes, 976773168 sectors
Disk model: CT500MX500SSD4
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 524F733F-9E98-4A28-8990-26F65E474499
Device Start End Sectors Size Type
/dev/sdc1 2048 4095 2048 1M BIOS boot
/dev/sdc2 4096 976771071 976766976 465.8G Linux filesystemSo my boot was on /dev/sdc, but it was trying to install grub on /dev/sda??? This was all very confusing.
Much reaching on the web, and I took the bold and perhaps stupid decision to reboot and see what happened. I reset the machine and then started pinging it … it didn’t come back. Now it didn’t come back not because of grub, but grub played a part.
IMPI to the rescue
luckily I have an IMPI console on the server, so once I remembered (1) that I had IMPI (2) the actual name was IMPI (3) how to connect (http) and (4) the username/password I was able to connect to the machine.
And it booted up fine.
I performed a couple of resets via the IMPI interface and via SSH, and every time it came up. So I disconnected IMPI and yay (sort of) it didn’t boot.
So why was my server no longer booting headless? All the we searches pointed to the BIOS, but I was certain I hadn’t updated that, so it had to be something else.
12:48:john@newpiggy:/home/john sudo nano /etc/default/grub
Uncomment to disable graphical terminal (grub-pc only) GRUB_TERMINAL=consoleSo that was it. The grub settings had been changed to use a graphical terminal. Disabling that and the system would start headless again.
Further Updates
I had successfully updated from 18.04 to 20.04. From there I upgraded to 22.04 and finally to 24.04 without further issues.