Textpattern CMS support forum
You are not logged in. Register | Login | Help
- Topics: Active | Unanswered
Re: Adventures in Linux Land
sed and awk are amazing tools but I confess to using them so infrequently that I have to search for potted examples to bastardize to suit my needs.
Regular expressions can be quite scary if you’re doing advanced things like negative lookahead/behind, but thankfully most of the times only a handful of simpler expressions are necessary.
The smd plugin menagerie — for when you need one more gribble of power from Textpattern. Bleeding-edge code available on GitHub.
Txp Builders – finely-crafted code, design and Txp
Online
Re: Adventures in Linux Land
I’ll break down the script into parts, so it makes more sense…usual house rules: let me know if anything isn’t clear.
Here’s the script in full:
if \
[[ $(awk -F= '$1=="VERSION_CODENAME" { print $2 ;}' /etc/os-release) = "bullseye" ]] \
; then \
export DEBIAN_FRONTEND=noninteractive \
&& apt update \
&& apt -y dist-upgrade \
&& apt -y full-upgrade \
&& apt -y autoclean \
&& apt -y autoremove --purge \
&& sed -i 's/bullseye/bookworm/g' /etc/apt/sources.list \
&& sed -i -- 's/bullseye/bookworm/g' /etc/apt/sources.list.d/*.list \
&& apt clean \
&& apt update \
&& apt -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" dist-upgrade \
&& apt -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" full-upgrade \
&& apt -y autoclean \
&& apt -y autoremove --purge \
&& reboot \
; fiThe first and last lines are an if statement, which breaks down like this:
if \
[[ CONDITION ]] \
; then \
DO STUFF \
; fiThe condition is:
$(awk -F= '$1=="VERSION_CODENAME" { print $2 ;}' /etc/os-release) = "bullseye"The $(…) wrapper can be interpreted as “the result of running the contents” and we can strip that away, which leaves:
awk -F= '$1=="VERSION_CODENAME" { print $2 ;}' /etc/os-releaseThe last part is a file with operating system release info. This is a standard file on Debian and derivates, and it contains a bunch of lines with one chunk of info per line, similar to this:
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"High level: the awk part is checking for the line beginning VERSION_CODENAME, and if the value is bookworm then the condition is true. This means the ‘do stuff’ part can proceed. If the condition is false (i.e., the line doesn’t exist, or the value is set to something that’s not bookworm, the ‘do stuff’ part won’t happen.
export DEBIAN_FRONTEND=noninteractive is setting a name-value pair as part of the environment. If you export a variable, it is passed to child processes. If you set a variable without using export (e.g. DEBIAN_FRONTEND=noninteractive), it won’t be passed to child processes. The upgrade procedure for Debian kicks off a lot of processes, so it’s more efficient to tell it to “crack on m8 lol” and just get the job done.
sed -i 's/bullseye/bookworm/g' /etc/apt/sources.list is using sed to find all instances of bullseye and replace with bookworm in the file /etc/apt/sources.list. The -i part does it in-place (or inline), so it makes changes to the file specified, rather than making changes to a newly-created file and leaving the original intact.
sed -i -- 's/bullseye/bookworm/g' /etc/apt/sources.list.d/*.list is similar to the above, but it’s a wildcard for *.list files, and I honestly can’t recall why -- is in there. It might have something to do with the wildcard.
apt -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" dist-upgrade is essentially telling apt to run with options with regards to config files. --force-confdef means “[I]f a conffile has been modified and the version in the package did change, always choose the default action without prompting.”, and --force-confold means “[I]f a conffile has been modified and the version in the package did change, always keep the old version without prompting, unless the --force-confdef is also specified, in which case the default action is preferred.”
Or, another way, don’t break my configs if I’ve changed it from the default.
In apt land, dist-upgrade and full-upgrade are the same thing, and I can probably remove one of them. It’s been a while since I used this script, so mea culpa there. I think I only made this script for the period where DigitalOcean had Debian 11 but not Debian 12, and I used it on a vanilla Debian 11 to force it to be Debian 12 after a reboot or two.
Bloke wrote #338056:
sed and awk are amazing tools but I confess to using them so infrequently that I have to search for potted examples to bastardize to suit my needs.
This, so much. It’s where a build guide comes into its own, once you’ve got a working version you can crib it for another task.
Compiling software is very similar in that respect. I started with compiling Nginx from source some years ago, and that gave me a framework for other software compiling. Take this example for ImageMagick:
imagemagick_source_version="7.1.1-39" \
&& sudo mkdir -p \
/opt/imagemagick/ \
&& rm -rf \
"$HOME"/imagemagick-source \
"$HOME"/imagemagick-source.tar.gz \
&& mkdir -p \
"$HOME"/imagemagick-source \
&& curl -Lo \
"$HOME"/imagemagick-source.tar.gz \
https://github.com/ImageMagick/ImageMagick/archive/"$imagemagick_source_version".tar.gz \
&& tar xzvf \
"$HOME"/imagemagick-source.tar.gz \
-C "$HOME"/imagemagick-source \
&& cd "$HOME"/imagemagick-source/ImageMagick-"$imagemagick_source_version" \
&& ./configure \
--disable-docs \
--prefix=/opt/imagemagick \
&& make -j"$(nproc)" \
&& sudo make -j"$(nproc)" install \
&& sudo ldconfig /usr/local/lib \
&& sudo make -j"$(nproc)" clean \
&& cd "$HOME" \
&& rm -rf \
"$HOME"/imagemagick-source \
"$HOME"/imagemagick-source.tar.gz \
&& echo -e '\n=> Checking ImageMagick version...' \
&& /opt/imagemagick/bin/identify -versionThe workflow there is: set the version number as a variable, build the destination scaffold, prep the build area scaffold, get the source, expand the source, configure the source, compile the source, install the compiled source, update shared libraries, clear the build area, check it’s working. When there’s a new version of ImageMagick, I update the version number and rebuild it.
That’s about 30 lines of code. A PHP compile is about 90 lines. An Nginx compile is about 240 lines because it uses a bunch of external modules and other config settings, but the workflow is largely the same.
A Percona MySQL 8.4 compile is about 25 lines, and takes hours to finish (at least on ARM64) because it uses a different build system. From memory, the Percona install I did over the weekend on a 4-core Hetzner ARM64 took 5 hours.
Last edited by gaekwad (2024-10-24 15:29:54)
Offline
#15 2024-12-12 15:37:40
- Algaris
- Member
- From: England
- Registered: 2006-01-27
- Posts: 565
Re: Adventures in Linux Land
I learn’t about Phasing today while upgrading one of my Ubuntu servers. Having never come across this I had to Google it.
Ubuntu Server:
The following upgrades have been deferred due to phasing: apport apport-core-dump-handler python3-apport python3-problem-report sosreport
What is Phasing in Ubuntu?
Phasing is a process used by Ubuntu to gradually roll out updates to users. Instead of making updates available to all users immediately, they are rolled out in phases to a subset of users. This approach helps identify potential issues early on and ensures stability for the majority of users. If no significant issues are reported, the updates become available to all users over time.
See also: Ask Ubuntu: What are phased updates?
Offline
Re: Adventures in Linux Land
Algaris wrote #338506:
Phasing is a process used by Ubuntu to gradually roll out updates to users. Instead of making updates available to all users immediately, they are rolled out in phases to a subset of users. This approach helps identify potential issues early on and ensures stability for the majority of users. If no significant issues are reported, the updates become available to all users over time.
Entitled ‘get off my lawn’ nerd moan incoming…
It’s stuff like this that really shows the difference between Ubuntu and e.g. Debian. I cannot imagine how many frustrated users out there were / are scratching their heads as to why such-and-such update wasn’t forthcoming when the Ubuntu packages site shows an updated version.
Anecdotally, Ubuntu (20.04) seems to have more frequent / regular updates than Debian (12). When I log into the few remaining Ubuntu servers I have in the fleet, there are almost always updates to apply. There’s a daily update check & install run in the small hours, and there’s still stuff to apply more often than not. I don’t object to installing updates – I like updates – but ffs let me have the updates if there are updates. I’m already on the free tier (5 seats; we use 2 of 5) for Ubuntu Pro and their ESM updates…here’s another frigging hoop to jump through to get the updates.
You’ve reminded me about another thing I was going to mention (file under ‘thin end of the wedge’). I wanted to install btop from packages on Debian & Ubuntu as a precursor to figuring out a source compile if I want the latest & greatest. Debian was absolutely painless:
$ sudo apt update && sudo apt install -y btop
Get:1 file:/etc/apt/mirrors/debian.list Mirrorlist [39 B]
Get:5 file:/etc/apt/mirrors/debian-security.list Mirrorlist [27 B]
Hit:6 http://repo.percona.com/pmm2-client/apt bookworm InRelease
Hit:7 http://repo.percona.com/prel/apt bookworm InRelease
Hit:9 http://repo.percona.com/ps-84-lts/apt bookworm InRelease
Hit:8 http://security.debian.org bookworm-security InRelease
Hit:10 http://repo.percona.com/telemetry/apt bookworm InRelease
Hit:11 https://repos-droplet.digitalocean.com/apt/droplet-agent main InRelease
Hit:12 https://repos.insights.digitalocean.com/apt/do-agent main InRelease
Hit:2 http://mirrors.digitalocean.com/debian bookworm InRelease
Get:13 https://apt.syncthing.net syncthing InRelease [15.7 kB]
Hit:3 http://mirrors.digitalocean.com/debian bookworm-updates InRelease
Hit:4 http://mirrors.digitalocean.com/debian bookworm-backports InRelease
Fetched 15.7 kB in 1s (13.1 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
btop
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 465 kB of archives.
After this operation, 1416 kB of additional disk space will be used.
Get:1 file:/etc/apt/mirrors/debian.list Mirrorlist [39 B]
Get:2 http://mirrors.digitalocean.com/debian bookworm/main amd64 btop amd64 1.2.13-1 [465 kB]
Fetched 465 kB in 0s (3456 kB/s)
Selecting previously unselected package btop.
(Reading database ... 123475 files and directories currently installed.)
Preparing to unpack .../btop_1.2.13-1_amd64.deb ...
Unpacking btop (1.2.13-1) ...
Setting up btop (1.2.13-1) ...
Processing triggers for mailcap (3.70+nmu1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for man-db (2.11.2-2) ...Ubuntu LTS (20.04, or ‘LTS minus 2’ but still supported):
$ sudo apt update && sudo apt install -y btop
Hit:1 http://repo.percona.com/pmm2-client/apt focal InRelease
Hit:2 http://mirrors.digitalocean.com/ubuntu focal InRelease
Hit:3 http://mirrors.digitalocean.com/ubuntu focal-updates InRelease
Hit:4 http://repo.percona.com/prel/apt focal InRelease
Hit:5 http://repo.percona.com/ps-80/apt focal InRelease
Hit:6 http://repo.percona.com/telemetry/apt focal InRelease
Hit:7 http://repo.percona.com/tools/apt focal InRelease
Hit:8 https://repos-droplet.digitalocean.com/apt/droplet-agent main InRelease
Hit:9 http://mirrors.digitalocean.com/ubuntu focal-backports InRelease
Hit:10 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:11 https://repos.insights.digitalocean.com/apt/do-agent main InRelease
Get:12 https://esm.ubuntu.com/apps/ubuntu focal-apps-security InRelease [7565 B]
Get:13 https://apt.syncthing.net syncthing InRelease [15.7 kB]
Get:14 https://esm.ubuntu.com/apps/ubuntu focal-apps-updates InRelease [7456 B]
Get:15 https://esm.ubuntu.com/infra/ubuntu focal-infra-security InRelease [7450 B]
Get:16 https://esm.ubuntu.com/infra/ubuntu focal-infra-updates InRelease [7449 B]
Fetched 45.6 kB in 4s (10.9 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree
Reading state information... Done
No apt package "btop", but there is a snap with that name.
Try "snap install btop"
E: Unable to locate package btopThe phrase No apt package "[…]", but there is a snap with that name is the pull quote. There’s a push toward snap for…so much stuff on Ubuntu. I realise it has advantages, but for nerdists like me who like change control, visibility and not having heaps of mounted opaque bullshit scattered around the filesystem, it’s so tiring. I deliberately stuck with Ubuntu 20.04 to minimise the snap stuff littering my servers, but I’m losing the battle there, clearly.
This is the push I needed to write the project strategy doc for the next two years. I’ll share it on the forum when it’s done. I will be re-applying to our pals at DigitalOcean to ask for more server magic beans, so the timing is good.
Spoiler: 2x Ubuntu servers will be decommissioned next year in favour of Debian.
Offline
Re: Adventures in Linux Land
Not to brag here, but my website server is running a cutting-edge 3.10 kernel. The C compiler is fully ISO C90 compliant, too. I thus avoid these pesky update concerns.
Offline
Re: Adventures in Linux Land
skewray wrote #338515:
I thus avoid these pesky update concerns.
I love the nihilism in this, by the way.
Offline
#19 2024-12-16 14:42:02
- Algaris
- Member
- From: England
- Registered: 2006-01-27
- Posts: 565
Re: Adventures in Linux Land
skewray wrote #338515:
Not to brag here, but my website server is running a cutting-edge 3.10 kernel. The C compiler is fully ISO C90 compliant, too. I thus avoid these pesky update concerns.
Now that is pretty impressive and way beyond my knowledge or comfort zone.
gaekwad wrote #338528:
I love the nihilism in this, by the way.
🤣
Offline
#20 2024-12-16 14:46:27
- Algaris
- Member
- From: England
- Registered: 2006-01-27
- Posts: 565
Re: Adventures in Linux Land
So, the other day I learnt the difference between /var and /srv.
The
/vardirectory contains variable data files that change frequently as the system operates. This includes files like logs, caches, and temporary files generated by running applications and system processes.
The
/srvdirectory is intended to store data for services that are provided by the system. This is service-specific data that is served to clients or users, such as websites, FTP files, or any other network services.
Looks like I’ve been doing it wrong using /var/www/html. Instead I should have been using /srv/www/html. Not that it matters I guess, but I’m going to use /srv/www/html from now on.
Offline
Re: Adventures in Linux Land
I use /var/www. Mainly out of habit. Maybe when I rebuild the server I’ll switch to /srv.
The smd plugin menagerie — for when you need one more gribble of power from Textpattern. Bleeding-edge code available on GitHub.
Txp Builders – finely-crafted code, design and Txp
Online
Re: Adventures in Linux Land
Algaris wrote #338547:
So, the other day I learnt the difference between
/varand/srv.Looks like I’ve been doing it wrong using
/var/www/html. Instead I should have been using/srv/www/html. Not that it matters I guess, but I’m going to use/srv/www/htmlfrom now on.
For what it’s worth, I use /var/www/. But then I’m also the person who compiles external stuff to /opt/ instead of the typical /etc/.
Further bedtime reading: en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
Offline
Re: Adventures in Linux Land
I am curious, why do you keep WWW assets outside of publisher’s private /home directory (/usr/home in FreeBSD).
Offline
Re: Adventures in Linux Land
Vienuolis wrote #338551:
I am curious, why do you keep WWW assets outside of publisher’s private /home directory (/usr/home in FreeBSD).
In my case, I have multi-tenant servers (i.e., >1 system user) and the www assets aren’t assigned to a system user. The www assets are owned by the web server, with access given to the PHP-FPM pool user. This keeps the system user (e.g., alice) and web server user (e.g., www-data) air gapped.
I have noticed web admin system control panels (e.g., cPanel & WHM) tend to steer admins towards using /home for www stuff, but that scares me.
Offline