Docker – detecting out-of-date containers

Amazingly there isn’t a built-in mechanism to check if docker containers are up-to-date, you can only do a pull and see if anything is downloaded. By doing so the images are anyway updated, so if the container is recreated for any reason it will also use those new images.

There’s a tool called Skopeo which should do this, but I could not get it to run (the Go dependencies are too complex on my older Linux installation).

There’s also Watchtower which automates the checking but does not ensure images aren’t downloaded.

I decided to write my own bash script to read the local docker container / image details and compare them against the latest version in the repo using the API. This turned out to be pretty complex – in the case of Docker Hub because the URLs and authentication aren’t very intuitive, and in the case of Github because they use the newer manifest schema which requires multiple calls to get the information.

However… I got a script which works, confirming whether local images are up-to-date (at least for my set of containers).

My scenario is that I run this script each week with the settings to check all containers and only output those that are out-of-date. Any output is then sent to me by email so I can decide when to update.

It can be found on Github: https://github.com/pseudocoder/docker-check

Example output from my media server

% docker-check -a
paperless-ng_gotenberg_1 (thecodingmachine/gotenberg) is up-to-date
mariadb (linuxserver/mariadb:latest) is up-to-date
wireguard (ghcr.io/linuxserver/wireguard) is up-to-date
heimdall (linuxserver/heimdall:latest) is up-to-date
plex (linuxserver/plex:latest) is up-to-date
wordpress-gee-family (wordpress:latest) is up-to-date
wordpress-pseudocode (wordpress:latest) is up-to-date
cops (linuxserver/cops) is up-to-date
paperless-ng_webserver_1 (jonaswinkler/paperless-ng:latest) is up-to-date
filebrowser (filebrowser/filebrowser:latest) is up-to-date
deluge (linuxserver/deluge) is up-to-date
filerun (afian/filerun:latest) is up-to-date
calibre-web (linuxserver/calibre-web) is up-to-date
influxdb2 (influxdb:latest) is up-to-date
syncthing (linuxserver/syncthing:latest) is up-to-date
grafana (grafana/grafana:latest) is up-to-date
paperless-ng_db_1 (postgres:13) is up-to-date
paperless-ng_broker_1 (redis:6.0) is up-to-date
paperless-ng_tika_1 (apache/tika) is up-to-date
portainer (portainer/portainer-ce) is up-to-date

Docker Compose – automated updates

Using docker-compose makes updating images and containers very easy – a simple command downloads any new images:

docker-compose pull

And another stops, updates and restarts containers:

docker-compose up -d

But what if you want to know about updates, without actually doing them – for example, to do a daily check but only update during non-critical times or when you’re around to fix any issues?

Unfortunately, neither docker or docker-compose seem to provide a way of reporting out-of-date images – but a simple pair of commands can do something similar.

docker-compose pull -q
docker images | fgrep '<none>'

This pulls any new images, quietly without reporting anything, and then looks for any images not associated with a tag – since the new images will now have the “latest” tag, the older ones show up.

After the update (docker-compose up -d) you should also delete the old images:

docker image prune -f

Probably there’s a better way to do this, but it works fine on my media centre to tell me of updates while still having control over when they are applied.

TICK stack

I’ve been using the TelegrafInfluxDBChronograf-Kapacitor stack for a couple of months at home and at work, for monitoring the state of devices, process and home automation.

We actually I’ve been using theĀ Telegraf-InfluxDB-Grafana stack – I have no idea why they decided to create Chronograf as a fork of Grafana, but it really is pretty rubbish in comparison.

That said, overall the solution is brilliant – Telegraf is pretty good at grabbing stats from your servers, and is highly configurable (at least on Linux – the Windows version could do with some work). The only area that really lets it down is the inability to sum up stats when monitoring processes, so anything that spawns child processes tends to make a mess of the stats.

Influx is very easy to use – the line protocol mechanism for adding data with a simple web request makes it very accessible, with a simple bash script and some sed reformatting able to create a data dump very easily. It seems pretty disk intensive, but I guess that’s always going to be the case with something writing datapoints every minute. Getting used to a timeseries database takes a bit of patience, with pretty limited options for querying, but it’s worth it for the performance and space saving. The only significant lack here is handling of offsets – it’s a very clear use-case to compare timeseries from two equivalent points in time, and surprising it isn’t supported.

Then Grafana tops it off with flexible and powerful visualisation.

I’d recommended anyone who is looking after any sort of IT system to have a play around with it.

Microsoft Spam !

You wouldn’t expect MS to send spam? No, neither would I – surely no major tech company is that dumb?

Well, they seem to have lost the plot on their Surface and OneNote emails.

Register your new Surface and you get a series of “helpful” emails telling you how to use it. Hmm, I don’t need this crap, if I want to know something I’ll Google it (yeah, not Bing!) – where’s the unsubscribe button?

OMG – no unsubscribe!

Must be something in my MS account – but no, it’s all set to the correct “don’t send me crap” settings. The not-well-known Profile Center also looks clean.

Grrr.

And then, I get an email from OneNote.

“Notebooks are social. So pass it on.” “Forward this email to family and friends so they can join the party!”

Ah, no. My notebook is absolutely not f*****g social, it’s my notebook, if I wanted it to be social I would have said so but the default setting should be as private as possible.

Microsoft, get a grip – I’m pretty sure most people who buy a Surface are not aiming to turn it into a social hub, and are quite happy to read the documentation in their own good time without being spammed.

Big Stumble by StumbleUpon

One year ago:

“Our badge/widget isn’t compatible with https sites; there are no plans at this time to change it.”

https://getsatisfaction.com/stumbleupon_help_center/topics/badge_widget_https_mixed_content_problem

Wow, how to ignore an ever growing set of your content base, specifically those most up-to-date and therefore probably interesting, in one sweeping statement.

Must be trying to compete with Reddit for the foot shooting prize.

Cute File Browser

I found this very neat jQuery based file browser:

http://tutorialzine.com/2014/09/cute-file-browser-jquery-ajax-php/

This uses a simply PHP script to form JSON output defining the filesystem, however in my case I want to serve files on a Windows PC using just static content. This is relatively easy to code – we just need to create the file listing in advance, e.g. into something called files.json, and modify the javascript to retrieve this instead of scan.php.

My batch script to do that is listed here – it simply scans recursively from the current directory creating the necessary JSON on the output; you can pipe it to the appropriate location for the files.json output.

@echo off
setlocal enableextensions disabledelayedexpansion

set p=%1
set n=%2
set comma=
set oldcomma=X

call:funcdo "%1" "%2"

endlocal
goto:eof

:funcdo
setlocal
set "p=%~1"
set "n=%~2"

echo {"name":"%n%","type":"folder","path":"%p%","items":[

set oldcomma=%oldcomma%%comma%
set comma=N

for /d %%d in (*) do (
    cd "%%d"
    call:funcdo "%p%/%%d" "%%d"
    echo ,
    cd ..
)

for /f "tokens=*" %%f in ('dir /b *.divx *.mpg *.mpeg *.avi *.mkv *.mp4 *.wmv 2^>nul ^| sort') do (
    echo {"name":"%%f","type":"file","path":"%p%/%%f","size":"%%~zf"},
)

echo {}]}

set comma=%oldcomma:~-1%
set oldcomma=%oldcomma:~0,-1%
endlocal
goto:eof

It’s not very polished and could no doubt be improved, but it works.

Mail forwarding blocked by SPF

Some time ago I moved away from Yahoo mail to my own hosted address, and used the forwarding function to pass through any residual mail. This worked fine for several years, but now my host has implemented strict SPF control and a lot of the forwarded mails get rejected.

Unfortunately Yahoo doesn’t give any sophisticated options to handle this, eg. there are a number of options they could have implemented for forwarding from the Yahoo address without looking like it’s spam:

  • normal forwarding, or as an attachment
  • putting the sender in the Reply-To header
  • or even a line of text saying who the original sender was

The only way I found to get around this was to stop the forwarding, and use a PHP script to periodically check my Yahoo account via POP and move the mail to my new IMAP account. This actually works fine, so in case it’s of use to others here is the script I use:

<html><body><pre>
<?php 

docopy("{pop.mail.yahoo.com:995/pop3/ssl}INBOX","Yahoo email","Yahoo password",
       "{Target imap:143}INBOX", "Target login", "Target password");

function docopy($smailbox, $suser, $spwd, $tmailbox, $tuser, $tpwd) {
  echo "--------------------\nCopying mail from $smailbox:$suser to $tmailbox:$tuser\n--------------------\n";

  if (!($source = imap_open($smailbox, $suser, $spwd))) {
    echo "Connect to source failed";
    exit(1);
  }

  echo "Connected source: $source\n";

  $msgcount = imap_num_msg($source);

  if ($msgcount == 0) {
    echo "No messages to process\n\n";
    imap_close($source);
    return;
  }

  echo "Messages: $msgcount\n";

  if (!($target = imap_open($tmailbox, $tuser, $tpwd))) {
    echo "Connect to target failed";
    imap_close($source);
    exit(1);
  }

  echo "Connected target: $target\n";

  for ($msg = 1; $msg <= $msgcount; $msg++) {
    echo "\nProcessing message $msg\n";

    $header = imap_fetchheader($source, $msg);
    $body = imap_body($source, $msg);

    if (!imap_append($target, $tmailbox, $header . $body)) {
      echo "Saving message failed";
      imap_close($source);
      imap_close($target);
      exit(1);
    }

    echo "Message saved\n";

    if (!imap_delete($source, $msg)) {
      echo "Deleting message failed";
      imap_close($source);
      imap_close($target);
      exit(1);
    }

    echo "Message deleted\n";
  }

  imap_expunge($source);

  imap_close($source);
  imap_close($target);

  echo "Completed copy\n\n";
}
?>
</pre></body></html>

Obviously you have to fill in the bits for Yahoo email, Yahoo password, Target imap etc. It should also work for other POP or IMAP sources/targets, but I haven’t tried it.

The function can be called several times if you have more than one account to copy, in my case two works fine.

It’s written to give some level of output if you run it through a web server, then once it’s working you can schedule it to run every few minutes via cron or similar.

App “upgrade” = Ransomware ?

I’ve been a user of SplashID password manager on my iPhone, iPad and Windows machine for a couple of years and found it pretty good. I paid $10 to use wifi sync which seemed reasonable value at the time.

Now, the developer has created an “upgraded” version which requires a $20 annual fee to use wifi sync. Of course they can offer this service and maybe some people will buy it, but I’m happy with my version thanks.

BUT – the version on my iDevices is linked to the App Store, and will automatically update if I enable this option or if I ever click the “Update All” button; there’s no easy way to stop this, and thus at some point my paid for functionality will inevitably be deleted by the developer.

In my view this is ransomware (or possibly theft). The developer almost certainly doesn’t intend it this way, but as a consumer I don’t care what their theory is, I care what I can use and what it costs me. When I buy something I don’t expect free upgrades, although they are often included, but I do expect it to not be remotely disabled and an additional fee demanded.

Of course there is a very simple solution – leave the application on the App Store and create a new one explicitely as “Version X”. If the old one breaks with an iOS update then fair enough, I didn’t buy perpetual support, but in the meantime don’t steal my functionality.

Posts on the SplashData forum received suggestions to use their permanent upgrade offer – so basically ransomware in email form rather than as an in-app purchase.

I also raised this with Apple but haven’t heard back yet.

British English??

I just upgraded my iPad to iOS7, and have mixed views on the changes overall. The new functionality seems in general to be useful, but the flat design seems to be less clear than the old one and in particular some of the icons are really not intuitive or clear to me.

I’ve found one thing which particularly annoys me in the Mail application though:

My iPad is set to use British English, and Trash is not a British term! It’s also inconsistent when there’s a perfectly good wastebin icon on the email screen itself.

This started a train of thought about a similar icon/term which has always bothered me:

Why is is called a Recycle Bin ?

Stuff I put in our physical recycle bin gets taken away, cut up, washed / melted, and reworked into a range of new items, which I don’t personally get back. If I expected to put in a Word document and get back an Excel with numbers from a bunch of other people maybe the analogy would be more accurate.

To be fair there are some similarities:

  • I always forget to empty both the electronic and real form of the bins until we run out of space.
  • A file is always needed just after emptying, and newspaper is always needed for surface protection the day after it is recycled.

I wonder what the correct term for this electronic store would be? Options I’ve thought of:

Attic – much better metaphore, but can’t really picture the icon for this.

Celler – not very common in British English.

Garage – having a car on the icon would clearly cause confusion!