I recently decided it was time to finally get around to upgrading my old 1.16 Kubernetes cluster to something modern. I mean, I’d been applying incremental upgrades via kops for years, but it was now getting to the point where moving forward would be difficult without updating the practices used to manage and run the cluster.

So my cluster consists of a few minor bits of work I do on the side:

• A web site for the home school coop my wife chairs
• A web site for a local state politician
• A couple Minecraft servers that run from time to time
• A few other miscellaneous web sites for various minor things of my own
• A small local business web site
• A ticketing web site for an annual walk-through nativity I help with
• And some tools I use for various other silly things at home, such as an accounting tool I wrote to track my kid’s allowances.

All this is running on kubernetes, which I used to run on shared hosting. I originally set this cluster up as a means of teaching myself how kubernetes works and what it’s good for. After I did it, I kept it because it was easier than not keeping, but also because kubernetes is really nice for providing a good way of tracking what you actually have running. When I ran on a shared hosting provider, I was never sure which things I was using were still setup or how they worked without doing a lot of digging.

Anyway, I mostly want to make this post to highlight one aspect of this migration from the old cluster to the shiny new one. I have a number of stateful services. I’ve slowly been trying to phase these out, but something like a Minecraft server is difficult to run without a disk storing the world files and backups. So, migrating data from the old cluster to the new one is a slight challenge. I mean, you can just go into the nodes, find the files, tar them up, move them, and untar them on the other server, but finding which node has the volume mounted, finding the name of the directory, and then running all the commands is super-tedious. However, kubernetes can easily provide you with everything needed to find the locations automatically to make this happen in a single line command.

I’ve written a quick little command to do just this. Note that this command makes some assumptions that almost certainly don’t work in every environment. But if you’re working with kops on AWS using Ubuntu hosts, I’m pretty sure this will work just fine for you.

Before running it, you’ll need to make sure you have your kube config setup with a context for each cluster you are working with and that you have currently authorized credentials for both. Set the $FROM_CLUSTER and $TO_CLUSTER settings below to the name of each context name.

You run it like this:

kube-transfer old-ns/old-pod-prefix/old-pv new-ns/new-pod-prefix/new-pv

It finds the names and external DNS entries for each of the nodes in each cluster, finds which node the first pod matching the prefix is on in that cluster and the external hostname it is currently running on. It then finds the name of the PersistenVolume associated with the given PersistenVolumeClaim of the drive you want to copy. Finally, it performs a tar cf on the old machine and a tar xf on the new machine and pipes data from one into the other.

This is not the most super-efficient way to do this, but it does the job for me. I learned a few more details about how kubernetes on AWS using kops works in the process and thought I’d share.

It is pretty easy, in my use, to guarantee that a service is shutdown or not being modified to guarantee that I’m not catching files that are in an inconsistent state. The data I’m migrating is changed every now then, but not often. The one situation where that wasn’t really the case was the minecraft servers. Minecraft servers are often fiddling with their files, even when idle. This solution requires the pods be live to work. To avoid problems, therefore, I just took the minecraft servers off line by replacing their command with my favorite standby command:

command: [ sleep, '10000' ]

With that in place to replace the entrypoint in the container, I guarantee the minecraft server is offline while I copy files. Then I just delete the deployment on the old cluster and delete that line on the new cluster. (Or in my case, delete the deployment in both clusters because we aren’t doing anything with our minecraft server at the moment.)

Here is a snapshot of the command I’ve written to do this in case it’s useful to anyone:

#!/usr/bin/env perl

# Copyright © 2021 Andrew Hanenkamp
# 
# Permission is hereby granted, free of charge, to any person obtaining a copy of
# this software and associated documentation files (the “Software”), to deal in
# the Software without restriction, including without limitation the rights to
# use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
# the Software, and to permit persons to whom the Software is furnished to do so,
# subject to the following conditions:
# 
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
# FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
# COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
# IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

use v5.30;
use warnings;

use DDP;
use JSON qw( decode_json );

my $FROM_CLUSTER = 'OLD-CLUSTER-NAME';
my $TO_CLUSTER = 'NEW-CLUSTER-NAME';

sub ns_name_and_pvc {
    my ($arg) = @_;
    my ($ns, $name, $pvc) = split m{/}, $arg;
    ($name, $pvc, $ns) = ($ns, $name, $pvc) if !$name;
    $ns //= 'default';
    return ($ns, $name, $pvc);
}

sub kubectl {
    my $cluster = shift @_;

    open my $cfh, '-|', 'kubectl', "--context=$cluster", @_, '-ojson'
        or die "cannot open kubectl: $!";

    my $json = decode_json(do { local $/; <$cfh> });

    close $cfh
        or die "cannot finish kubectl: $!";

    return $json;
}

sub node_map {
    my ($cluster) = @_;

    my $json = kubectl($cluster, 'get', 'node');

    return map {
        my ($ext) = grep { $_->{type} eq 'ExternalDNS' } @{ $_->{status}{addresses} };
        my ($int) = grep { $_->{type} eq 'Hostname' } @{ $_->{status}{addresses} };
        $int->{address} => $ext->{address};
    } @{$json->{items}}
}

sub get_pod {
    my ($cluster, $ns, $name) = @_;

    my $json = kubectl($cluster, 'get', 'pod', '-n', $ns);

    for my $pod (@{ $json->{items} }) {
        if ($pod->{metadata}{name} =~ /^\Q$name\E-/) {
            return $pod
        }
    }

    return;
}

sub get_pv {
    my ($cluster, $ns, $pvc) = @_;

    my $json = kubectl($cluster, 'get', 'pvc', '-n', $ns, $pvc);

    return $json->{spec}{volumeName};
}

sub get_volume_dir {
    my ($host, $pv) = @_;

    open my $sshfh, '-|', 'ssh', 'ubuntu@'.$host, 'mount'
        or die "cannot open ssh to $host: $!";

    my $final;
    while (<$sshfh>) {
        my @fields = split /\s+/;
        if ($fields[2] =~ /\Q$pv\E/) {
            $final = $fields[2];
            last;
        }
    }

    close $sshfh
        or die "cannot finish ssh to $host: $!";

    return $final;
}

my ($from, $to) = @ARGV;

my ($from_ns, $from_name, $from_pvc) = ns_name_and_pvc($from);
my ($to_ns, $to_name, $to_pvc) = ns_name_and_pvc($to);

my %from_nodes = node_map($FROM_CLUSTER);
my %to_nodes = node_map($TO_CLUSTER);

my $from_pod = get_pod($FROM_CLUSTER, $from_ns, $from_name);
my $to_pod = get_pod($TO_CLUSTER, $to_ns, $to_name);

my $from_node_name = $from_pod->{spec}{nodeName};
my $to_node_name = $to_pod->{spec}{nodeName};

my $from_host = $from_nodes{ $from_node_name };
my $to_host = $to_nodes{ $to_node_name };

my $from_pv = get_pv($FROM_CLUSTER, $from_ns, $from_pvc);
my $to_pv = get_pv($TO_CLUSTER, $to_ns, $to_pvc);

my $from_dir = get_volume_dir($from_host, $from_pv);
my $to_dir = get_volume_dir($to_host, $to_pv);

say "Copying $from_host to $to_host ...";
say "Using folder $from_dir to $to_dir ...";

open my $cfh, '-|', 'ssh', "ubuntu\@$from_host", 'sudo', 'tar', '-C', $from_dir, '-cf', '-', './'
    or die "cannot copy from $from_host: $!";

open my $xfh, '|-', 'ssh', "ubuntu\@$to_host", 'sudo', 'tar', '-C', $to_dir, '-xf', '-'
    or die "cannot copy from $to_host: $!";

while (<$cfh>) {
    print $xfh $_;
}

close $cfh
    or die "cannot finish copying from $from_host: $!";

close $xfh
    or die "cannot finish copying to $to_host: $!";

Cheers.