Kooking Kontainers With Kubernetes: A Recipe for Dual-Stack Deliciousness

If you have a mild allergy to ASCII or YAML you might want to avert your eyes. You’ve been warned.

Now, lets imagine you have a largish server hanging around, not earning its keep. And on the other hand, you have a desire to run some CI pipelines on it, and think Kubernetes is the answer.

You’ve tried ‘kube-spawn’ and ‘minikube’ etc, but they stubbornly allocate just a ipv4/32 to your container, and, well, your CI job does something ridiculous like bind to ::1, failing miserably. Don’t despair, lets use Calico with a host-local ipam.

For the most part the recipe speaks for itself. The ‘awk’ in the calico install is to switch from calico-ipam (single-stack) to host-local with 2 sets of ranges. Technically Kubernetes doesn’t support dual stack (cloud networking is terrible. Just terrible. its all v4 and proxy server despite sometimes using advanced things like BGP). But, we’ll fool it!

Well, here’s the recipe. Take one server running ubuntu 18.04 (probably works with anything), run as follows, sit back and enjoy, then install your gitlab-runner.

rm -rf ~/.kube
sudo kubeadm reset -f
sudo kubeadm init --apiserver-advertise-address 172.16.0.3 --pod-network-cidr 192.168.0.0/16 
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

until kubectl get nodes; do echo -n .; sleep 1; done; echo              

kubectl apply -f \
 https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/etcd.yaml
kubectl apply -f \
 https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/rbac.yaml

curl -s https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/calico.yaml |\
  awk '/calico-ipam/ { print "              \"type\": \"host-local\",\n"
                     print "              \"ranges\": [ [ { \"subnet\": \"192.168.0.0/16\", \"rangeStart\": \"192.168.0.10\", \"rangeEnd\": \"192.168.255.254\" } ], [ { \"subnet\": \"fc00::/64\", \"rangeStart\": \"fc00:0:0:0:0:0:0:10\", \"rangeEnd\": \"fc00:0:0:0:ffff:ffff:ffff:fffe\" } ] ]"
                     printed=1
}
{
    if (!printed) {
        print $0
    }
    printed = 0;
}' > /tmp/calico.yaml

kubectl apply -f /tmp/calico.yaml

kubectl apply -f - << EOF
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           upstream
           fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        proxy . 8.8.8.8
        cache 30
        reload
        loadbalance
    }
EOF

kubectl taint nodes --all node-role.kubernetes.io/master-

kubectl create serviceaccount -n kube-system tiller
kubectl create clusterrolebinding tiller-binding --clusterrole=cluster-admin --serviceaccount kube-system:tiller
helm init --service-account tiller