Blog / DevOps / Dockerhub and pull limits: cache some insurance

Dockerhub and pull limits: cache some insurance

Like many of you we use an indeterminate number of dockerhub container pulls per hour. The base layers of something? The CI? Bits and bobs inside our GKE? One area of particular concern: we host applications from our customers, what base layers do they use?

Our general strategy is to not use upstream containers, to build our own. For security. For control. But, the exceptions are, for well-known projects with strong security cultures. E.g. ‘FROM ubuntu:20.04‘ is allowed, ‘FROM myname‘ is not.

We run our own image registry, it authenticates (via JWT) to our Gitlab. It works very well with our CI, with our production systems. So our use of the Dockerhub registry is probably minimal. But its hard to say, we run a lot of CI jobs, we run pre-emptible nodes in GKE, they can(and do) restart often. So we have some risk with the new Rate-Limits announced.

GKE has a mirror of some images, and we do use that (both with a command-line to kubelet on the nodes, but also by injected into our CI runners in case they still use dind). Recently gitlab added some documentation on using registry mirror with dind. We inject this environment variable into the runner services just-in-case.

DOCKER_OPTS: "--registry-mirror=https://mirror.gcr.io --registry-mirror=https://dockerhub-cache.agilicus.ca"

Now its time to bring up a cache. First I create an account in Dockerhub. This gets us 200/6hr limit. Then, I create a pull-through cache. You can see the below code in our github repo, but, first a Dockerfile. I create a non-root-user and run as that. Effectively this gives us an upstream-built registry, in our own registry, with a non-root user.

FROM registry
ENV REGISTRY_PROXY_REMOTEURL="https://registry-1.docker.io"
RUN : \
&& adduser --disabled-password --gecos '' web
USER web

And now for the wall of YAML. This is running in my older cluster which does not have Istio, the one that just exists to host our gitlab etc. I use my trick to share an nginx-ingress while using a different namespace, reducing the need for public IPv4. I give this a 64GiB PVC, a unique name (dhub-cache), I set some environment variables. And we let it fly.

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: dhub-cache
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
fluentbit.io/parser: nginx
kubernetes.io/tls-acme: "true"
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
spec:
tls:
- hosts:
- "dhub-cache.MYDOMAIN.ca"
secretName: dhub-cache-tls
rules:
- host: "dhub-cache.MYDOMAIN.ca"
http:
paths:
- path: /
backend:
serviceName: dhub-cache-ext
servicePort: 5000
---
kind: Service
apiVersion: v1
metadata:
name: dhub-cache-ext
namespace: default
spec:
type: ExternalName
externalName: dhub-cache.dhub-cache.svc.cluster.local
ports:
- port: 5000
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: dhub-cache
namespace: dhub-cache
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 64Gi
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: dhub-cache
namespace: dhub-cache
automountServiceAccountToken: false
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: dhub-cache
name: dhub-cache
labels:
name: dhub-cache
spec:
replicas: 1
selector:
matchLabels:
name: dhub-cache
template:
metadata:
labels:
name: dhub-cache
spec:
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
serviceAccountName: dhub-cache
automountServiceAccountToken: false
imagePullSecrets:
- name: regcred
containers:
- image: MYREGISTRY/dhub-cache
name: dhub-cache
imagePullPolicy: Always
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
ports:
- name: http
containerPort: 5000
livenessProbe:
httpGet:
path: /
port: 5000
periodSeconds: 30
timeoutSeconds: 4
failureThreshold: 4
readinessProbe:
httpGet:
path: /
port: 5000
periodSeconds: 20
timeoutSeconds: 4
failureThreshold: 4
env:
- name: REGISTRY_PROXY_REMOTEURL
value: "https://registry-1.docker.io"
- name: REGISTRY_PROXY_USERNAME
value: "MY_USER"
- name: REGISTRY_PROXY_PASSWORD
value: "MY_USER_TOKEN"
volumeMounts:
- mountPath: /var/lib/registry
name: dhub-cache
volumes:
- name: dhub-cache
persistentVolumeClaim:
claimName: dhub-cache
---
apiVersion: v1
kind: Service
metadata:
namespace: dhub-cache
name: dhub-cache
labels:
name: dhub-cache
spec:
type: NodePort
ports:
- port: 5000
targetPort: 5000
selector:
name: dhub-cache
---
apiVersion: v1
data:
.dockerconfigjson: MYINTERNALREGISTRYINFO
kind: Secret
metadata:
name: regcred
namespace: dhub-cache
type: kubernetes.io/dockerconfigjson

Leave a Reply

Your email address will not be published.