# Dockerhub and pull limits: cache some insurance

- Link: https://www.agilicus.com/dockerhub-pull-limit-cache-insurance/
- Published: 2020-11-02T10:45:00+00:00
- Author: Don Bowman

Like many of you we use an indeterminate number of dockerhub container pulls per
hour. The base layers of something? The CI? Bits and bobs inside our GKE? One area
of particular concern: we host applications from our customers, what base layers
do they use?

Our general strategy is to not use upstream containers, to build our own. For security.
For control. But, the exceptions are, for well-known projects with strong security
cultures. E.g. ‘_FROM ubuntu:20.04_‘ is allowed, ‘_FROM myname_‘ is not.

We run our own image registry, it authenticates (via [JWT](https://www.agilicus.com/jwt-token-is-you-on-the-wire/))
to our Gitlab. It works very well with our CI, with our production systems. So our
use of the Dockerhub registry is probably minimal. But its hard to say, we run a
lot of CI jobs, we run pre-emptible nodes in GKE, they can(and do) restart often.
So we have some risk with the new Rate-Limits [announced](https://docs.docker.com/docker-hub/download-rate-limit/).

GKE has a mirror of some images, and we do use that (both with a command-line to
kubelet on the nodes, but also by injected into our CI runners in case they still
use [dind](https://hub.docker.com/_/docker?tab=tags)). Recently gitlab added some
[documentation](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46241/diffs)
on using registry mirror with dind. We inject this environment variable into the
runner services _just-in-case_.

    ```wp-block-code
    DOCKER_OPTS: "--registry-mirror=https://mirror.gcr.io --registry-mirror=https://dockerhub-cache.agilicus.ca"
    ```

Now its time to bring up a cache. First I create an account in Dockerhub. This gets
us 200/6hr limit. Then, I create a pull-through cache. You can see the below code
in our [github repo](https://github.com/Agilicus/dockerhub-cache), but, first a 
Dockerfile. I create a non-root-user and run as that. Effectively this gives us 
an upstream-built registry, in our own registry, with a non-root user.

    ```wp-block-code
    FROM registry

    ENV REGISTRY_PROXY_REMOTEURL="https://registry-1.docker.io"

    RUN : \
     && adduser --disabled-password --gecos '' web

    USER web
    ```

And now for the wall of YAML. This is running in my older cluster which does not
have Istio, the one that just exists to host our gitlab etc. I use my trick to [share](https://blog.donbowman.ca/2018/09/06/accessing-a-service-in-a-different-namespace-from-a-single-ingress-in-kubernetes/)
an nginx-ingress while using a different namespace, reducing the need for public
IPv4. I give this a 64GiB PVC, a unique name (dhub-cache), I set some environment
variables. And we let it fly.

    ```wp-block-code
    ---
    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      name: dhub-cache
      namespace: default
      annotations:
        kubernetes.io/ingress.class: nginx
        fluentbit.io/parser: nginx
        kubernetes.io/tls-acme: "true"
        certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    spec:
      tls:
        - hosts:
            - "dhub-cache.MYDOMAIN.ca"
          secretName: dhub-cache-tls
      rules:
        - host: "dhub-cache.MYDOMAIN.ca"
          http:
            paths:
              - path: /
                backend:
                  serviceName: dhub-cache-ext
                  servicePort: 5000
    ---
    kind: Service
    apiVersion: v1
    metadata:
      name: dhub-cache-ext
      namespace: default
    spec:
      type: ExternalName
      externalName: dhub-cache.dhub-cache.svc.cluster.local
      ports:
        - port: 5000
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: dhub-cache
      namespace: dhub-cache
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 64Gi
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: dhub-cache
      namespace: dhub-cache
    automountServiceAccountToken: false
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      namespace: dhub-cache
      name: dhub-cache
      labels:
        name: dhub-cache
    spec:
      replicas: 1
      selector:
        matchLabels:
          name: dhub-cache
      template:
        metadata:
          labels:
            name: dhub-cache
        spec:
          securityContext:
            runAsUser: 1000
            runAsGroup: 1000
            fsGroup: 1000
          serviceAccountName: dhub-cache
          automountServiceAccountToken: false
          imagePullSecrets:
            - name: regcred
          containers:
            - image: MYREGISTRY/dhub-cache
              name: dhub-cache
              imagePullPolicy: Always
              securityContext:
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: true
              ports:
                - name: http
                  containerPort: 5000
              livenessProbe:
                httpGet:
                  path: /
                  port: 5000
                periodSeconds: 30
                timeoutSeconds: 4
                failureThreshold: 4
              readinessProbe:
                httpGet:
                  path: /
                  port: 5000
                periodSeconds: 20
                timeoutSeconds: 4
                failureThreshold: 4
              env:
                - name: REGISTRY_PROXY_REMOTEURL
                  value: "https://registry-1.docker.io"
                - name: REGISTRY_PROXY_USERNAME
                  value: "MY_USER"
                - name: REGISTRY_PROXY_PASSWORD
                  value: "MY_USER_TOKEN"
              volumeMounts:
                - mountPath: /var/lib/registry
                  name: dhub-cache
          volumes:
            - name: dhub-cache
              persistentVolumeClaim:
                claimName: dhub-cache
    ---
    apiVersion: v1
    kind: Service
    metadata:
      namespace: dhub-cache
      name: dhub-cache
      labels:
        name: dhub-cache
    spec:
      type: NodePort
      ports:
        - port: 5000
          targetPort: 5000
      selector:
        name: dhub-cache
    ---
    apiVersion: v1
    data:
      .dockerconfigjson: MYINTERNALREGISTRYINFO
    kind: Secret
    metadata:
      name: regcred
      namespace: dhub-cache
    type: kubernetes.io/dockerconfigjson
    ```