DockerHub rate limits

DockerHub has a strict rate limit policy, especially for anonymous pulls.

  • for anynomyous users: 100 pulls per 6 hours per IP address.
  • for authenticated users: 200 pulls per 6 hours per IP address.
  • => if your usage exceeds these limits, your requests are denied until the six-hour window elapses.

Despite the fact that for authenticated users the rate limit is higher, these limits can easily break the CI/CD pipelines which use DockerHub as a source of base images, especially if runners aren’t ephemeral and use the same IP address for multiple builds.

There are multiple known solutions to overcome these limits:

  • Using a paid plan for DockerHub (additional cost - you either have a single limited user for CI/CD or a team plan with per-seat pricing, which is not cheap)
  • Storing base images in a private registry (additional management overhead / some additional cost)
  • Using a pull-through cache via ECR / ACR (additional cost / some management overhead)

Cloudflare Serverless Registry

An interesting alternative is Cloudflare Serverless Registry, which is effectively free and doesn’t have any rate limits.

Main advantages:

I’ve tried using this as a production-ready registry, but experienced some issues related to Cloudflare Workers limits for docker push operations. Long story short - 413 Request Entity Too Large for the image layers more than 100MB on Free and Pro plans, 200MB on Business and 500MB on Enterprise plans. BUT it doesn’t affect the usage of the registry as a pull-through cache - so let’s see how it works.

graph TD 
    title[<b>đź“ť How pull-through from upstream registry works</b>]
    A[Docker engine] -->|"docker pull &lt;r2-registry-url&gt;/node:20"| B[Cloudflare serverless registry];
    B -->|"Does image exist locally?"| C{"Image in R2 bucket?"};
    
    C -- "Yes" --> D["Serve node:20 from R2"];
    C -- "No" --> E["Check for upstream configuration"];
    
    E -->|"Upstream config set?"| F{"DockerHub upstream registry configured?"};
    F -- "No" --> G["Error: Manifest Unknown"];
    F -- "Yes" --> H["Pull node:20 from DockerHub"];
    
    H -->|"Pull successful?"| I["Save node:20 image to R2"];
    H -- "Fail" --> J["Error: Authentication or Rate Limit Exceeded"];
    I --> D;
    
    D -->|"Return cached image"| K[Docker engine];

Configuring the Cloudflare registry

I’ve prepared a script that simplifies the deployment and configuration of the registry. Please note that it’s not a production-ready solution (and I would be happy to see the suggestions / contributions to improve it) and should be used for testing purposes only - refer to the official Cloudflare repo for more details. If you want to proceed without a script, refer to the official README.

Prerequisites:

  • pnpm
  • wrangler
  • Cloudflare token -> “Create token” -> “Create custom token”. Choose the following permissions:
    • Account:Workers R2 Storage:Edit
    • Account:Workers KV Storage:Edit
    • Account:Workers Scripts:Edit
    • User:Memberships:Read
    • User:User Details:Read Cloudflare token permissions
  • DockerHub PAT for pulling images from DockerHub (needed for the pull-through cache, no anonymous pull option supported atm, unfortunately). Token’s scope should be set to Read & Write (yes, that’s counterintuitively, but token should have Write permissions for docker inspect operations). Set expiration to None if this DockerHub account would be used only for upstream registry. DockerHub PAT

Important notes

  1. Pull-through cache via upstream registry won’t work by default - you should provide the upstream registry url, username and password in the script / command line arguments to enable it.
  2. Custom domain won’t be assigned by default - provide the custom domain in the script / command line arguments to enable it (and then both Cloudflare default one and your custom one will work).
  3. The following lifecycle rules are set by default (to avoid unnecessary costs):
    • Expire R2 blobs after 30 days
    • Abort multipart uploads after 1 day
    • Move to Infrequent Access tier after 14 days
  4. Worker logs are enabled by default - the script enabled Cloudflare Worker logs for your registry, so you can see the logs in the Cloudflare dashboard and troubleshoot the issues.

Deploying the registry

After cloning the repo with the script, provide all the required values at the top of the script / use command line arguments. Then run the script via

./cf-registry-automation.sh

After the script is finished, you’ll see the output with the registry URL.

Congratulations! Authenticate by running docker login <registry-url>, supply the username and password you’ve provided in the script / command line arguments, and you’re ready to go.

Now you can test the registry by running docker pull <registry-url>/node:20 and see if it works - it should pull the image from DockerHub and save it to the Cloudflare R2 bucket -> serve it from there to your local Docker engine.

Next steps

In order to use the registry in your CI/CD pipeline, you need to configure your workflows to use this registry, and change your Dockerfiles / override registry url via ARGs. In all places where you pull images from DockerHub, you should replace the registry url with the one you’ve got from the script:

# instead of 
FROM node:20
# use
FROM <registry-url>/node:20

Next, configure authentication for the registry in your CI/CD pipeline. Let’s use Github Actions as an example:

- name: Login to Cloudflare registry
  uses: docker/login-action@v3
  with:
    username: ${{ vars.CLOUDFLARE_REGISTRY_USERNAME }}
    password: ${{ vars.CLOUDFLARE_REGISTRY_PASSWORD }}

Note: you should set CLOUDFLARE_REGISTRY_USERNAME and CLOUDFLARE_REGISTRY_PASSWORD as secrets in your repository settings.

Known issues

  • not found: manifest unknown: manifest unknown - with pull-through cache enabled, the registry will return this error not only for the missing images, but also when authentication for upstream registry fails. Check the logs of the Cloudflare Worker to find the root cause.
  • No anonymous pull support - you need to use a PAT for pulling images from DockerHub / other registries. See and support corresponding issue