Skip to content

Instantly share code, notes, and snippets.

@trisberg
Last active January 26, 2024 17:47
Show Gist options
  • Star 37 You must be signed in to star a gist
  • Fork 6 You must be signed in to fork a gist
  • Save trisberg/37c97b6cc53def9a3e38be6143786589 to your computer and use it in GitHub Desktop.
Save trisberg/37c97b6cc53def9a3e38be6143786589 to your computer and use it in GitHub Desktop.
Using a Local Registry with Minikube

Using a Local Registry with Minikube

Install a local Registry

These instructions include running a local registry accessible from Kubernetes as well as from the host development machine at registry.dev.svc.cluster.local:5000.

  1. Use the docker CLI to run the registry:2 container from Docker, listening on port 5000, and persisting images in the ~/.registry/storage directory.

    docker run -d -p 5000:5000 --restart=always --volume ~/.registry/storage:/var/lib/registry registry:2
    
  2. Edit the /etc/hosts file on your development machine, adding the name registry.dev.svc.cluster.local on the same line as the entry for localhost.

  3. Validate that the registry is running.

    docker ps
    
    CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
    02ea46d51f58        registry:2          "/entrypoint.sh /etc…"   About an hour ago   Up About a minute   0.0.0.0:5000->5000/tcp   sharp_pike
    
  4. Validate that the registry at registry.dev.svc.cluster.local:5000 is reachable from your development machine.

    curl registry.dev.svc.cluster.local:5000/v2/_catalog
    
    {"repositories":[]}
    
  5. Configure the docker daemon with an insecure registy at registry.dev.svc.cluster.local:5000.

    On macOS in ~/.docker/daemon.json on Linux in /etc/docker/daemon.json (craete the file if it does not exist)

    {
      "insecure-registries": ["registry.dev.svc.cluster.local:5000"]
    }
    

Start Minikube

minikube start --cpus 4 --memory 4096 --insecure-registry registry.dev.svc.cluster.local:5000

Configure a fixed IP address

This IP address will allow processes in Minikube to reach the registry running on your host. Configuring a fixed IP address avoids the problem of the IP address changing whenever you connect your machine to a different network. If your machine already uses the 172.16.x.x range for other purposes, choose an address in a different range e.g. 172.31.x.x..

export DEV_IP=172.16.1.1

Create an alias on MacOS:

sudo ifconfig lo0 alias $DEV_IP

Create an alias on Linux:

sudo ifconfig lo:0 $DEV_IP

Note that the alias will need to be reestablished when you restart your machine. This can be avoided by using a launchdeamon on MacOS or by editing /etc/network/interfaces on Linux.

Minikube /etc/hosts

Add an entry to /etc/hosts inside the minikube VM, pointing the registry to the IP address of the host. This will result in registry.dev.svc.cluster.local resolving to the host machine allowing the docker daemon in minikube to pull images from the local registry. This uses the DEV_IP environment variable from the previous step.

export DEV_IP=172.16.1.1
minikube ssh "echo \"$DEV_IP       registry.dev.svc.cluster.local\" | sudo tee -a  /etc/hosts"

Kubernetes Service and Endpoint

Create a kubernetes service without selectors called registry in the dev namespace and a kubernetes endpoint with the same name pointing to the static IP address of your development machine. This will result in registry.dev.svc.cluster.local resolving to the host machine, allowing container builds running in the cluster, to work with the local registry.

kubectl create namespace dev
cat <<EOF | kubectl apply -n dev -f -
---
kind: Service
apiVersion: v1
metadata:
  name: registry
spec:
  ports:
  - protocol: TCP
    port: 5000
    targetPort: 5000
---
kind: Endpoints
apiVersion: v1
metadata:
  name: registry
subsets:
  - addresses:
      - ip: $DEV_IP
    ports:
      - port: 5000
EOF

Relocate app images

Install irel CLI from https://github.com/pivotal/image-relocation/releases

For this example we are using a registry prefix of registry.dev.svc.cluster.local:5000; you would need to change this to match your registry (you also need to be authenticated with this registry).

To copy the time sink app to your own registry run:

irel copy springcloudstream/time-source-rabbit:2.1.2.RELEASE registry.dev.svc.cluster.local:5000/time-source-rabbit:2.1.2.RELEASE

To copy the log sink app to your own registry run:

irel copy springcloudstream/log-sink-rabbit:2.1.3.RELEASE registry.dev.svc.cluster.local:5000/log-sink-rabbit:2.1.3.RELEASE

You can now register the relocated apps using SCDF Shell:

dataflow:>app register --name time --type source --uri docker:registry.dev.svc.cluster.local:5000/time-source-rabbit:2.1.2.RELEASE
dataflow:>app register --name log --type sink --uri docker:registry.dev.svc.cluster.local:5000/log-sink-rabbit:2.1.3.RELEASE
@iamtodor
Copy link

for me, this curl registry.dev.svc.cluster.local:5000/v2/_catalog does not work, altho curl http://localhost:5000/v2/_catalog works

@DonnyWhoLovedBowling
Copy link

@iamtodor had the same issue, this was due to my proxy. Adding registry.dev.svc.cluster.local to your no_proxy/NO_PROXY environment variables fixed it.

@DonnyWhoLovedBowling
Copy link

@trisberg On which machine do you configure the fixed ip-address? Your local (development) machine, inside your registry container or inside your minikube container?

@trisberg
Copy link
Author

@DonnyWhoLovedBowling That would have been on my local dev machine. This is an old gist and I haven't used it for quite a while, so not sure if it still works as intended.

@DonnyWhoLovedBowling
Copy link

Thanks!

@Chasun-fhd
Copy link

well, i am using minikube , and following the above instructions, but always get errors in kubernetes as below:

Failed to pull image "registry.dev.svc.cluster.local:5001/fenghaidong/k8s-demo-app:v1.1": rpc error: code = Unknown desc = Error response from daemon: Get "http://registry.dev.svc.cluster.local:5001/v2/": dial tcp 127.0.0.1:5001: connect: connection refused

my local registry port is 5001, call http://registry.dev.svc.cluster.local:5001/v2/_catalog and response correctly. {"repositories":["4kb/jdk8","4kb/k8s-demo-app"]}.

i tried different ways to solve out, but failed. wish for help, thks.

@iamtodor
Copy link

iamtodor commented May 3, 2022

@Chasun-fhd verify that your image is in the registry by curl http://localhost:5001/v2/_catalog || curl http://127.0.0.1:5001/v2/_catalog

@Chasun-fhd
Copy link

@Chasun-fhd verify that your image is in the registry by curl http://localhost:5001/v2/_catalog || curl http://127.0.0.1:5001/v2/_catalog

i tried too, returns the same like http://registry.dev.svc.cluster.local:5001/v2/_catalog

@Chasun-fhd
Copy link

finally, i solved this problem. Configure static ip address on my mac. then replace ip in these instructions with new static ip. it worked.

@oscarCCOnis
Copy link

I have my image in my repo, but my minikube is trying to find https instead of http and its failing

$ curl registry.dev.svc.cluster.local:5000/v2/_catalog
{"repositories":["hpesaconnmonitor"]}

Events:                                                                                                                                  │
│   Type     Reason     Age              From               Message                                                                        │
│   ----     ------     ----             ----               -------                                                                        │
│   Normal   Scheduled  5s               default-scheduler  Successfully assigned monitoring/monitoring-deployment-696c48f8d4-flp5v to min │
│ ikube                                                                                                                                    │
│   Normal   Pulling    5s               kubelet            Pulling image "registry.dev.svc.cluster.local:5000/hpesaconnmonitor:1.0"       │
│   Warning  Failed     4s               kubelet            Failed to pull image "registry.dev.svc.cluster.local:5000/hpesaconnmonitor:1.0 │
│ ": rpc error: code = Unknown desc = Error response from daemon: Get "https://registry.dev.svc.cluster.local:5000/v2/": dial tcp: lookup  │
│ registry.dev.svc.cluster.local: no such host                                                                                             │
│   Warning  Failed     4s               kubelet            Error: ErrImagePull                                                            │
│   Normal   BackOff    3s (x2 over 4s)  kubelet            Back-off pulling image "registry.dev.svc.cluster.local:5000/hpesaconnmonitor:1 │
│ .0"                                                                                                                                      │
│   Warning  Failed     3s (x2 over 4s)  kubelet            Error: ImagePullBackOff`

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment