Notes for switching from Docker Desktop to Minikube

Minikube for Mac Users.md

With Docker Desktop becoming more restricted, I've decided to move on to just using minikube. In doing so, I've consolidated my notes as follows.

Installation

Use brew to install the docker cli and minikube.

$ brew install minikube docker kubectl hyperkit

Running Minikube

The first time you start minikube, you should specify any settings you desire.

$ minikube start --addons=registry --cni=calico --driver=hyperkit --cpus=8 --memory=8g

When you start with --cni=calico it might take some extra time for the CNI to become active.

$ kubectl -n kube-system wait --for=condition=ready --all pods --timeout=10m

You can check the status with this command if you're unsure if minikube is running or not.

$ minikube status

Anytime you need to start minikube back up, you don't need to set additional flags anymore

$ minikube start

If you need to stop minikube (to free up RAM or CPU), run this command.

$ minikube stop

Resetting Minikube

It doesn't matter if minikube is running or not, you can issue the delete command.

$ minikube delete
🔥  Deleting "minikube" in hyperkit ...
💀  Removed all traces of the "minikube" cluster.

Then just do the start command as mentioned in the Running Minikube section.

Using Docker CLI

Minikube needs to be running and you need to use envirnment variables so that the docker cli tool can contact minikube.

This command does the environment setup for you, after which docker commands should just work.

eval $(minikube docker-env)

If you add this to your ~/.bashrc or ~/.bash_profile, it will automatically setup the docker environment variables for you when minikube is running, in new terminal sessions.

minikube status > /dev/null && eval $(minikube docker-env)

Something important to know, when using --volume and --port it will be applied to the Hyperkit VM, not localhost like with Docker Desktop.

You can get the minikube ip with:

$ minikube ip

As a pro-tip if you need to just curl something, you can do this:

$ docker run -d -p 8000:80 httpd
$ curl http://$(minikube ip):8000

Accessing the Minikube VM

When using the hyperkit driver, you can access the VM if necessary.

The simple way.

$ minikube ssh

The verbose way, in case you need to login manually or use sftp or something.

$ ssh -i ~/.minikube/machines/minikube/id_rsa docker@$(minikube ip)

Cluster Switching

The drop down menu in Docker Desktop is a very convenient tool for switching between clusters. The same thing can be done with kubectl directly though.

$ kubectl config get-contexts
$ kubectl config use-context CLUSTERNAME

Issues After Switching

Homebrew Cleanup Error

When I went to install something, homebrew attempted to do brew cleanup after 30 days. It presented an error.

[protosam@localhost]$ brew install openshift-cli
...
==> `brew cleanup` has not been run in the last 30 days, running now...
Removing: /Users/pilot/Library/Caches/Homebrew/bash--5.1.8... (3.0MB)
Removing: /Users/pilot/Library/Caches/Homebrew/bdw-gc--8.0.4_2... (513.9KB)
Removing: /usr/local/Cellar/c-ares/1.17.1... (85 files, 672.5KB)
Removing: /Users/pilot/Library/Caches/Homebrew/devspace--5.14.4... (14.7MB)
...
Removing: /Users/pilot/Library/Logs/Homebrew/lzo... (64B)
Removing: /Users/pilot/Library/Logs/Homebrew/python@3.9... (2 files, 4.3KB)
Removing: /Users/pilot/Library/Logs/Homebrew/gnutls... (64B)
Error: Permission denied @ apply2files - /usr/local/lib/docker/cli-plugins

The problem was that it couldn't reach a symlinked directory. The directory /usr/local/lib/docker/cli-plugins was pointing to /Applications/Docker.app/Contents/Resources/cli-plugins.

Deleting /usr/local/lib/docker solved the issue.

$ sudo rm -rf /usr/local/lib/docker

Thanks for putting this guide together!

Something important to know, when using --volume and --port it will be applied to the Hyperkit VM, not localhost

Is there any way to mitigate this? It would be nice to have them running on localhost. Maybe a proxy? But then how do we deal with changing Minikube IPs?

Also I have two more issues, not sure if I missed a step:

• The docker command is not available and I'm not sure what to do about that. It's not that much of a problem as I just installed docker-compose (which does work), but still.
• Folders mapped via volumes don't seem to work.

Do you have any ideas about these issues? Thanks a lot!

Hi! thanks for the great guide, one question, would docker-cli installed this way conflict if you are using another k8s cluster? for example:

# Install everything as in the guide
eval $(minikube docker-env)
docker run -d -p 8000:80 httpd # ---> this works

# Switch contexts
kubectl config use-context aDifferentContext

docker run -d -p 8000:80 httpd # ---> will this still work?
kubectl get pods # ---> will this be pointed to `aDifferentContext`?

Hi! thanks for the great guide, one question, would docker-cli installed this way conflict if you are using another k8s cluster? for example:

# Install everything as in the guide
eval $(minikube docker-env)
docker run -d -p 8000:80 httpd # ---> this works

# Switch contexts
kubectl config use-context aDifferentContext

docker run -d -p 8000:80 httpd # ---> will this still work?
kubectl get pods # ---> will this be pointed to `aDifferentContext`?

It does not conflict and will still work as expected. In fact eval $(minikube docker-env) will always work whether you are in the minikube k8s context or otherwise.

As for accessing the minikube VM section, you can simply run minikube ssh instead.

@stefandesu I don't think there is. The changing IP problem I address later in the document. I personally just use $(minikube ip) in places where I used to use localhost.

The docker command should have come from brew install docker: https://formulae.brew.sh/formula/docker#default

When you use --volume the source is in minikube's VM, not your workstation. If you absolutely need to persist some data for some reason, there are steps included on accessing the minikube VM over SSH. Most of my own use cases, I've gotten into the habit of using COPY in Dockerfiles. If I'm changing files locally and not seeing the results of my Dockerfile being built, there's more room for human error in the development process.

---

@abiosoft, thanks for sharing that, I like it. Updated the guide to include it. :)

Thanks for the gist.

There's something about the instructions or Minikube or something that makes this not quite work as a drop-in replacement for Docker Desktop for my project. I'm trying to start the docker environment with the command docker-compose up -d --build. With Docker Desktop it works fine, but after following these instructions I'm getting errors like this for some of the services:

ERROR: for service-name-here  Cannot start service service-name-here: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/var/www/html/path/to/foo.entrypoint.sh": stat /var/www/html/path/to/foo.entrypoint.sh: no such file or directory: unknown

Also "volume" mounts with files don't work. Having this in docker-compose.yml for a service:

    volumes:
      - ./foo-config/foo.ini:/etc/foo/foo.ini

Results in error: "mount through procfd: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type"

I have installed docker and docker-compose from homebrew. Unfortunately this is not a shareable project, but I can maybe try to produce a reduced example that replicates the behavior if somebody wants to help fix the issue.

I put up a script together with a similar goal https://github.com/abiosoft/colima

@kumikumi you can give it a try.

@kumikumi read my prior comment regarding volume. I suspect this is what you're running into.

@abiosoft
Does your project address the volumes issue at all? I'm not sure how using lima is different than using minikube with hyperkit, virtualbox, parallels, or vmwarefusion. Is there any additional advantages you've seen to it in the context of Docker or Kubernetes support?

@kumikumi read my prior comment regarding volume. I suspect this is what you're running into.

@abiosoft
Does your project address the volumes issue at all? I'm not sure how using lima is different than using minikube with hyperkit, virtualbox, parallels, or vmwarefusion. Is there any additional advantages you've seen to it in the context of Docker or Kubernetes support?

@protosam yes, it addresses the volume issue (though read-only for now) as well as port forwarding. And unlike minikube, kubernetes is optional if you want to keep it lightweight.

Thanks @abiosoft, you have made my day. All the containers start up fine now, unmodified, without any errors. The only remaining problem is something that probably has to do with file system permissions. I will report it as an issue to your project to not derail the conversation here.

@abiosoft very nice. Thanks for sharing.

isn't it minikube ip rather than docker ip in the curl command?

@mteodori you are correct.

Didn't notice that since I type it in everytime. lol

Updated the gist.