Recently, I've been trying to move away from Docker Desktop for Mac and I think I finally got it working so here is what I did:

1. Uninstall Docker Desktop
2. Delete ~/.docker/config.json (Don't worry, Colima will recreate it correctly)
3. Install docker-cli with Homebrew

brew install --formula docker

1. Install colima

brew install colima

1. Setup a VM

colima start -a x86_64

You are only gonna need to run start with options this time, next time, colima start is enough.

1. Install BuildKit for Mac with the following script

#!/bin/bash

ARCH=arm64
VERSION=v0.10.4
curl -LO https://github.com/docker/buildx/releases/download/${VERSION}/buildx-${VERSION}.darwin-${ARCH}
mkdir -p ~/.docker/cli-plugins
mv buildx-${VERSION}.darwin-${ARCH} ~/.docker/cli-plugins/docker-buildx
chmod +x ~/.docker/cli-plugins/docker-buildx
docker buildx version

And that's it 🎉