Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Running 'crc' on a remote server

Overview: running crc on a remote server

This document shows how to deploy an OpenShift instance on a server using CodeReady Containers (crc) that can be accessed remotely from one or more client machines (sometimes called a "headless" instance). This provides a low-cost test and development platform that can be shared by developers. Deploying this way also allows a user to create an instance that uses more cpu and memory resources than may be available on his or her laptop.

While there are benefits to this type of deployment, please note that the primary use case for crc is to deploy a local OpenShift instance on a workstation or laptop and access it directly from the same machine. The headless setup is configured completely outside of crc itself, and supporting a headless setup is beyond the mission of the crc development team. Please do not ask for changes to crc to support this type of deployment, it will only cost the team time as they politely decline :)

The instructions here were tested with Fedora on both the server (F30) and a laptop (F29).

Thanks to

Thanks to Marcel Wysocki from Red Hat for the haproxy solution and the entire CodeReady Containers team for crc!

Useful links

Red Hat blog article on CodeReady Containers

Download page on cloud.redhat.com

CRC documentation on github.io

Project sources on github

Download and setup CRC on a server

Go to the download page and get crc for Linux. You’ll also need the pull secret listed there during the installation process. Make sure to copy the crc binary to /usr/local/bin or somewhere on your path.

The initial setup command only needs to be run once, and it creates a ~/.crc directory. Your user must have sudo privileges since crc will install dependencies for libvirt and modify the NetworkManager config:

$ crc setup

Note: occasionally on some systems this may fail with “Failed to restart NetworkManager”. Just rerun crc setup a few times until it works

Create an OpenShift Instance with CRC

$ crc start

You will be asked for the pull secret from the download page, paste it at the prompt.

Optionally, use the -m and -c flags to increase the VM size, for example a 32GiB with 8 cpus:

$ crc start -m 32768 -c 8

See the documentation or crc -h for other things you can do

If you want to just use crc locally on this machine, you can stop here, you’re all set!

Make sure you have haproxy and a few other things

sudo dnf -y install haproxy policycoreutils-python-utils jq

Modify the firewall on the server

$ sudo systemctl start firewalld
$ sudo firewall-cmd --add-port=80/tcp --permanent
$ sudo firewall-cmd --add-port=6443/tcp --permanent
$ sudo firewall-cmd --add-port=443/tcp --permanent
$ sudo systemctl restart firewalld
$ sudo semanage port -a -t http_port_t -p tcp 6443

Configure haproxy on the server

The steps below will create an haproxy config file with placeholders, update the SERVER_IP and CRC_IP using sed, and copy the new file to the correct location. If you would like to edit the file manually, feel free :)

$ sudo cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak
$ tee haproxy.cfg &>/dev/null <<EOF
global
        debug

defaults
        log global
        mode    http
        timeout connect 5000
        timeout client 5000
        timeout server 5000

frontend apps
    bind SERVER_IP:80
    bind SERVER_IP:443
    option tcplog
    mode tcp
    default_backend apps

backend apps
    mode tcp
    balance roundrobin
    option ssl-hello-chk
    server webserver1 CRC_IP check

frontend api
    bind SERVER_IP:6443
    option tcplog
    mode tcp
    default_backend api

backend api
    mode tcp
    balance roundrobin
    option ssl-hello-chk
    server webserver1 CRC_IP:6443 check
EOF

$ export SERVER_IP=$(hostname --ip-address)
$ export CRC_IP=$(crc ip)
$ sed -i "s/SERVER_IP/$SERVER_IP/g" haproxy.cfg
$ sed -i "s/CRC_IP/$CRC_IP/g" haproxy.cfg
$ sudo cp haproxy.cfg /etc/haproxy/haproxy.cfg
$ sudo systemctl start haproxy

Setup NetworkManager on the client machine

NetworkManager needs to be configured to use dnsmasq for DNS. Make sure you have dnsmasq installed:

$ sudo dnf install dnsmasq

Add a file to /etc/NetworkManager/conf.d to enable use of dnsmasq. (Some systems may already have this setting in an existing file, depending on what's been done in the past. If that's the case, continue on without creating a new file)

$ sudo tee /etc/NetworkManager/conf.d/use-dnsmasq.conf &>/dev/null <<EOF
[main]
dns=dnsmasq
EOF

Add dns entries for crc:

$ tee external-crc.conf &>/dev/null <<EOF
address=/apps-crc.testing/SERVER_IP
address=/api.crc.testing/SERVER_IP
EOF

$ export SERVER_IP=”your server’s external IP address”
$ sed -i "s/SERVER_IP/$SERVER_IP/g" external-crc.conf
$ sudo cp external-crc.conf /etc/NetworkManager/dnsmasq.d/external-crc.conf
$ sudo systemctl reload NetworkManager

Note: if you've previously run crc locally on the client machine, you likely have a /etc/NetworkManager/dnsmasq.d/crc.conf file that sets up dns for a local VM. Comment out those entries.

Get the oc binary on the client machine

If you don't already have it, you can get the oc client here:

https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/

Login to the OpenShift instance from the client machine

The password for the kubeadmin account is printed when crc starts, but if you don't have it handy you can do this as the user running crc on the server:

$ crc console --credentials

Now just login to OpenShift from your client machine using the standard crc url

$ oc login -u kubeadmin -p <kubeadmin password>  https://api.crc.testing:6443

The OpenShift console will be available at https://console-openshift-console.apps-crc.testing

Renewal of expired certificates

Beginning in version 1.2.0, CodeReady Containers will renew embedded certificates when they expire (prior to 1.2.0 it was necessary to download and install a new version). When the certificates need to be renewed, this will be noted in the CRC log output and may take up to 5 minutes.

@rcarrata

This comment has been minimized.

Copy link

@rcarrata rcarrata commented Mar 26, 2020

Very useful guide @tmckayus!

Just a little comment, into the haproxy config file is missing the port in the backend apps (CRC_IP instead of CRC_IP:Port), and the haproxy fails at start:

$ systemctl status haproxy -l
● haproxy.service - HAProxy Load Balancer
      Active: failed (Result: exit-code) since Thu 2020-03-26 14:33:00 CET; 4s ago
         [/etc/haproxy/haproxy.cfg:22] : server webserver1 has neither service port nor check port nor tcp_check rule 'connect' with port information. Check has been disabled

I fixed it adding the 443 port to that line:

server webserver1 CRC_IP:443 check
@tmckayus

This comment has been minimized.

Copy link
Owner Author

@tmckayus tmckayus commented Mar 26, 2020

@rcarrata interesting, thanks! I haven't seen that. What OS version are you using? I've done all my stuff on F30

@rcarrata

This comment has been minimized.

Copy link

@rcarrata rcarrata commented Mar 26, 2020

@tmckayus I've installed the CRC on top of my Nuc running Centos7.

$ cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
$ crc version
crc version: 1.7.0+fa7e558
OpenShift version: 4.3.1 (embedded in binary)

By the way, an small idea for the haproxy config file is to split the traffic for apps with two fe-be (http and https):

frontend apps-http
    bind SERVER_IP:80
    option tcplog
    mode tcp
    default_backend apps-http

backend apps-http
    mode tcp
    balance roundrobin
    server webserver1 CRC_IP:80 check

frontend apps-https
    bind SERVER_IP:443
    option tcplog
    mode tcp
    default_backend apps-https

backend apps-https
    mode tcp
    balance roundrobin
    option ssl-hello-chk
    server webserver1 CRC_IP:443 check

I tested from my laptop (to the CRC in remote server), and works well enough :)

$ curl http://nodejs-basic-tst.apps-crc.testing -I
HTTP/1.1 200 OK

$ curl -Ik https://console-openshift-console.apps-crc.testing
HTTP/1.1 200 OK
@loxtank

This comment has been minimized.

Copy link

@loxtank loxtank commented May 2, 2020

For anyone trying to use a macOS client machine:

@cfchase

This comment has been minimized.

Copy link

@cfchase cfchase commented Jun 12, 2020

@rcarrata interesting, thanks! I haven't seen that. What OS version are you using? I've done all my stuff on F30

Got the same thing on Fedora 32 haproxy 2.1.5

@svsiva

This comment has been minimized.

Copy link

@svsiva svsiva commented Jun 19, 2020

I am using Windows 10 Laptop. How do I configure DNS client with the settings mentioned in the dnsmasq settings?

@rsletten

This comment has been minimized.

Copy link

@rsletten rsletten commented Jun 23, 2020

If you use Pi-Hole (and you should),

$ sudo tee /etc/dnsmasq.d/external-crc.conf &>/dev/null <<EOF
address=/apps-crc.testing/SERVER_IP
address=/api.crc.testing/SERVER_IP
EOF

and restart the DNS resolver

@kambei

This comment has been minimized.

Copy link

@kambei kambei commented Jul 18, 2020

Maybe a stupid question, but... If I have CRC running in a CentOS docker container how can I apply this for accessing the web console of CRC from Host machine? I can't figure out how can I do it!

EDIT: with this changes to haproxy.conf

global
log 127.0.0.1 local0
debug

defaults
log global
mode http
timeout connect 5000
timeout check 5000
timeout client 30000
timeout server 30000

frontend apps
bind CONTAINER_IP:80
bind CONTAINER_IP:443
option tcplog
mode tcp
default_backend apps

backend apps
mode tcp
balance roundrobin
option ssl-hello-chk
server webserver1 CRC_IP:6443 check

frontend api
bind CONTAINER_IP:6443
option tcplog
mode tcp
default_backend api

backend api
mode tcp
balance roundrobin
option ssl-hello-chk
server webserver1 CRC_IP:6443 check

and enabling forwarding for the container:

$ sysctl net.ipv4.conf.all.forwarding=1
$ sudo iptables -P FORWARD ACCEPT

I can call the url https://console-openshift-console.apps-crc.testing:6443/ from the Host machine... but I get this error:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {
    
  },
  "code": 403
}
@tmckayus

This comment has been minimized.

Copy link
Owner Author

@tmckayus tmckayus commented Jul 20, 2020

Maybe a stupid question, but... If I have CRC running in a CentOS docker container how can I apply this for accessing the web console of CRC from Host machine? I can't figure out how can I do it!

Thanks for the question! This is a good one for the community; as with several of the questions here I have no idea :)

@jboxman

This comment has been minimized.

Copy link

@jboxman jboxman commented Aug 15, 2020

With the latest Fedora, I am now getting this after upgrading:

Aug 15 15:58:45 stink haproxy[54797]: [ALERT] 227/155845 (54797) : config: backend 'apps': server 'webserver1' has neither service port nor check port nor tcp_check rule 'connect' with port information.
Aug 15 15:58:45 stink haproxy[54797]: [ALERT] 227/155845 (54797) : Fatal errors found in configuration.

I've never used haproxy, so I'm not sure what to do with this.

@DevOpsRookie

This comment has been minimized.

Copy link

@DevOpsRookie DevOpsRookie commented Aug 16, 2020

For anyone trying to use a macOS client machine:

Thanks for this - I have the web portal working remotely but cannot get the command line working remotely - oc. Did you get the command line working with this method?

@jboxman

This comment has been minimized.

Copy link

@jboxman jboxman commented Aug 17, 2020

For anyone trying to use a macOS client machine:

Thanks for this - I have the web portal working remotely but cannot get the command line working remotely - oc. Did you get the command line working with this method?

A longstanding issue with Go prevents the oc binary from understanding DNS on OS X. For the CLI client only, you still must have an entry in /etc/hosts, such as:

192.168.70.30 api.crc.testing oauth-openshift.apps-crc.testing
@EmmanuelKasper

This comment has been minimized.

Copy link

@EmmanuelKasper EmmanuelKasper commented Aug 22, 2020

With a recent haproxy, I am using the following configuration:

$ rpm -qi haproxy
haproxy-1.8.23-3.el8.x86_64

works well and does not spit warnings everywhere in the haproxy log:)

$ cat /etc/haproxy.cfg

global
        debug
        log 127.0.0.1 local0

defaults
        log global
        mode http
        timeout connect 0
        timeout client 0
        timeout server 0

listen  apps-http
        bind HOST_IP:80
        option tcplog
        mode tcp
        server webserver1 CRC_IP:80 check

listen  apps-https
        bind HOST_IP:443
        option tcplog
        mode tcp
        option ssl-hello-chk
        server webserver1 CRC_IP:443 check

listen  api
        bind HOST_IP:6443
        option tcplog
        mode tcp
        option ssl-hello-chk
        server webserver1 CRC_IP:6443 check
@jboxman

This comment has been minimized.

Copy link

@jboxman jboxman commented Aug 22, 2020

That works also with:

rpm -qi haproxy
Name        : haproxy
Version     : 2.1.7
Release     : 1.fc32

Thanks!

@evanshortiss

This comment has been minimized.

Copy link

@evanshortiss evanshortiss commented Sep 28, 2020

If you run into issues with Pod logs in the UI failing to load, give this haproxy.cfg a try. It will support the WebSocket upgrade:

defaults
    mode http
    log global
    option httplog
    option  http-server-close
    option  dontlognull
    option  redispatch
    option  contstats
    retries 3
    backlog 10000
    timeout client          25s
    timeout connect          5s
    timeout server          25s
    timeout tunnel        3600s
    timeout http-keep-alive  1s
    timeout http-request    15s
    timeout queue           30s
    timeout tarpit          60s
    default-server inter 3s rise 2 fall 3
    option forwardfor

frontend apps
    bind HOST_IP:80
    bind HOST_IP:443
    option tcplog
    mode tcp
    default_backend apps

backend apps
    mode tcp
    balance roundrobin
    option tcp-check
    server webserver1 CRC_IP check port 80

frontend api
    bind HOST_IP:6443
    option tcplog
    mode tcp
    default_backend api

backend api
    mode tcp
    balance roundrobin
    option tcp-check
    server webserver1 CRC_IP:6443 check port 6443
@rlzh

This comment has been minimized.

Copy link

@rlzh rlzh commented Oct 28, 2020

For anyone trying to use a macOS client machine:

Thanks for this - I have the web portal working remotely but cannot get the command line working remotely - oc. Did you get the command line working with this method?

I am currently facing the same situation. Were you able to resolve this?

@loxtank

This comment has been minimized.

Copy link

@loxtank loxtank commented Oct 28, 2020

For anyone trying to use a macOS client machine:

Thanks for this - I have the web portal working remotely but cannot get the command line working remotely - oc. Did you get the command line working with this method?

I am currently facing the same situation. Were you able to resolve this?

Hm, if I remember correctly I had it working. But I don’t have the setup around anymore so I can’t confirm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.