Skip to content

Instantly share code, notes, and snippets.

@pandeybk
Last active May 6, 2024 19:58
Show Gist options
  • Save pandeybk/bbff1b7031762ad1e8aef60f91358171 to your computer and use it in GitHub Desktop.
Save pandeybk/bbff1b7031762ad1e8aef60f91358171 to your computer and use it in GitHub Desktop.
redhat-openshift-ai-notebooks-as-sudo-user

This Gist outlines the process of adding a sudo user to a Red Hat OpenAI (RHOAI) Workbench container image. Whether building the image from scratch using a Containerfile or utilizing a pre-built image, the steps provided here ensure the inclusion of a sudo user for enhanced permissions within the containerized environment.

  1. Build the Container image using Containerfile file. The important part is. You can also use pre-built image which is published here quay.io/bpandey/rhoai-workbench:sudo-jupyteruser-v3
  2. Create notebook using notebook.yaml file. Make sure to replace namespace, container image, OCP domain name. You can also create a workbench from RHOAI ui and patch following lines later.
          securityContext:
            allowPrivilegeEscalation: true
            runAsGroup: 1001
            runAsUser: 1001

One issue I've encountered with the patch is the following error message. The root cause of this error is that the /opt/app-root/src/ directory is persistent. When you launch the workbench from the RHOAI UI for the first time, it creates this directory with a random user. After applying the patch, we attempt to run the workbench as user 1001, which causes permission conflicts. To address this, we could implement an initialization script to resolve the permission issue automatically. Alternatively, we could create the workbench using the notebook.yaml file from the outset, eliminating this issue altogether

[I 2024-05-06 18:27:50.042 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[E 2024-05-06 18:27:50.042 ServerApp] Failed to write server-info to /opt/app-root/src/.local/share/jupyter/runtime/jpserver-13.json: PermissionError(13, 'Permission denied')
Traceback (most recent call last):
 File "/opt/app-root/bin/jupyter-lab", line 8, in <module>
   sys.exit(main())
 File "/opt/app-root/lib64/python3.9/site-packages/jupyter_server/extension/application.py", line 617, in launch_instance
   serverapp.start()
 File "/opt/app-root/lib64/python3.9/site-packages/jupyter_server/serverapp.py", line 2952, in start
   self.start_app()
 File "/opt/app-root/lib64/python3.9/site-packages/jupyter_server/serverapp.py", line 2856, in start_app
   self.write_browser_open_files()
 File "/opt/app-root/lib64/python3.9/site-packages/jupyter_server/serverapp.py", line 2723, in write_browser_open_files
   self.write_browser_open_file()
 File "/opt/app-root/lib64/python3.9/site-packages/jupyter_server/serverapp.py", line 2746, in write_browser_open_file
   with open(self.browser_open_file, "w", encoding="utf-8") as f:
PermissionError: [Errno 13] Permission denied: '/opt/app-root/src/.local/share/jupyter/runtime/jpserver-13-open.html'
[INFO 2024-05-06 18:27:52.152 TabnineDownloader]: Finish download Tabnine Binary to /opt/app-root/lib/python3.9/site-packages/jupyterlab_tabnine/binaries/4.159.0/x86_64-unknown-linux-musl
  1. Add scc profile
oc adm policy add-scc-to-user privileged -z sudo-jupyteruser

Security Disclaimer:

Adding a sudo user introduces security vulnerabilities by granting elevated privileges within the containerized environment. Exercise caution and consider the security implications before implementing this solution.

Additionally, creating the workbench from the RHOAI UI introduces complexity due to potential permission conflicts and error logs as mentioned previously.

Blog Reference

FROM registry.access.redhat.com/ubi8/python-39:1-176.1712880517
LABEL name="workbench-images:cuda-jupyter-pytorch-c9s-py39_2024b_20240418" \
summary="cuda-jupyter-pytorch workbench image with Python py39 based on c9s" \
description="cuda-jupyter-pytorch workbench image with Python py39 based on c9s" \
io.k8s.description="cuda-jupyter-pytorch workbench image with Python py39 based on c9s for ODH or RHODS" \
io.k8s.display-name="cuda-jupyter-pytorch workbench image with Python py39 based on c9s" \
authoritative-source-url="https://github.com/opendatahub-contrib/workbench-images" \
io.openshift.build.commit.ref="2024b" \
io.openshift.build.source-location="https://github.com/opendatahub-contrib/workbench-images" \
io.openshift.build.image="https://quay.io/opendatahub-contrib/workbench-images:cuda-jupyter-pytorch-c9s-py39_2024b_20240418"
##########################
# Deploy Python packages #
##########################
USER 1001
WORKDIR /opt/app-root/bin/
# Copy packages list
COPY --chown=1001:0 requirements.txt ./
# Install packages and cleanup
# (all commands are chained to minimize layer size)
RUN echo "Installing softwares and packages" && \
# Install Python packages \
pip install --no-cache-dir -r requirements.txt && \
# Fix permissions to support pip in Openshift environments \
chmod -R g+w /opt/app-root/lib/python3.9/site-packages && \
fix-permissions /opt/app-root -P
WORKDIR /opt/app-root/src/
##########################
###########################
# Deploy OS Packages #
###########################
USER 0
WORKDIR /opt/app-root/bin/
# COPY --chown=1001:0 os-ide/os-packages.txt ./os-ide/os-packages.txt
# RUN yum install -y $(cat os-ide/os-packages.txt) && \
# rm -f os-ide/os-packages.txt && \
# yum -y clean all --enablerepo='*' && \
# rm -rf /var/cache/dnf && \
# find /var/log -type f -name "*.log" -exec rm -f {} \;
###########################
# Deploy CUDA Packages #
###########################
# Add the CUDA repository and GPG key
RUN dnf install -y dnf-plugins-core \
&& dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo \
&& rpm --import https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/7fa2af80.pub
# Install the CUDA toolkit (adjust version as needed)
RUN dnf clean all \
&& dnf -y install cuda-toolkit-11-2 \
&& dnf clean all
# Set PATH so it includes CUDA's bin directory
ENV PATH /usr/local/cuda-11.2/bin:${PATH}
# Set LD_LIBRARY_PATH to include CUDA's lib64 directory
ENV LD_LIBRARY_PATH /usr/local/cuda-11.2/lib64:${LD_LIBRARY_PATH}
###########################
##############################
# Deploy Jupyterlab packages #
##############################
USER 0
RUN dnf install -y jq
USER 1001
WORKDIR /opt/app-root/bin
# Copy packages list
COPY --chown=1001:0 requirements-jupyter.txt ./
# Copy notebook launcher and utils
COPY --chown=1001:0 utils utils/
COPY --chown=1001:0 start-notebook.sh ./
# Streamlit extension installation
COPY --chown=1001:0 streamlit-launcher.sh ./
COPY --chown=1001:0 streamlit-menu/dist/jupyterlab_streamlit_menu-0.1.0-py3-none-any.whl ./
# Copy Elyra setup to utils so that it's sourced at startup
COPY --chown=1001:0 setup-elyra.sh ./utils/
# Install packages and cleanup
# (all commands are chained to minimize layer size)
RUN echo "Installing softwares and packages" && \
# Install Python packages \
pip install --no-cache-dir -r requirements-jupyter.txt && \
pip install --no-cache-dir ./jupyterlab_streamlit_menu-0.1.0-py3-none-any.whl && \
rm -f ./jupyterlab_streamlit_menu-0.1.0-py3-none-any.whl && \
# setup path for runtime configuration \
mkdir /opt/app-root/runtimes && \
# switch to Data Science Pipeline \
cp utils/pipeline-flow.svg /opt/app-root/lib/python3.9/site-packages/elyra/static/icons/kubeflow.svg && \
sed -i "s/Kubeflow Pipelines/Data Science/g" /opt/app-root/lib/python3.9/site-packages/elyra/pipeline/runtime_type.py && \
sed -i "s/Kubeflow Pipelines/Data Science Pipelines/g" /opt/app-root/lib/python3.9/site-packages/elyra/metadata/schemas/kfp.json && \
sed -i "s/kubeflow-service/data-science-pipeline-service/g" /opt/app-root/lib/python3.9/site-packages/elyra/metadata/schemas/kfp.json && \
sed -i "s/\"default\": \"Argo\",/\"default\": \"Tekton\",/g" /opt/app-root/lib/python3.9/site-packages/elyra/metadata/schemas/kfp.json && \
# Workaround for passing ssl_sa_cert and to ensure that Elyra redirects to a correct pipeline run URL \
patch /opt/app-root/lib/python3.9/site-packages/elyra/pipeline/kfp/kfp_authentication.py -i utils/kfp_authentication.patch && \
patch /opt/app-root/lib/python3.9/site-packages/elyra/pipeline/kfp/processor_kfp.py -i utils/processor_kfp.patch && \
# switch to Data Science Pipeline in component catalog \
DIR_COMPONENT="/opt/app-root/lib/python3.9/site-packages/elyra/metadata/schemas/local-directory-catalog.json" && \
FILE_COMPONENT="/opt/app-root/lib/python3.9/site-packages/elyra/metadata/schemas/local-file-catalog.json" && \
URL_COMPONENT="/opt/app-root/lib/python3.9/site-packages/elyra/metadata/schemas/url-catalog.json" && \
tmp=$(mktemp) && \
jq '.properties.metadata.properties.runtime_type = input' $DIR_COMPONENT utils/component_runtime.json > "$tmp" && mv "$tmp" $DIR_COMPONENT && \
jq '.properties.metadata.properties.runtime_type = input' $FILE_COMPONENT utils/component_runtime.json > "$tmp" && mv "$tmp" $FILE_COMPONENT && \
jq '.properties.metadata.properties.runtime_type = input' $URL_COMPONENT utils/component_runtime.json > "$tmp" && mv "$tmp" $URL_COMPONENT && \
sed -i "s/metadata.metadata.runtime_type/\"DATA_SCIENCE_PIPELINES\"/g" /opt/app-root/share/jupyter/labextensions/@elyra/pipeline-editor-extension/static/lib_index_js.*.js && \
# Remove Elyra logo from JupyterLab because this is not a pure Elyra image \
sed -i "s/widget\.id === \x27jp-MainLogo\x27/widget\.id === \x27jp-MainLogo\x27 \&\& false/" /opt/app-root/share/jupyter/labextensions/@elyra/theme-extension/static/lib_index_js.*.js && \
# Replace Notebook's launcher, "(ipykernel)" with Python's version 3.x.y \
sed -i -e "s/Python.*/$(python --version | cut -d '.' -f-2)\",/" /opt/app-root/share/jupyter/kernels/python3/kernel.json && \
# Remove default Elyra runtime-images \
rm /opt/app-root/share/jupyter/metadata/runtime-images/*.json && \
# Fix permissions to support pip in Openshift environments \
chmod -R g+w /opt/app-root/lib/python3.9/site-packages && \
fix-permissions /opt/app-root -P
# Copy Elyra runtime-images definitions and set the version
COPY --chown=1001:0 runtime-images/ /opt/app-root/share/jupyter/metadata/runtime-images/
RUN sed -i "s/RELEASE/2024b/" /opt/app-root/share/jupyter/metadata/runtime-images/*.json
# Jupyter Server config to allow hidden files/folders in explorer. Ref: https://jupyterlab.readthedocs.io/en/latest/user/files.html#displaying-hidden-files
# Jupyter Lab config to hide disabled exporters (WebPDF, Qtpdf, Qtpng)
COPY --chown=1001:0 etc/ /opt/app-root/etc/jupyter/
USER 0
RUN echo "sudoers: files" >> /etc/nsswitch.conf
RUN dnf install -y sudo
RUN echo "default ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
USER 1001
WORKDIR /opt/app-root/src
ENTRYPOINT ["start-notebook.sh"]
apiVersion: kubeflow.org/v1
kind: Notebook
metadata:
annotations:
notebooks.opendatahub.io/inject-oauth: 'true'
notebooks.opendatahub.io/last-image-selection: 'custom-sudo-jupyteruser:sudo-jupyteruser'
notebooks.opendatahub.io/last-size-selection: Small
notebooks.opendatahub.io/oauth-logout-url: >-
https://rhods-dashboard-redhat-ods-applications.apps.cloud1c.example.denver.ocp.run/projects/modelserver?notebookLogout=sudo-jupyteruser
opendatahub.io/accelerator-name: ''
opendatahub.io/image-display-name: sudo-jupyteruser
opendatahub.io/username: cloud-user
openshift.io/description: ''
openshift.io/display-name: sudo-jupyteruser
labels:
app: sudo-jupyteruser
opendatahub.io/dashboard: 'true'
opendatahub.io/odh-managed: 'true'
opendatahub.io/user: cloud-2duser
name: sudo-jupyteruser
namespace: modelserver
spec:
template:
spec:
affinity: {}
initContainers:
- name: fix-permissions
image: registry.access.redhat.com/ubi8/python-39:1-176.1712880517
command: ["sh", "-c", "chown -R default:1001 /opt/app-root/src/"]
securityContext:
runAsUser: 0
allowPrivilegeEscalation: false
volumeMounts:
- mountPath: /opt/app-root/src
name: sudo-jupyteruser
containers:
- env:
- name: NOTEBOOK_ARGS
value: |-
--ServerApp.port=8888
--ServerApp.token=''
--ServerApp.password=''
--ServerApp.base_url=/notebook/modelserver/sudo-jupyteruser
--ServerApp.quit_button=False
--ServerApp.tornado_settings={"user":"cloud-2duser","hub_host":"https://rhods-dashboard-redhat-ods-applications.apps.cloud1c.example.denver.ocp.run","hub_prefix":"/projects/modelserver"}
- name: JUPYTER_IMAGE
value: >-
image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/custom-sudo-default-user-v3:sudo-jupyteruser-v3
image: >-
image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/custom-sudo-default-user-v3:sudo-jupyteruser-v3
imagePullPolicy: Always
livenessProbe:
failureThreshold: 3
httpGet:
path: /notebook/modelserver/sudo-jupyteruser/api
port: notebook-port
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
name: sudo-jupyteruser
ports:
- containerPort: 8888
name: notebook-port
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /notebook/modelserver/sudo-jupyteruser/api
port: notebook-port
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: '2'
memory: 8Gi
requests:
cpu: '1'
memory: 8Gi
securityContext:
allowPrivilegeEscalation: true
runAsGroup: 1001
runAsUser: 1001
volumeMounts:
- mountPath: /opt/app-root/src
name: sudo-jupyteruser
- mountPath: /dev/shm
name: shm
workingDir: /opt/app-root/src
- args:
- '--provider=openshift'
- '--https-address=:8443'
- '--http-address='
- '--openshift-service-account=sudo-jupyteruser'
- '--cookie-secret-file=/etc/oauth/config/cookie_secret'
- '--cookie-expire=24h0m0s'
- '--tls-cert=/etc/tls/private/tls.crt'
- '--tls-key=/etc/tls/private/tls.key'
- '--upstream=http://localhost:8888'
- '--upstream-ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
- '--email-domain=*'
- '--skip-provider-button'
- >-
--openshift-sar={"verb":"get","resource":"notebooks","resourceAPIGroup":"kubeflow.org","resourceName":"sudo-jupyteruser","namespace":"$(NAMESPACE)"}
- >-
--logout-url=https://rhods-dashboard-redhat-ods-applications.apps.cloud1c.example.denver.ocp.run/projects/modelserver?notebookLogout=sudo-jupyteruser
env:
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: >-
registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46
imagePullPolicy: Always
livenessProbe:
failureThreshold: 3
httpGet:
path: /oauth/healthz
port: oauth-proxy
scheme: HTTPS
initialDelaySeconds: 30
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
name: oauth-proxy
ports:
- containerPort: 8443
name: oauth-proxy
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /oauth/healthz
port: oauth-proxy
scheme: HTTPS
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 100m
memory: 64Mi
requests:
cpu: 100m
memory: 64Mi
volumeMounts:
- mountPath: /etc/oauth/config
name: oauth-config
- mountPath: /etc/tls/private
name: tls-certificates
enableServiceLinks: false
serviceAccountName: sudo-jupyteruser
volumes:
- name: sudo-jupyteruser
persistentVolumeClaim:
claimName: sudo-jupyteruser
- emptyDir:
medium: Memory
name: shm
- name: oauth-config
secret:
defaultMode: 420
secretName: sudo-jupyteruser-oauth-config
- name: tls-certificates
secret:
defaultMode: 420
secretName: sudo-jupyteruser-tls
apiVersion: kubeflow.org/v1
kind: Notebook
metadata:
annotations:
notebooks.opendatahub.io/inject-oauth: 'true'
notebooks.opendatahub.io/last-image-selection: 'custom-sudo-jupyteruser:sudo-jupyteruser'
notebooks.opendatahub.io/last-size-selection: Small
notebooks.opendatahub.io/oauth-logout-url: >-
https://rhods-dashboard-redhat-ods-applications.apps.cloud1c.example.denver.ocp.run/projects/modelserver?notebookLogout=sudo-jupyteruser
opendatahub.io/accelerator-name: ''
opendatahub.io/image-display-name: sudo-jupyteruser
opendatahub.io/username: cloud-user
openshift.io/description: ''
openshift.io/display-name: sudo-jupyteruser
labels:
app: sudo-jupyteruser
opendatahub.io/dashboard: 'true'
opendatahub.io/odh-managed: 'true'
opendatahub.io/user: cloud-2duser
name: sudo-jupyteruser
namespace: modelserver
spec:
template:
spec:
affinity: {}
containers:
- env:
- name: NOTEBOOK_ARGS
value: |-
--ServerApp.port=8888
--ServerApp.token=''
--ServerApp.password=''
--ServerApp.base_url=/notebook/modelserver/sudo-jupyteruser
--ServerApp.quit_button=False
--ServerApp.tornado_settings={"user":"cloud-2duser","hub_host":"https://rhods-dashboard-redhat-ods-applications.apps.cloud1c.example.denver.ocp.run","hub_prefix":"/projects/modelserver"}
- name: JUPYTER_IMAGE
value: >-
image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/custom-sudo-default-user-v3:sudo-jupyteruser-v3
image: >-
image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/custom-sudo-default-user-v3:sudo-jupyteruser-v3
imagePullPolicy: Always
livenessProbe:
failureThreshold: 3
httpGet:
path: /notebook/modelserver/sudo-jupyteruser/api
port: notebook-port
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
name: sudo-jupyteruser
ports:
- containerPort: 8888
name: notebook-port
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /notebook/modelserver/sudo-jupyteruser/api
port: notebook-port
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: '2'
memory: 8Gi
requests:
cpu: '1'
memory: 8Gi
securityContext:
allowPrivilegeEscalation: true
runAsGroup: 1001
runAsUser: 1001
volumeMounts:
- mountPath: /opt/app-root/src
name: sudo-jupyteruser
- mountPath: /dev/shm
name: shm
workingDir: /opt/app-root/src
- args:
- '--provider=openshift'
- '--https-address=:8443'
- '--http-address='
- '--openshift-service-account=sudo-jupyteruser'
- '--cookie-secret-file=/etc/oauth/config/cookie_secret'
- '--cookie-expire=24h0m0s'
- '--tls-cert=/etc/tls/private/tls.crt'
- '--tls-key=/etc/tls/private/tls.key'
- '--upstream=http://localhost:8888'
- '--upstream-ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
- '--email-domain=*'
- '--skip-provider-button'
- >-
--openshift-sar={"verb":"get","resource":"notebooks","resourceAPIGroup":"kubeflow.org","resourceName":"sudo-jupyteruser","namespace":"$(NAMESPACE)"}
- >-
--logout-url=https://rhods-dashboard-redhat-ods-applications.apps.cloud1c.example.denver.ocp.run/projects/modelserver?notebookLogout=sudo-jupyteruser
env:
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: >-
registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46
imagePullPolicy: Always
livenessProbe:
failureThreshold: 3
httpGet:
path: /oauth/healthz
port: oauth-proxy
scheme: HTTPS
initialDelaySeconds: 30
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
name: oauth-proxy
ports:
- containerPort: 8443
name: oauth-proxy
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /oauth/healthz
port: oauth-proxy
scheme: HTTPS
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 100m
memory: 64Mi
requests:
cpu: 100m
memory: 64Mi
volumeMounts:
- mountPath: /etc/oauth/config
name: oauth-config
- mountPath: /etc/tls/private
name: tls-certificates
enableServiceLinks: false
serviceAccountName: sudo-jupyteruser
volumes:
- name: sudo-jupyteruser
persistentVolumeClaim:
claimName: sudo-jupyteruser
- emptyDir:
medium: Memory
name: shm
- name: oauth-config
secret:
defaultMode: 420
secretName: sudo-jupyteruser-oauth-config
- name: tls-certificates
secret:
defaultMode: 420
secretName: sudo-jupyteruser-tls
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment