Unable to set SCHED_FIFO in a container

README.md

Unable to set SCHED_FIFO in a container

Build the image from the Dockerfile in this gist while making sure sched.cc is available in the build directory. Then, run the following steps in a bash shell.

Run the container (please note this applies the suggestions from scylladb/seastar#382 (comment), that is runs as root in the container, and assigns the container the CAP_SYS_NICE capability):

docker run --cap-add=sys_nice --rm -it --user root --entrypoint /bin/bash 63eb9aaa36520fc08994f87caba98c90f18f86581bb95e4e98803732ceca08a6

Within the container:

root@7f39c815a0ad:/# capsh --print | grep Current: | grep sys_nic
Current: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_sys_nice,cap_mknod,cap_audit_write,cap_setfcap=eip

That's good, cap_sys_nice is in the list of enabled capabilities.

Let's check kernel version (this, again, confirms we follow the best practice from scylladb/seastar#382 (comment)):

root@4c97290d99a5:/# uname -a
Linux 4c97290d99a5 5.10.76-linuxkit #1 SMP Mon Nov 8 10:21:19 UTC 2021 x86_64 GNU/Linux

Let's run the test program:

root@7f39c815a0ad:/# ./sched
created child thread: 1913747200
Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE

The test program cannot set SCHED_FIFO - why?

For comparison, the same program on a host macOS Catalina 10.15.7:

$ ./sched
created child thread: 172994560
set scheduling policy SCHED_FIFO
I'm the child thread

Dockerfile

FROM vectorized/redpanda:v21.10.2

USER root

RUN apt-get update \
 && apt-get install -y build-essential gcc g++ libcap2-bin \
 && apt-get clean \
 && rm -rf /var/cache/apt/archives

COPY sched.cc sched.cc

RUN CXXFLAGS=-pthread make sched

sched.cc

// See Seastar - a Redpanda component - code at
// https://github.com/vectorizedio/seastar/blob/f8ec733c36f0829d56a17103c916154a946128be/src/core/reactor_backend.cc#L705
//
//void reactor_backend_epoll::start_tick() {
//    _task_quota_timer_thread = std::thread(&reactor_backend_epoll::task_quota_timer_thread_fn, this);

//    ::sched_param sp;
//    sp.sched_priority = 1;
//    auto sched_ok = pthread_setschedparam(_task_quota_timer_thread.native_handle(), SCHED_FIFO, &sp);
//    if (sched_ok != 0 && _r._id == 0) {
//        seastar_logger.warn("Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE");
//    }
//}

#include <cstdio>
#include <cstdlib>
#include <pthread.h>

void* run(void*) {
        printf("I'm the child thread\n");
        return NULL;
}

int main() {
        pthread_t thread;
        auto create_ok = pthread_create(&thread, NULL, run, NULL);
        if (create_ok != 0) {
                printf("cannot create thread: %d\n", create_ok);
                return -1;
        }
        printf("created child thread: %d\n", thread);

        ::sched_param sp;
        auto sched_ok = pthread_setschedparam(thread, SCHED_FIFO, &sp);
        if (sched_ok != 0) {
                printf("Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE\n");
                return -2;
        }
        printf("set scheduling policy SCHED_FIFO\n");

        pthread_join(thread, NULL);
}

Relevant issues:

• scylladb/scylla-operator#107
• scylladb/seastar#382
• redpanda-data/redpanda#3593