Kha khanetor

## joblib_on_docker.md

      
              1 file
            
          
              0 forks
            
          
              0 comments
            
          
              1 star
            
          
                harusametime
                / joblib_on_docker.md
            
            
              Last active
              February 11, 2022 20:18
            
              
                Note on joblib with docker
              
          
    Problem

When passing big arrays to joblib.Parallel in docker container, parallel processing does not start immediately.
Why?

Joblib uses the folder specified by "JOBLIB_TEMP_FOLDER" for memmap of the arrays. Without specifying, JOBLIB_TEMP_FOLDER is set to /dev/shm, which usually has small size and is not enough for the big arrays.
https://pythonhosted.org/joblib/generated/joblib.Parallel.html
Solution


Set JOBLIB_TEMP_FOLDER