Intel Colfax Cluster - Notes - Index Page
In this post we are going to show you a very simple application that will use up all 256 threads on a Knights Landing (KNL) node to simultaneously print "hello world". Imagine 256 people all say "hello world" in the same time. Each person is represented by a thread. We will use the OpenMP framework to write our parallel (Multi-thread) application.
Here comes the demo...
johnny@Chuns-MBP $ ssh colfax
######################################################################
# Welcome to Colfax Cluster!
######################################################################
#
# Pre-compiled Machine Learning Frameworks, as well as other tools
# are available in the /opt/ directory
#
# If you have any questions or feature requests, post them on our
# forum at:
# https://colfaxresearch.com/discussion/forum/
#
# Colfax Research Team
######################################################################
Last login: Wed Aug 9 05:49:11 2017 from 10.5.0.7
[u4443@c001 ~]$
We are now at the Login Node of Cluster c001.
Navigate to a working directory of your choice. For example:
[u4443@c001 ~]$ cd deepdive/lec-04/
[u4443@c001 lec-04]$ pwd
/home/u4443/deepdive/lec-04
Create a C++ code called hello-parallel.cc:
[u4443@c001 lec-04]$ emacs hello-parallel.cc
Code:
#include <omp.h>
#include <cstdio>
int main() {
// This code is executed by 1 thread
const int num_threads = omp_get_max_threads();
printf("OpenMP with %d threads\n", num_threads);
#pragma omp parallel
{ // This code is executed in parallel
// by multiple threads
printf("Hello World from thread %d\n", omp_get_thread_num());
};
};Note:
- the
omp.hdirectory provides us the required utilities to write multi-thread codes - the
omp_get_max_threads()returns the max thread value. If the system has 8 threads, this would be 8 (e.g. Login Node). If the system has 256 threads, this would be 256 (e.g. KNL Node). - the
omp_get_thread_num()returns the thread ID. We print the thread ID so we can visualize the parallelism in work later on. - the
#pragma omp parallel- this is the secret sauce that executes whatever instructions in the scope below in parallel, using all threads available on the system.
Save and exit editor.
We can compile the code with the Intel C++ Compiler (icpc) or GCC Compiler.
If we use icpc (use the -qopenmp option to use OpenMP):
[u4443@c001 lec-04]$ icpc -qopenmp -o hello-parallel hello-parallel.cc
If we use gcc (use the -fopenmp option to use OpenMP):
[u4443@c001 lec-04]$ gcc -fopenmp -o hello-parallel hello-parallel.cc
This will create a binary executable hello-parallel.
A shell script will enable us to run the application easily.
Create a shell script hello-parallel.sh in the same directory like this (tweak the working directory to the one of your choice):
echo "hello-parallel starts"
#export OMP_NUM_THREADS=10
cd /home/u4443/deepdive/lec-04
./hello-parallel
echo "hello-parallel ends"Within the working direcotyr where all our files are, submit the shell script to execute on a KNL node (if you would prefer a non KNL node, just replace knl with something else as long as it's available):
[u4443@c001 lec-04]$ qsub hello-parallel.sh -l nodes=1:knl
21140.c001
Make a note of the job number returned by the command. Now view the output file. Notice we are printing the 256 "hello" concurrently.
[u4443@c001 lec-04]$ cat hello-parallel.sh.o21140
########################################################################
# Colfax Cluster - https://colfaxresearch.com/
# Date: Thu Aug 10 02:07:38 PDT 2017
# Job ID: 21140.c001
# User: u4443
# Resources: neednodes=1:knl,nodes=1:knl,walltime=24:00:00
########################################################################
hello-parallel starts
OpenMP with 256 threads
Hello World from thread 131
Hello World from thread 199
Hello World from thread 1
Hello World from thread 132
Hello World from thread 69
Hello World from thread 200
Hello World from thread 141
Hello World from thread 124
Hello World from thread 177
Hello World from thread 33
Hello World from thread 59
Hello World from thread 190
Hello World from thread 156
Hello World from thread 148
Hello World from thread 221
Hello World from thread 149
Hello World from thread 4
Hello World from thread 24
Hello World from thread 157
Hello World from thread 104
Hello World from thread 66
Hello World from thread 29
Hello World from thread 243
Hello World from thread 254
Hello World from thread 255
Hello World from thread 0
Hello World from thread 65
Hello World from thread 68
Hello World from thread 71
Hello World from thread 136
Hello World from thread 8
Hello World from thread 72
Hello World from thread 204
Hello World from thread 12
Hello World from thread 76
Hello World from thread 140
Hello World from thread 205
Hello World from thread 13
Hello World from thread 77
Hello World from thread 137
Hello World from thread 201
Hello World from thread 73
Hello World from thread 9
Hello World from thread 80
Hello World from thread 208
Hello World from thread 16
Hello World from thread 60
Hello World from thread 188
Hello World from thread 240
Hello World from thread 48
Hello World from thread 176
Hello World from thread 112
Hello World from thread 224
Hello World from thread 160
Hello World from thread 96
Hello World from thread 32
Hello World from thread 241
Hello World from thread 49
Hello World from thread 113
Hello World from thread 17
Hello World from thread 179
Hello World from thread 51
Hello World from thread 242
Hello World from thread 114
Hello World from thread 115
Hello World from thread 81
Hello World from thread 209
Hello World from thread 145
Hello World from thread 50
Hello World from thread 178
Hello World from thread 125
Hello World from thread 189
Hello World from thread 61
Hello World from thread 97
Hello World from thread 161
Hello World from thread 225
Hello World from thread 248
Hello World from thread 120
Hello World from thread 11
Hello World from thread 74
Hello World from thread 10
Hello World from thread 139
Hello World from thread 78
Hello World from thread 143
Hello World from thread 184
Hello World from thread 14
Hello World from thread 79
Hello World from thread 203
Hello World from thread 202
Hello World from thread 56
Hello World from thread 75
Hello World from thread 138
Hello World from thread 207
Hello World from thread 142
Hello World from thread 15
Hello World from thread 206
Hello World from thread 123
Hello World from thread 122
Hello World from thread 187
Hello World from thread 58
Hello World from thread 186
Hello World from thread 62
Hello World from thread 126
Hello World from thread 191
Hello World from thread 127
Hello World from thread 63
Hello World from thread 249
Hello World from thread 83
Hello World from thread 146
Hello World from thread 211
Hello World from thread 18
Hello World from thread 57
Hello World from thread 147
Hello World from thread 82
Hello World from thread 121
Hello World from thread 185
Hello World from thread 210
Hello World from thread 19
Hello World from thread 44
Hello World from thread 220
Hello World from thread 92
Hello World from thread 180
Hello World from thread 28
Hello World from thread 244
Hello World from thread 52
Hello World from thread 116
Hello World from thread 84
Hello World from thread 212
Hello World from thread 20
Hello World from thread 3
Hello World from thread 152
Hello World from thread 93
Hello World from thread 25
Hello World from thread 217
Hello World from thread 89
Hello World from thread 153
Hello World from thread 7
Hello World from thread 213
Hello World from thread 21
Hello World from thread 6
Hello World from thread 85
Hello World from thread 31
Hello World from thread 222
Hello World from thread 223
Hello World from thread 158
Hello World from thread 5
Hello World from thread 159
Hello World from thread 94
Hello World from thread 30
Hello World from thread 95
Hello World from thread 67
Hello World from thread 252
Hello World from thread 87
Hello World from thread 150
Hello World from thread 214
Hello World from thread 215
Hello World from thread 23
Hello World from thread 22
Hello World from thread 219
Hello World from thread 154
Hello World from thread 151
Hello World from thread 86
Hello World from thread 91
Hello World from thread 218
Hello World from thread 26
Hello World from thread 155
Hello World from thread 90
Hello World from thread 27
Hello World from thread 130
Hello World from thread 2
Hello World from thread 106
Hello World from thread 43
Hello World from thread 39
Hello World from thread 230
Hello World from thread 103
Hello World from thread 102
Hello World from thread 167
Hello World from thread 38
Hello World from thread 166
Hello World from thread 234
Hello World from thread 235
Hello World from thread 107
Hello World from thread 42
Hello World from thread 170
Hello World from thread 171
Hello World from thread 231
Hello World from thread 195
Hello World from thread 193
Hello World from thread 192
Hello World from thread 64
Hello World from thread 163
Hello World from thread 34
Hello World from thread 98
Hello World from thread 35
Hello World from thread 162
Hello World from thread 226
Hello World from thread 99
Hello World from thread 227
Hello World from thread 53
Hello World from thread 117
Hello World from thread 128
Hello World from thread 181
Hello World from thread 245
Hello World from thread 236
Hello World from thread 108
Hello World from thread 172
Hello World from thread 168
Hello World from thread 40
Hello World from thread 232
Hello World from thread 45
Hello World from thread 237
Hello World from thread 109
Hello World from thread 173
Hello World from thread 247
Hello World from thread 246
Hello World from thread 55
Hello World from thread 183
Hello World from thread 118
Hello World from thread 100
Hello World from thread 182
Hello World from thread 54
Hello World from thread 228
Hello World from thread 164
Hello World from thread 36
Hello World from thread 169
Hello World from thread 105
Hello World from thread 233
Hello World from thread 41
Hello World from thread 229
Hello World from thread 37
Hello World from thread 165
Hello World from thread 101
Hello World from thread 47
Hello World from thread 110
Hello World from thread 111
Hello World from thread 174
Hello World from thread 175
Hello World from thread 46
Hello World from thread 239
Hello World from thread 238
Hello World from thread 196
Hello World from thread 135
Hello World from thread 194
Hello World from thread 198
Hello World from thread 216
Hello World from thread 88
Hello World from thread 253
Hello World from thread 251
Hello World from thread 134
Hello World from thread 250
Hello World from thread 119
Hello World from thread 129
Hello World from thread 197
Hello World from thread 133
Hello World from thread 70
Hello World from thread 144
hello-parallel ends
########################################################################
# Colfax Cluster
# End of output for job 21140.c001
# Date: Thu Aug 10 02:07:40 PDT 2017
########################################################################
[u4443@c001 lec-04]$
A bit of explaination:
hello-parallel starts, this is our 1st (single-thread) process in the serial stream.#export OMP_NUM_THREADS=4, this line is currently commented out. If we uncomment this line however, the job will use 4 threads instead of all 256 threads. This line may be used to control how many threads we wish to use at a time. Try uncomment this line and run job again - you shall see less parallel processes.OpenMP with 256 threads, this is our 2nd (single-thread) process in the serial stream.Hello World from thread XXX, this is our 3rd (multi-thread) process in the serial stream, handled by 256 threads concurrently.hello-parallel ends, this is our 4th (single-thread) process in the serial stream.
So here we are, a simple Hello World Multi-Thread Application running on a KNL node, using all 256 threads!
It is usually not recommended to run any heavy lifting job at the Login Node. (Instead, remote submit a job via qsub to a cluster node like a KNL node is recommended). For illustrating purpose, let's run this hello world application on the login node (as it doesn't really do much other than to print hello). We should expect to see 8 parallel threads printing "Hello World" (as there are only 8 threads max on Login node):
[u4443@c001 lec-04]$ ./hello-parallel
OpenMP with 8 threads
Hello World from thread 0
Hello World from thread 2
Hello World from thread 3
Hello World from thread 1
Hello World from thread 4
Hello World from thread 6
Hello World from thread 7
Hello World from thread 5
[u4443@c001 lec-04]$
Here we go, 8 threads on the Login node. So at maximum we can only handle 8 parallel processes at a time. This is one of the reasons why we should always submit job on a cluster node - more CPUs / resources over there!
In this post we have demonstrate a simple C++ Hello World application using a combination of serial and parallel processing. We used the OpenMP framework to handle the Parallel Processing part - having 256 threads to print "hello world" concurrently.