Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
a collection of my notes while working on nanopore basecalling on the Jetson Xavier

Jetson Xavier basecalling notes

initial basecalling runs

'fast' flip-flop calling on the Jetson Xavier

guppy_basecaller --disable_pings --compress_fastq -c dna_r9.4.1_450bps_fast.cfg -i flongle_fast5_pass/ -s flongle_test2 -x 'auto' --recursive 
high-accuracy calling with base modifications on the Jetson Xavier
guppy_basecaller --disable_pings --compress_fastq -c dna_r9.4.1_450bps_modbases_dam-dcm-cpg_hac.cfg --fast5_out -i flongle_fast5_pass/ -s flongle_hac_fastq -x 'auto' --recursive 
$ guppy_basecaller --compress_fastq -c dna_r9.4.1_450bps_modbases_dam-dcm-cpg_hac.cfg -i flongle_fast5_pass/ -s flongle_hac_fastq -x 'auto' --recursive
ONT Guppy basecalling software version 3.4.1+213a60d0
config file:        /opt/ont/guppy/data/dna_r9.4.1_450bps_modbases_dam-dcm-cpg_hac.cfg
model file:         /opt/ont/guppy/data/template_r9.4.1_450bps_modbases_dam-dcm-cpg_hac.jsn
input path:         flongle_fast5_pass/
save path:          flongle_hac_fastq
chunk size:         1000
chunks per runner:  512
records per file:   4000
fastq compression:  ON
num basecallers:    1
gpu device:         auto
kernel path:
runners per device: 4

Found 105 fast5 files to process.
Init time: 2790 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 2493578 ms, Samples called: 3970728746, samples/s: 1.59238e+06
Finishing up any open output files.
Basecalling completed successfully.

So from the above we see in high accuracy mode it take the Xavier ~41 minutes to complete the base calling using the default configuration files. For reference the fast calling mode was ~8 minutes.

optimising settings for Jetson Xavier

  • When performing GPU basecalling there is always one CPU support thread per GPU caller, so the number of callers (--num_callers) dictates the maximum number of CPU threads used.
  • Max chunks per runner (--chunks_per_runner): The maximum number of chunks which can be submitted to a single neural network runner before it starts computation. Increasing this figure will increase GPU basecalling performance when it is enabled.
  • Number of GPU runners per device (--gpu_runners_per_device): The number of neural network runners to create per CUDA device. Increasing this number may improve performance on GPUs with a large number of compute cores, but will increase GPU memory use. This option only affects GPU calling.

There is a rough equation to estimate amount of ram:

runners * chunks_per_runner * chunk_size < 100000 * [max GPU memory in GB]

For example, a GPU with 8 GB of memory would require:

runners * chunks_per_runner * chunk_size < 800000

some suggested settings from ONT

NVIDIA Jetson TX2
--num_callers 1
--gpu_runners_per_device 2
--chunks_per_runner 48

from hac config file (dna_r9.4.1_450bps_modbases_dam-dcm-cpg_hac.cfg)

chunk_size                          = 1000
gpu_runners_per_device              = 4
chunks_per_runner                   = 512
chunks_per_caller                   = 10000

modified testing

'fast' flip-flop calling on the Jetson Xavier

guppy_basecaller --disable_pings --compress_fastq -c dna_r9.4.1_450bps_fast.cfg -i flongle_fast5_pass/ \
  -s flongle_test2 -x 'auto' --recursive --num_callers 4 --gpu_runners_per_device 8 --chunks_per_runner 256
$ guppy_basecaller --disable_pings --compress_fastq -c dna_r9.4.1_450bps_fast.cfg \
-i flongle_fast5_pass/ -s flongle_test2 -x 'auto' --recursive --num_callers 4 \
--gpu_runners_per_device 8 --chunks_per_runner 256
ONT Guppy basecalling software version 3.4.1+213a60d0
config file:        /opt/ont/guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /opt/ont/guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass/
save path:          flongle_test2
chunk size:         1000
chunks per runner:  256
records per file:   4000
fastq compression:  ON
num basecallers:    4
gpu device:         auto
kernel path:
runners per device: 8

Found 105 fast5 files to process.
Init time: 880 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 428745 ms, Samples called: 3970269916, samples/s: 9.26021e+06
Finishing up any open output files.
Basecalling completed successfully.

I was able to shave a minute off the fast model on the Xavier (above) getting it down to ~7 minutes.

jetson_xavier_jtop_screenshot

Update: (13th Dec 2019)

Just modifying the number of chunks per runner has allowed me to get the time down to under 6.5 mins (see table below).

chunks_per_runner time
(160) default ~8 mins
256 7 mins 6 secs
512 6 mins 28 secs
1024 6 min 23 secs

It looks like we might have reached an optimal point here. Next I'll test some of the other parameters and see if we can speed this up further.

high-accuracy calling with base modifications on the Jetson Xavier

guppy_basecaller --disable_pings --compress_fastq -c dna_r9.4.1_450bps_modbases_dam-dcm-cpg_hac.cfg \
  --num_callers 4 --gpu_runners_per_device 8 --fast5_out -i flongle_fast5_pass/ \
  -s flongle_hac_basemod_fastq -x 'auto' --recursive
increased number of callers
$ guppy_basecaller --disable_pings --compress_fastq -c dna_r9.4.1_450bps_fast.cfg \
  -i flongle_fast5_pass/ -s flongle_test2 -x 'auto' --recursive --num_callers 8 \
  --gpu_runners_per_device 8 --chunks_per_runner 1024
ONT Guppy basecalling software version 3.4.1+213a60d0
config file:        /opt/ont/guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /opt/ont/guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass/
save path:          flongle_test2
chunk size:         1000
chunks per runner:  1024
records per file:   4000
fastq compression:  ON
num basecallers:    8 
gpu device:         auto
kernel path:
runners per device: 8 

Found 105 fast5 files to process.
Init time: 897 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 383865 ms, Samples called: 3970269916, samples/s: 1.03429e+07
Finishing up any open output files.
Basecalling completed successfully.
increased chunk size
$ guppy_basecaller --disable_pings --compress_fastq -c dna_r9.4.1_450bps_fast.cfg \
  -i flongle_fast5_pass/ -s flongle_test2 -x 'auto' --recursive --num_callers 4 \
  --gpu_runners_per_device 8 --chunks_per_runner 1024 --chunk_size 2000
ONT Guppy basecalling software version 3.4.1+213a60d0
config file:        /opt/ont/guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /opt/ont/guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass/
save path:          flongle_test2
chunk size:         2000
chunks per runner:  1024
records per file:   4000
fastq compression:  ON
num basecallers:    4
gpu device:         auto
kernel path:
runners per device: 8

Found 105 fast5 files to process.
Init time: 1180 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 503532 ms, Samples called: 3970269916, samples/s: 7.88484e+06
Finishing up any open output files.
Basecalling completed successfully.
increased runners per device and number of callers
$ guppy_basecaller --disable_pings --compress_fastq -c dna_r9.4.1_450bps_fast.cfg \
  -i flongle_fast5_pass/ -s flongle_test2 -x 'auto' --recursive --num_callers 8 \
  --gpu_runners_per_device 16 --chunks_per_runner 1024 --chunk_size 1000
ONT Guppy basecalling software version 3.4.1+213a60d0
config file:        /opt/ont/guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /opt/ont/guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass/
save path:          flongle_test2
chunk size:         1000
chunks per runner:  1024
records per file:   4000
fastq compression:  ON
num basecallers:    8
gpu device:         auto
kernel path:
runners per device: 16

Found 105 fast5 files to process.
Init time: 1113 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 383466 ms, Samples called: 3970269916, samples/s: 1.03536e+07
Finishing up any open output files.
Basecalling completed successfully.

current 'optimal' parameters

The below parameters seem to provide the 'optimal' speed increase with a resultant run time of 6 mins and 23 secs.

$ guppy_basecaller --disable_pings --compress_fastq -c dna_r9.4.1_450bps_fast.cfg \
  -i flongle_fast5_pass/ -s flongle_test2 -x 'auto' --recursive --num_callers 4 \
  --gpu_runners_per_device 8 --chunks_per_runner 1024 --chunk_size 1000
ONT Guppy basecalling software version 3.4.1+213a60d0
config file:        /opt/ont/guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /opt/ont/guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass/
save path:          flongle_test2
chunk size:         1000
chunks per runner:  1024
records per file:   4000
fastq compression:  ON
num basecallers:    4
gpu device:         auto
kernel path:
runners per device: 8

Found 105 fast5 files to process.
Init time: 926 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 382714 ms, Samples called: 3970269916, samples/s: 1.0374e+07
Finishing up any open output files.
Basecalling completed successfully.

exploring portable batteries and power modes

We are currently using a 27000mAh AC Portable Charger from Ravpower.

Below: Ravpower Xtreme Series 27000mAh AC Portable Charger ravpower battery package This battery bank/charger has a built in 220V AC outlet and 1 usb-c and 2 usb 3.1 outputs.

Below: powerbank charging from the wall. ravpower battery on charge Ravpower claim this powerbank will charge a smartphone 11 times, a tablet 4 times or a laptop 3 times.

Below: running our first portable Xavier GPU basecalling of nanopore data! xavier running on battery power

changing power modes

Running the Xavier in different power states obviously influences the amount of run time on the battery.

Power mode Time
10W 33.4 mins
15W 14.3 mins
30W 2 cores 10.8 mins
30W 4 cores 10.8 mins
30W MAX (8 cores) 7.5 mins

The above benchmarks were performed on data generated from a flongle run (~0.5 Mb of sequence or 5.5 Gb of actual data).


potential V100 examples

V100 config example for high accuracy model
guppy_basecaller \
--disable_pings \
--compress_fastq \
-c dna_r9.4.1_450bps_modbases_dam-dcm-cpg_hac.cfg \
--ipc_threads 16 \
--num_callers 8 \
--gpu_runners_per_device 4 \
--chunks_per_runner 512 \
--device "cuda:0 cuda:1" \ # this parameter should now scale nicely across both cards, I haven't checked though
--recursive \
--fast5_out \
-i fast5_input \
-s fastq_output
V100 config example for fast calling model
guppy_basecaller \
--disable_pings \
--compress_fastq \
-c dna_r9.4.1_450bps_fast.cfg \
--ipc_threads 16 \
--num_callers 8 \
--gpu_runners_per_device 64 \
--chunks_per_runner 256 \
--device "cuda:0 cuda:1" \
--recursive \
-i fast5_input \
-s fastq_output

Guppy basecalling benchmarking on a Titan RTX

There has been some discussion about the recent release of Guppy (3.4.1 and 3.4.2) in terms of speed. I was interested in running some benchmarks across different versions. I had a hunch it may have been something to do with the newly introduced compression of the fast5 files...

Test parameters

The only things I am changing are the version of Guppy being used, and in the case of 3.4.3 I am trying with and without vbz compression of the fast5 files. Everything else is as below:

System:

  • Debian Sid (unstable)
  • 2x 12-Core Intel Xeon Gold 5118 (48 threads)
  • 256Gb RAM
  • Titan RTX
  • Nvidia drivers: 418.56
  • CUDA Version: 10.1

Guppy GPU basecalling parameters:

  • --disable_pings
  • --compress_fastq
  • --dna_r9.4.1_450bps_fast.cfg
  • --num_callers 8
  • --gpu_runners_per_device 64
  • --chunks_per_runner 256
  • --device "cuda:0"
  • --recursive

For each Guppy version I ran the basecaller three times in an attempt to ensure that results were consistent*.

Note: I chose the fast basecalling model as I wanted to do a quick set of benchmarks. If I feel up to it I may do the same thing for the high accuracy caller...

* Spoiler, I didn't originally do this and it proved misleading...

Results

guppy version time (seconds) samples/s
3.1.5# 93.278 4.25638e+07
3.2.4# 94.141 4.21737e+07
3.3.0# 94.953 4.1813e+07
3.3.3# 95.802 4.14425e+07
3.4.1 (no vbz compressed fast5) 79.913 4.96824e+07
3.4.1 (vbz compressed fast5) 90.895 4.36797e+07
3.4.3 (no vbz compressed fast5) 90.674 4.37862e+07
3.4.3 (vbz compressed fast5) 82.877 4.79056e+07

# these versions of Guppy did not support vbz compression of fast5 files (pre 3.4.X from memory).

Summary to date

I initially thought that there was something off with the compression imlementation in 3,4,3 as my first run on uncompressed data was ~3x slower than the run on the compressed data. When I grabbed 3.4.1 to perform the same check I noticed that it was fairly consistent between compressed and not. So I went back and was more rigorous and performed 3 iterations of each run for each version, ditto for versions 3.4.X compressed and not. This proved that the initial run was an anomaly and should be disregarded.

What was quite interesting is that running on vbz compressed fast5 data appears to be in the range of 8-10 seconds faster than uncompressed. So there is a slight added speed benefit on top of the nice reduction in file size - which is a little nicer for the SSD/HDD.

So at this stage I can't confirm any detrimental speed issues when using Guppy version 3.4.X, but this needs to be caveated with all the usual disclaimers:

  • all systems are different (I'm not on Ubuntu for instance).
  • drivers are different (I need to update).
  • GPUs are very different, i.e. many people (including me) are using 'non-supported' GPUs - in my case a Titan RTX which is no slouch.
    • For what it's worth, I can add a comment here saying that I haven't had any speed issues with basecalling on our Nvidia Jetson Xaviers using Guppy 3.4.1.
  • our 'little' Linux server isn't exactly a slouch either - so laptop/desktop builds could be very different.
  • I ran the fast basecaller (I'm currently flat out and can't wait for the high accuracy caller) - I may take a subset of data and revist with hac at some stage.

You can view the 'raw' results/output for each run below:

Guppy 3.1.5

~/Downloads/software/guppy/3.1.5/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    --num_callers 8 \
    --gpu_runners_per_device 64 \
    --chunks_per_runner 256 \
    --device "cuda:0" \
    --recursive \
    -i flongle_fast5_pass \
    -s testrun_fast_3.1.5

ONT Guppy basecalling software version 3.1.5+781ed57
config file:        /home/miles/Downloads/software/guppy/3.1.5/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /home/miles/Downloads/software/guppy/3.1.5/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass
save path:          testrun_fast_3.1.5
chunk size:         1000
chunks per runner:  256
records per file:   4000
fastq compression:  ON
num basecallers:    8
gpu device:         cuda:0
kernel path:        
runners per device: 64

Found 105 fast5 files to process.
Init time: 1000 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 93278 ms, Samples called: 3970269916, samples/s: 4.25638e+07
Finishing up any open output files.
Basecalling completed successfully.

Guppy 3.2.4

~/Downloads/software/guppy/3.2.4/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    --num_callers 8 \
    --gpu_runners_per_device 64 \
    --chunks_per_runner 256 \
    --device "cuda:0" \
    --recursive \
    -i flongle_fast5_pass \
    -s testrun_fast_3.2.4

ONT Guppy basecalling software version 3.2.4+d9ed22f
config file:        /home/miles/Downloads/software/guppy/3.2.4/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /home/miles/Downloads/software/guppy/3.2.4/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass
save path:          testrun_fast_3.2.4
chunk size:         1000
chunks per runner:  256
records per file:   4000
fastq compression:  ON
num basecallers:    8
gpu device:         cuda:0
kernel path:        
runners per device: 64

Found 105 fast5 files to process.
Init time: 836 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 94141 ms, Samples called: 3970269916, samples/s: 4.21737e+07
Finishing up any open output files.
Basecalling completed successfully.

Guppy 3.3.0

~/Downloads/software/guppy/3.3.0/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    --num_callers 8 \
    --gpu_runners_per_device 64 \
    --chunks_per_runner 256 \
    --device "cuda:0" \
    --recursive \
    -i flongle_fast5_pass \
    -s testrun_fast_3.3.0

ONT Guppy basecalling software version 3.3.0+ef22818
config file:        /home/miles/Downloads/software/guppy/3.3.0/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /home/miles/Downloads/software/guppy/3.3.0/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass
save path:          testrun_fast_3.3.0
chunk size:         1000
chunks per runner:  256
records per file:   4000
fastq compression:  ON
num basecallers:    8
gpu device:         cuda:0
kernel path:        
runners per device: 64

Found 105 fast5 files to process.
Init time: 722 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 94953 ms, Samples called: 3970269916, samples/s: 4.1813e+07
Finishing up any open output files.
Basecalling completed successfully.

Guppy 3.3.3

~/Downloads/software/guppy/3.3.3/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    --num_callers 8 \
    --gpu_runners_per_device 64 \
    --chunks_per_runner 256 \
    --device "cuda:0" \
    --recursive \
    -i flongle_fast5_pass \
    -s testrun_fast_3.3.3

ONT Guppy basecalling software version 3.3.3+fa743a6
config file:        /home/miles/Downloads/software/guppy/3.3.3/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /home/miles/Downloads/software/guppy/3.3.3/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass
save path:          testrun_fast_3.3.3
chunk size:         1000
chunks per runner:  256
records per file:   4000
fastq compression:  ON
num basecallers:    8
gpu device:         cuda:0
kernel path:        
runners per device: 64

Found 105 fast5 files to process.
Init time: 726 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 95802 ms, Samples called: 3970269916, samples/s: 4.14425e+07
Finishing up any open output files.
Basecalling completed successfully.

Guppy 3.4.1 (not compressed)

~/Downloads/software/guppy/3.4.1/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    --num_callers 8 \
    --gpu_runners_per_device 64 \
    --chunks_per_runner 256 \
    --device "cuda:0" \
    --recursive \
    -i flongle_fast5_pass \
    -s testrun_fast_3.4.1

ONT Guppy basecalling software version 3.4.1+ad4f8b9
config file:        /home/miles/Downloads/software/guppy/3.4.1/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /home/miles/Downloads/software/guppy/3.4.1/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass
save path:          testrun_fast_3.4.1
chunk size:         1000
chunks per runner:  256CUDA Version: 10.1
fastq compression:  ON
num basecallers:    8
gpu device:         cuda:0
kernel path:        
runners per device: 64

Found 105 fast5 files to process.
Init time: 728 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 90895 ms, Samples called: 3970269916, samples/s: 4.36797e+07
Finishing up any open output files.
Basecalling completed successfully.

Guppy 3.4.1 (compressed)

~/Downloads/software/guppy/3.4.1/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    --num_callers 8 \
    --gpu_runners_per_device 64 \
    --chunks_per_runner 256 \
    --device "cuda:0" \
    --recursive \
    -i flongle_compressed \
    -s testrun_fast_3.4.1

ONT Guppy basecalling software version 3.4.1+ad4f8b9
config file:        /home/miles/Downloads/software/guppy/3.4.1/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /home/miles/Downloads/software/guppy/3.4.1/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_compressed
save path:          testrun_fast_3.4.1
chunk size:         1000
chunks per runner:  256
records per file:   4000
fastq compression:  ON
num basecallers:    8
gpu device:         cuda:0
kernel path:        
runners per device: 64

Found 105 fast5 files to process.
Init time: 725 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 79913 ms, Samples called: 3970269916, samples/s: 4.96824e+07
Finishing up any open output files.
Basecalling completed successfully.

Guppy 3.4.3 (not compressed)

~/Downloads/software/guppy/3.4.3/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    --num_callers 8 \
    --gpu_runners_per_device 64 \
    --chunks_per_runner 256 \
    --device "cuda:0" \
    --recursive \
    -i flongle_fast5_pass \
    -s testrun_fast_3.4.3_uncompressed
first run (it looks like this is anomaly)
ONT Guppy basecalling software version 3.4.3+f4fc735
config file:        /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass
save path:          testrun_fast_3.4.3_uncompressed
chunk size:         1000
chunks per runner:  256
records per file:   4000
fastq compression:  ON
num basecallers:    8
gpu device:         cuda:0
kernel path:
runners per device: 64

Found 105 fast5 files to process.
Init time: 738 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 270953 ms, Samples called: 3970269916, samples/s: 1.4653e+07
Finishing up any open output files.
Basecalling completed successfully.
second run
ONT Guppy basecalling software version 3.4.3+f4fc735
config file:        /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass
save path:          testrun_fast_3.4.3_uncompressed
chunk size:         1000
chunks per runner:  256
records per file:   4000
fastq compression:  ON
num basecallers:    8
gpu device:         cuda:0
kernel path:        
runners per device: 64

Found 105 fast5 files to process.
Init time: 705 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 90674 ms, Samples called: 3970269916, samples/s: 4.37862e+07
Finishing up any open output files.
Basecalling completed successfully.

third run

ONT Guppy basecalling software version 3.4.3+f4fc735 config file: /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg model file: /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/template_r9.4.1_450bps_fast.jsn input path: flongle_fast5_pass save path: testrun_fast_3.4.3_uncompressed3 chunk size: 1000 chunks per runner: 256 records per file: 4000 fastq compression: ON num basecallers: 8 gpu device: cuda:0 kernel path:
runners per device: 64

Found 105 fast5 files to process. Init time: 719 ms

0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----|


Caller time: 94516 ms, Samples called: 3970269916, samples/s: 4.20063e+07 Finishing up any open output files. Basecalling completed successfully.

Guppy 3.4.3 (compressed)

~/Downloads/software/guppy/3.4.3/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    --num_callers 8 \
    --gpu_runners_per_device 64 \
    --chunks_per_runner 256 \
    --device "cuda:0" \
    --recursive \
    -i flongle_compressed \
    -s testrun_fast_3.4.3

ONT Guppy basecalling software version 3.4.3+f4fc735
config file:        /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_compressed
save path:          testrun_fast_3.4.3
chunk size:         1000
chunks per runner:  256
records per file:   4000
fastq compression:  ON
num basecallers:    8
gpu device:         cuda:0
kernel path:        
runners per device: 64

Found 105 fast5 files to process.
Init time: 721 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 82877 ms, Samples called: 3970269916, samples/s: 4.79056e+07
Finishing up any open output files.
Basecalling completed successfully.
@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Dec 12, 2019

Example jtop screen shot from a high accuracy basecalling run on the Jetson Xavier showing the system resources:

hac_basecalling_jtop_screenshot

@lancer-lu

This comment has been minimized.

Copy link

@lancer-lu lancer-lu commented Jan 1, 2020

Sorry to disturb you, I'm a beginner in nanopore, does the character 'guppy version 3.3.3+fa743a6 flowcell FLO-PRO002 --kit SQK-LSK109' means the fast model? Other people do the basecalling, they just give me the the character, so I'm very confused.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Jan 12, 2020

You're not disturbing at all, thanks for the comment.

There are several ways to 'engage' fast basecalling, with the easiest being what I've got documented above, i.e. by setting the config file using the -c flag. The config file to use depends on what version of flow cell and kit you're running (9.4.X, 10.X, ...) and what piece of hardware (i.e. minion, promethion, ...). The testing above was done on a minion using the 9.4.1 flow cell, so for fast calling we use dna_r9.4.1_450bps_fast.cfg.

You can browse all available config files by running guppy_basecaller --print_workflows.

I recommend that if you are doing a lot of local basecalling that you use a GPU, the speed increase is massive and well worth it.

Note: my experience is solely on Linux, so I'm not familiar with how commands translate across to Windows/Mac.

Update: I've added some info from the Guppy guide below which may be useful:


Config files - Variable parameters

In addition, Guppy must know which basecalling configuration to use. This can be provided in one of two ways:

By selecting a flow cell and a kit:

  • Flow cell (-f or --flowcell): the name of the flow cell used for sequencing (e.g. FLO-MIN106).
  • Kit (-k or --kit): the name of the kit used for sequencing (e.g. SQK-LSK109).

[Expert users] By selecting a config file:

  • Config (-c or --config): either the name of the config file to use, or a full path to a config file (see the section below). If the argument is only the name of a config file then it must correspond to one of the standard configuration files provided by the package.

This option is covered in the "Expert users" section below.

Note: If you use the --config argument the --flowcell and --kit arguments will be ignored. When in doubt, select a flow cell and kit and do not use --config.


Config files - Selecting kit and flow cell

These should be clearly labelled on the corresponding boxes. Flow cells almost always start with "FLO" and kits almost always start with "SQK" or "VSK".

To see the supported flow cells and kits, run Guppy with the --print_workflows option:

guppy_basecaller --print_workflows

...which will produce output like this:

Available flowcell + kit combinations are:
flowcell kit barcoding config_name
FLO-PRO001 SQK-LSK109 dna_r9.4.1_450bps_prom
FLO-MIN106 SQK-DCS108 dna_r9.4.1_450bps
FLO-MIN106 SQK-LRK001 dna_r9.4.1_450bps
FLO-MIN106 SQK-LSK108 dna_r9.4.1_450bps
FLO-MIN106 SQK-LSK109 dna_r9.4.1_450bps
FLO-MIN106 SQK-LWP001 dna_r9.4.1_450bps
FLO-MIN106 SQK-PCS108 dna_r9.4.1_450bps
FLO-MIN106 SQK-PSK004 dna_r9.4.1_450bps
FLO-MIN106 SQK-RAD002 dna_r9.4.1_450bps
FLO-MIN106 SQK-RAD003 dna_r9.4.1_450bps
FLO-MIN106 SQK-RAD004 dna_r9.4.1_450bps
FLO-MIN106 SQK-RAS201 dna_r9.4.1_450bps
FLO-MIN106 SQK-RLI001 dna_r9.4.1_450bps
FLO-MIN106 VSK-VBK001 dna_r9.4.1_450bps
FLO-MIN106 VSK-VSK001 dna_r9.4.1_450bps
FLO-MIN106 SQK-RBK001 included dna_r9.4.1_450bps
FLO-MIN106 SQK-RBK004 included dna_r9.4.1_450bps
FLO-MIN106 SQK-RLB001 included dna_r9.4.1_450bps
FLO-MIN106 SQK-LWB001 included dna_r9.4.1_450bps
FLO-MIN106 SQK-PBK004 included dna_r9.4.1_450bps
FLO-MIN106 SQK-RAB201 included dna_r9.4.1_450bps
FLO-MIN106 SQK-RAB204 included dna_r9.4.1_450bps
FLO-MIN106 SQK-RPB004 included dna_r9.4.1_450bps
FLO-MIN106 VSK-VMK001 included dna_r9.4.1_450bps
FLO-MIN106 SQK-RNA001 rna_r9.4.1_70bps
[...]

In the case of kits which come with their own barcodes included, the barcoding column will specify "included". Reads which have been prepared with these kits will be able to be demultiplexed using guppy_barcoder (see below).

The config_name column lists the Guppy config file which corresponds to the flow cell and kit combination. This information is intended for expert users - it is not required for normal operation.

@Tetrakis

This comment has been minimized.

Copy link

@Tetrakis Tetrakis commented Mar 23, 2020

I've been inspired by the fact that you made this work and have made it my quarantine project to set up a Jetson Xaviar for basecalling and, I hope, some basic Read-Until. However, I have rapidly run into some basic setup problems. Do you have any basic notes on how you set this up that you would be willing to share? Thanks.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Mar 25, 2020

I've been inspired by the fact that you made this work and have made it my quarantine project to set up a Jetson Xaviar for basecalling and, I hope, some basic Read-Until. However, I have rapidly run into some basic setup problems. Do you have any basic notes on how you set this up that you would be willing to share? Thanks.

Awesome! Glad it was useful.

There are currently several issues with getting to where you're talking about (and where we all want to be). The first one is that we can't currently run minknow on the xavier, so can't actually sequence on the device. We have minknow installed, but are working through a whole lot of issues. We knew it was going to be tough because the version of minknow is made for the minIT, but we're still hopeful we can hack something together. So until the xavier can actually have a minion connected and generate sequence, read-until and such tools are still a pipe dream unfortunately.

So for now we're trying to figure out exactly how we get minknow working and talking with a minion and xavier. I think I'll need to keep working on ONT for compiled binaries, but we'll do what we can with the minIT versions at this stage.

I can assist with initial setup in terms of getting guppy running if you haven't got to that point yet.

@Tetrakis

This comment has been minimized.

Copy link

@Tetrakis Tetrakis commented Apr 1, 2020

Thanks for the reply! That's frustrating to hear that about minknow. Any idea why? I would have thought that between the MinIT basically being a Jetson TX2 and the ability to install and run minknow on a stand alone Linux machine, it should be possible to make it work.
I'd love some pointers in getting guppy to work. ONT's instructions for deb install have not worked and trying to install it manually hasn't panned out. I thought manual worked, I installed the Ubuntu 16 arm64 version with a similar path structure to what exists in the MinIT (including symbolic links in /usr/bin). Guppy tries to run but I get the following error message after it starts:

[guppy/error] Common::LoadModuleFromFatbin: Loading fatbin file shared.fatbin failed with: CUDA error at /builds/ofan/ont_core_cpp/ont_core/common/cuda_common.cpp:54: CUDA_ERROR_NO_BINARY_FOR_GPU

At this point I'm stumped. Thanks in advance.

@mahalel

This comment has been minimized.

Copy link

@mahalel mahalel commented Apr 16, 2020

Hey thanks for this information.
I'm fairly new at this and trying to run the basecaller on a Tesla P40 card. I've been playing with the various flags trying to optimize it, but I wonder is the objective to fill the vRAM as much as possible?
EG: with the following flags I seem to fill about 16GB of vRAM (out of 22GB):

--num_callers 16 --gpu_runners_per_device 16 --chunks_per_runner 1024 --chunk_size 1000

I fail to understand the formula for optimizing works, do I need to decrease the chunks_per_runner and chunk_size if I increase the other two?

image

image

@PhiloVa

This comment has been minimized.

Copy link

@PhiloVa PhiloVa commented Jun 17, 2020

Hi!
I am having the same issue with @Tetrakis as for the CUDA_ERROR. Have you managed to solve it?

Followed @sirselim previous post although installing the latest version of guppy 3.6.1. on various Jetpack versions (as I couldn't find the arm version of 3.4.1. or 3.4.3). Cuda DeviceQuery and Tensorflow run fine. However, whenever I try to run guppy through the GPU (-x "auto" or -x "cuda:0") I get the error mentioned above. When running through CPU, it's working correctly.

Screenshot from 2020-06-12 16-57-47

I tried to contact ONT, they replied the guppy team is working on it. They know some Xavier AGX users have this problem and others don't but actually can't really explain why... (?)

Could it be a kernel issue?

@neuropathbasel

This comment has been minimized.

Copy link

@neuropathbasel neuropathbasel commented Jul 13, 2020

I have just tried the same on google colaboratory..
(The "!" is there because this version of colaboraty runs python, hence there is no direct shell access. System commands are declared with a leading "!")

!nvidia-smi
The GPU I was assigned is this one, running in an ubuntu 18.04-like environment:

Mon Jul 13 07:05:43 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.05    Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   32C    P8    29W / 149W |      0MiB / 11441MiB |      0%      Default |
|                               |                      |                 ERR! |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

I started guppy as I also do the on a local PC with GPU (GeForce RTX 2070, Driver 440.33.01 CUDA 10.2, Ubuntu 16.04)

!./ont-guppy/bin/guppy_basecall_server --config ./ont-guppy/data/dna_r9.4.1_450bps_hac.cfg --gpu_runners_per_device 16 --num_callers 32 --max_queued_reads 100000 --max_block_size 2048 --log_path ./guppy_server_log --port 5556 --device "cuda:0"

With this result:

ONT Guppy basecall server software version 3.4.3+f4fc735
config file:        ./ont-guppy/data/dna_r9.4.1_450bps_hac.cfg
model file:         /content/ont-guppy/data/template_r9.4.1_450bps_hac.jsn
log path:           ./guppy_server_log
chunk size:         1000
chunks per runner:  512
max queued reads:   100000
num basecallers:    32
num socket threads: 2
max returned events: 2048
gpu device:         cuda:0
kernel path:        
runners per device: 16

[guppy/error] *Common::LoadModuleFromFatbin: Loading fatbin file shared.fatbin failed with: CUDA error at /builds/ofan/ont_core_cpp/ont_core/common/cuda_common.cpp:48: CUDA_ERROR_NO_BINARY_FOR_GPU

I am not sure whether this helps but I would be great to find the reason for this. My guess is that it has to do with the way the driver and CUDA are installed. On the local PC (x86_64) I installed the driver and the CUDA toolchain directly from NVIDIA, nothing from the Ubuntu repos. I have tried before with the ubuntu repos but got all sorts of errors when trying to execute code on the GPU. With the installers from NVIDIA according to their requirements concerning versions and operating systems, it seems to work. Unfortunately, this can't be tried out in colaboratory.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Jul 13, 2020

Hi @neuropathbasel, your issue is the allocated GPU. The architecture of the Tesla K80 (Kepler 2.0) is too old and not compatible with Guppy. From a quick search that card was released 2014 and has CUDA compute capability of 3.7 while Guppy requires 6.0 or higher. So unfortunately its never going to work for basecalling with Guppy. I think this requirement means that only Nvidia cards of the Pascal or newer architecture will be supported.

Edit: the table on this Wikipedia page is quite useful (https://en.m.wikipedia.org/wiki/Nvidia_Tesla).

@neuropathbasel

This comment has been minimized.

Copy link

@neuropathbasel neuropathbasel commented Jul 13, 2020

Thanks a lot for your clarification. I completely forgot about that. I thought the GPUs were somewhat more modern but of course, I should have looked up the particular model on the NVIDIA website. It actually explains another issue I recently had when trying to move some code over to colaboratory for demonstration purposes.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Jul 13, 2020

Hi @Tetrakis and @PhiloVa, sorry for the delay responding.

I've started to get back into this recently as my time has freed up somewhat (and I now have a Xavier NX dev kit and solar panels). I went straight to the ONT forums when I learnt that the ARM binaries being provided are essentially just lifted over from the MinIT (which is now discontinued).

Those binaries are never going to work with Xavier boards due to the compiling being done against different OS and CUDA versions, hence the errors that you and many others are getting. I was provided a compiled binary at the start of the year which was built on Ubuntu 18.04 and CUDA 10.1, but I haven't been able to share this as it was early access. I created a docker image and asked ONT whether this could be shared in the short term with the community. Here was their response to that:

"Just a quick update: we've got separate CUDA 10 aarch64 guppy builds now that work on the Xavier. We have to finish a bit of plumbing to get the Xavier hooked into our CI infrastructure but once that's done we'll start pushing this release out along with the other ones."

I haven't been allowed to release the binary but ONT are meant to be releasing Xavier compatible builds 'soon'. So for now we wait some more, and maybe keep vocal in the forums and on social media (ONT are active on twitter), saying that we are very eager to see these builds released. :)

I'm also pushing for appropriate builds of MinKnow now as well, I figure they have the pipelines in place to make this happen.

Note: if you are part of the ONT community forum, and I assume you both are as you're downloading the Guppy binaries, you can follow the discussion here: https://community.nanoporetech.com/posts/getting-over-20gb-per-run

@PhiloVa

This comment has been minimized.

Copy link

@PhiloVa PhiloVa commented Jul 15, 2020

Thanks for the update @sirselim,

don't worry for the delay, we are all having some turnarounds in 2020. I wasn't expecting to be tickling the Xavier till the end of the year but the lockdown got me to it sooner.

This is good news, the technical team of ONT told me also they would have it ready by the end of the month. Crossing fingers on that.
It's interesting you say about MinKnow, when I asked ONT (not Guppy team).they told me it was not in their plans to release an aarch64 version. But that was before they decided to discontinue the Minit, so obviously the more we try and "gently" push towards it, the more possibilities we have for it to happen.

Have fun with the solared NX and thanks for the update...

@Tetrakis

This comment has been minimized.

Copy link

@Tetrakis Tetrakis commented Oct 4, 2020

Since ONT has released a version of guppy that runs on the Xavier, I've been waiting for some movement on MinKNOW. Then that twitter thread started by @lfaino got me thinking. I knew I'd seen that Xenial build of MinKNOW in the MinIT instructions. I'd tried to install it in the past which did not work, but maybe with fresh eyes... Anyway, as I was searching around I came across this:

https://cse.buffalo.edu/~jzola/smarten/ 

Not only had they apparently got it working on a no-TX2 Jetson platform, they had it working on the Nano. I looking at their instructions and they were what I had tried before and as before it didn't work. However, being a stubborn mule I kept banging on it until eventually I got it all to install. Here are the changes that I did to make it install. Please keep in mind I'm sure there are more elegant ways to do this, but that's not where my level of knowledge is at.

echo "deb http://mirror.oxfordnanoportal.com/apt xenial-stable-minit non-free" > /etc/apt/sources.list.d/nanoporetech.list

For whatever reason even with sudo it would not let me create the nanoporetech.list so I created it with nano after running sudo su
Also there is a problem with the signing so it didn't want to download material from the oxford mirror so I modified the address as:

deb [trusted=yes] http://mirror.oxfordnanoportal.com/apt xenial-stable-minit non-free

For the next parts of the install I had to create a user that was minit:minit (just like the MinIT). Apparently dpkg looks for that during the install of some of the pieces and I have no idea how to change it, so we roll with it.

apt update && apt upgrade

apt install minknow-core-minit-offline ont-bream4-minit ont-configuration-customer-minit ont-minknow-frontend-minit ont-minknow-static-frontend

And it installed. Truly, there was a lot of backward and for problem solving with the error messages, but after all was said and done I'm pretty sure those are the modifications I had to make.

I was super excited, ran off to get the MinION from the lab even though it was after midnight on a Saturday, got back, plugged it in and... it doesn't see it.

My guess is that some of those installs function like daemons but I don't know where they are. I figure that the MinIT must have some script that it runs on startup (maybe in the .bashrc or .profile?) to activate these. The next time I have a chance I will try to get access to a MinIT to investigate and will keep you posted (if you have not already figured it out by then). Anyway, it was progress so I thought I'd share.

Cheers,
-John

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 5, 2020

OK, so in light of this twitter thread on the weekend and various communications (thanks John/@Tetrakis) we're getting closer to running MinKnow of the Jetson Xavier / Xavier NX.

Here is a screen shot showing that the detection and identification of a minion device are working as intended:

image

I ran a quick test of polling the device for data, looks good (see below) so next step is to put a flow cell in and see what happens.

image

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 5, 2020

Update: 5th Oct 2020 - setting up MinKnow?

So this is still very much a work in progress. As can be seen in the above we are getting closer to 'hacking' the minIT Xenial version of MinKnow on to a Nvidia Jetson Xavier system. I wanted to start capturing notes here to both compare with others doing similar and provide a platform for those that would like to start breaking trying the same thing on their Jetson boards.

Again, credit to @lfaino on twitter for posting he's set up working on a Jetson TX2 dev kit here.

First thing is starting with the easy part, Guppy.

Guppy

A pre-compiled version of Guppy can now be downloaded from the ONT community portal in the software section. These are the following steps I took to get it up and running.

  1. download the compressed pre-compiled version of Guppy for CUDA 10 (same as what a Xavier should be running)
  2. extract
  3. mv folder to /opt/ont/ i.e. sudo mv ont-guppy /opt/ont/
  4. create sym links in /usr/bin
  • sudo ln -s /opt/ont/ont-guppy/bin/guppy_basecall_server /usr/bin/guppy_basecall_server
  • sudo ln -s /opt/ont/ont-guppy/bin/guppy_basecaller /usr/bin/guppy_basecaller

You should now be able to run guppy by typing guppy_basecaller in a terminal.
image

minIT Xenial MinKnow

So here goes a very much work in progress (here be dragons!) overview of what we're hacking together. Credit as above to LFaino on twitter and @Tetrakis for their dedication and persistence.

  1. set up a new user minit and add to sudo group. This step might be required to 'mimic' the minit environment for which the packages are compiled.
sudo adduser minit
sudo usermod -aG sudo minit
  1. add ONT ARM Xenial build of minIT software repos to system, with trusted flag:
sudo touch /etc/apt/sources.list.d/nanoporetech.list
# add the below line to this file, I use nano to edit in place
deb [trusted=yes] http://mirror.oxfordnanoportal.com/apt xenial-stable-minit non-free
  1. update and upgrade the system
sudo apt update
sudo apt upgrade
  1. install specific ONT minIT packages:
sudo apt install minknow-core-minit-offline ont-bream4-minit ont-configuration-customer-minit ont-minknow-frontend-minit ont-minknow-static-frontend
  1. at this point you should be able to connect your minion and run a few of the scripts from /opt/ont/minion/bin/ (see above comment for what this looks like if working correctly):
  • config device and check firmware:
/opt/ont/minknow/bin/./device_control_testing
  • get minion ID:
/opt/ont/minknow/bin/./minion_ctl list

NOTE: you may have to set some permissions for the USB device, I need a more permanent way to do this but for now:

  • locate connected minion:
$ lsusb
Bus 002 Device 002: ID 0bda:0489 Realtek Semiconductor Corp. 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 13d3:3549 IMC Networks 
Bus 001 Device 015: ID 2a1d:0001  
Bus 001 Device 005: ID 046d:c07d Logitech, Inc. 
Bus 001 Device 004: ID 046d:c31c Logitech, Inc. Keyboard K120
Bus 001 Device 002: ID 0bda:5489 Realtek Semiconductor Corp. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

This is the entry for the minion Bus 001 Device 015: ID 2a1d:0001

  • change permissions:
sudo chmod 777 /dev/bus/usb/001/015

WARNING: there is a lot wrong with the above, proceed at your own risk!
6. now is when things start to get really iffy. MinKnow should be running as a service, you can check with:

service minknow status

You will probably get info back saying it's not running. This is because it's trying to run as root rather than our minit user. I'm not 100% sure how I hack this process but my service status now looks like it's running as expected (I think).

Here are some of the commands I used to check on services:

ps aux | grep -i 'minknow'
service minknow status
service minknow start
service minknow stop
service minknow restart
  1. launch MinKnow GUI:
/opt/ont/minknow-ui/./MinKnow
  1. ... MinKnow GUI is still not detecting the attached minion...

So that's where I'm leaving this for today. Hopefully a few more eyes on this will help. I'll jot a few thoughts/notes below.


Current Issues

  • MinKnow GUI is not able to detect attached minion, even when minknow.service is running.
  • minion seems to stop being accessible after restarting the minknow service. If the service is stopped (service minknow stop) then the minion is once again accessible using various ONT scripts in /opt/ont/minknow/bin.
  • it has been mentioned by one user that h5py needs to be built from source and placed in the `ont-python' dir. I didn't come across any issues leading me to suspect h5py, but am more than open to suggestions.
  • in the log for minknow I noticed that crashpad_server is not running - is that enough to cause the issues we're seeing here?
@Tetrakis

This comment has been minimized.

Copy link

@Tetrakis Tetrakis commented Oct 5, 2020

I was able to connect to the MinION using MinKNOW running from another machine after testing lots of scripts in /opt/ont/minknow/bin to see what they did. I ran all the ones that you mentioned (except the permissions for the USB). Then started poking around. The one that seemed to have the most significant impact was:
sudo ./mk_manager_svc
Suddenly, the fan (which had been going nuts) slowed down to normal.
I went to a different computer (the MinIT was never designed to have a monitor plugged in and the UI run from it) and launched MinKNOW UI, punched in the IP address and there it was!
I started a run and...
RUN FAILED
No reason given. It's a really old flowcell so that might be it, but I'm pretty sure I'm missing something else. But it's closer still. We'll get there yet, but that's all I can do for today.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 7, 2020

Update 7th Oct 2020

@Tetrakis - thanks for the additional insight. We tried what you suggested but, likely due to our network restrictions at work, weren't able to replicate your process. However, we have had great success running minknow locally on the Xavier!

Frustrated and on a whim, I installed a couple of other ont packages (currently trying to determine which because I didn't write them down! EDIT: may have been ont-minknow-frontend-minion) and then opened the minknow-ui on the Xavier itself.

/opt/ont/minknow-ui/./MinKnow

What do you know?! This time I was greeted with a page displaying the connected minion!
image

I went to run a hardware test and received an error. Looking in the logs (/var/log/minknow/[minionID]) I noticed that it was a permission error trying to write to /data. Time to attack that with a sledge hammer...

sudo chmod -R 777 /data

Re-run hardware test and bingo:

image

Yes! Hardware test completed successfully. Now to deal with the fact that the set directory to write to is root and not the nvme ssd. There is a user config file located at /opt/ont/minknow/conf/user_conf. In it there is a line that will have /data, I edited this to be where I want minknow to write output:

        "output_dirs": {
            "cereal_class_version": 2,
            "base": {
                "value0": "/xavier_ssd/minknow"

After restarting the minknow service (service minknow restart) the directory is correctly assigned.

Next thing was to put a flow cell in. It detected straight away. However, running a flow cell test ended in an error. Checking the logs implicates the h5py python package, which was mentioned in the tweet by @lfaino. So tomorrow I'll be building this library from source and hopefully get up and running!

Here is a screen shot of the desktop showing the Xavier AGX specs and minknow etc:
image

... and here is the current mess that is my desk:
image

I'll tidy up for the proper sequence run, plus there is a small bottle of whisky that has been waiting for this moment! I will also work on replicating this process on the Xavier NX and another Xavier AGX that we have to ensure it's reproducible. Once we're at that point I'll put together a much better set of notes than my ramblings in this gist!

@Tetrakis

This comment has been minimized.

Copy link

@Tetrakis Tetrakis commented Oct 7, 2020

Sweet! I was checking through the logs and also found that h5py error message. Of course, today is one of those days that I absolutely cannot afford to spend time on this -_-. I look forward to seeing your success!

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 7, 2020

Update 8th Oct 2020 - we got there!

The last 'hurdle' was frustrating, but the thing is finally up and running!

image
The above is a picture of a flow cell being checked in MinKnow running on a Jetson Xavier AGX.

So time to jot down my notes for this last issue around h5py that was stopping progress.

compiling h5py from source

At this stage things are looking good, apart from when we try to run a flow cell check or an actual sequencing run. The error reported in the log points to the h5py library and suggests it hasn't been compiled against the correct headers/version of python (?). This is also what @lfaino seems to have been suggesting as well. So here is what finally worked:

Note: to compile 'h5py' from source you'll need to ensure that you have libhdf5-dev and python-dev installed. If you don't want to compile yourself (and I really don't blame you!) please feel free to grab the pre-compiled version that I uploaded from here.

  1. first download and extract the latest release of h5py that supports python2.7
  • this was throwing me for a loop for a while. The current latest version is 2.10, but it turns out support for python 2 has been dropped. Once I downloaded the source from the 2.9.0 release (here) I started having a better time of it.
  • once extracted the base directory contains the build script setup.py, you should be working from within this directory to build the library.
  1. next is to configure the build script and let it know where the hdf5 headers are:
  • something like this will point you in the right direction:
dpkg -L libhdf5-dev
  • on my install the headers are here: /usr/lib/aarch64-linux-gnu/hdf5/serial/
  • Note: ONT are using 'their' own version of Python that we need to use to compile, found here: /opt/ont/minknow/ont-python/bin/python2.7
  • the final configure step looks like this:
/opt/ont/minknow/ont-python/bin/python2.7 setup.py configure --hdf5=/usr/lib/aarch64-linux-gnu/hdf5/serial/
  1. if that completes successfully then it's time to build:
/opt/ont/minknow/ont-python/bin/python2.7 setup.py build -q

Note: the -q flag is just telling the build script to not be so verbose.

  1. if the build is successful then there will be a build directory that has been created. Mine looked like this:
    /home/minit/Downloads/h5py-2.9.0/build/lib.linux-aarch64-2.7/h5py/
  • this is the directory that needs to replace the one located in the ont-python location.
  • first remove the existing ONT h5py library:
sudo rm -rf /opt/ont/minknow/ont-python/lib/python2.7/site-packages/h5py/
  • then copy the new build over to this location:
sudo cp -R /home/minit/Downloads/h5py-2.9.0/build/lib.linux-aarch64-2.7/h5py/ /opt/ont/minknow/ont-python/lib/python2.7/site-packages/

Note: remember to check your directories, they may not be identical to the ones I've described.

  1. [probably optional] restart minknow
service minknow restart
  1. launch the MinKnow-UI
/opt/ont/minknow-ui/./MinKnow
  1. run a flow cell check... and hopefully it works!

I have also created a GitHub repository where I'm going to start collating resources, including a more 'official' guide to getting this set up working. I have uploaded a compressed version of the compiled h5py library so that you don't have to compile from source if you don't want to, find it here.

Watch this space for further updates. :)

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 8, 2020

Congratulations! We (at neuropathology Basel) have been following your progress and a Xavier is already hooked up to our LAN to start sequencing. Thank you for making all this possible!

I just installed guppy and MinKNOW and can confirm that this works like a charm (local time 11pm). Can't wait to be in the lab to physically connect a MinION, hopefully, tomorrow. I am right now on the AGX Xavier from @home via a remote connection; our Xavier is configured as a VNC device. I will post the results a.s.a.p.

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 9, 2020

Hi from Basel again. I have followed the instructions - unfortunately, ONT was "faster", i.e. they changed the version on their repository. There is, as usual, no way to retrieve the older packages. Hence, I am now having these:

MinKNOW core 4.0.5, GUI 3.6.16.

These have python 3.7 included, which is good. It seems that I could skip the part with h5py. Also, guppy works fine (current version).

However, I am running into a permission problem that was not solved by creating a user/group "minit" and also not by reinstalling the ont packages. I have also tried to give permissions to use the minion USB device.

I can follow the problem in the mk_manager_svg log files, e.g.:
cat /var/log/minknow/mk_manager_svc_log-0.txt

2020-10-09 14:07:41.932790    INFO: device_unplugged (device)
    device_path: 2a1d:0001@001:007
2020-10-09 14:07:44.410787    INFO: loading_firmware (usb)
    device_path: 2a1d:0000@001:008
2020-10-09 14:07:44.412136   ERROR: libusb: error [_get_usbfs_fd] libusb couldn't open USB device /dev/bus/usb/001/008: Permission denied
 (usb::libusb)
2020-10-09 14:07:44.413233   ERROR: libusb: error [_get_usbfs_fd] libusb requires write access to USB device nodes.
 (usb::libusb)
2020-10-09 14:07:44.414007 WARNING: failed_to_claim_device (usb)
    device_path: 2a1d:0000@001:008
2020-10-09 14:07:49.414766    INFO: loading_firmware (usb)
    device_path: 2a1d:0000@001:008
2020-10-09 14:07:49.803734   ERROR: libusb: error [_get_usbfs_fd] libusb couldn't open USB device /dev/bus/usb/001/009: Permission denied
 (usb::libusb)
2020-10-09 14:07:49.805153   ERROR: libusb: error [_get_usbfs_fd] libusb requires write access to USB device nodes.
 (usb::libusb)
2020-10-09 14:07:49.806112 WARNING: reattempting_fpga_config_load (device)
    device_path: 2a1d:0001@001:009
    device_id: 

I have checked on a MinION PC, if the user under which we are running MinKNOW was a member of special groups and have added minit on the jetson to those groups as well:

minit@meqneuropat08:/opt/ont/ont-guppy/bin$ groups
minit adm sudo audio dip video plugdev i2c lpadmin gdm sambashare weston-launch crypto trusty gpio

Somehow, also this does not change anything. Am I missing something here? Has MinKNOW changed meanwhile so that we need to adapt the setup again?

@Tetrakis

This comment has been minimized.

Copy link

@Tetrakis Tetrakis commented Oct 9, 2020

Ugh. It looks like I'm in the same boat where there is no python2.7 from ONT, just that 3.7 creature. The Xavier still has python 2.7 loaded and I tried to compile h5py using python 2.7 but it threw up a bunch of errors. grumble grumble Back to the drawing board.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 12, 2020

Update 12th Oct 2020 - sequencing ...

@juhench - thanks for the updates and kind words!

  • It's frustrating isn't it! We'll get there though, I'm currently sequencing covid on the Xavier (without live basecalling as it's not playing ball at the momemnt). Hopefully some of the below is helpful.
  • did you try sudo chmod 777 /dev/bus/usb/001/008 to see if it would sort your write permissions?

@Tetrakis - so you were correct in your thinking, it seems that @lfaino is remote accessing MinKnow on his Tx2 (can see IP address in the screen shots on twitter). In terms of the Python3.7 issue, yes it appears that a 'perfect storm' has lead to me being able to cobble together a local version on one of our Xaviers (more on this below). If you're interested I compiled the hpy5 library and uploaded here - might be worth a shot.

OK, so a short update of where things are at:

  • it turns out that the Xavier AGX I'm using must have had an older version of MinKnow installed, hence the python2.7 issue. But it also seems to be the reason I've been able to 'cobble' together a local running MinKnow rather than having to access remotely from another machine.
  • so far we have not been able to replcate success on our other AGX or NX using the newer software. However, these machines aren't currently allowed on our network so we can't test the remote access option. Fingers crossed this works when we get access sorted.
  • if it's at all useful I have compiled from source h5py using the ONT python3.7 version of the latest MinKnow (here). If that's all it takes for you to get up and running great!
  • with the currently running AGX sequencing works beautifully, however the guppy basecalling server keeps crashing - so no live calling at this stage. Guppy however runs great when run over the newly generated fast5 files. So we'll get there!

I have some more ideas and avenues to try, but I'm going to have to try and fit this work in around other pressing jobs. I'll keep checkingin often though, so hopefully you fine people make some more progress as well. :)

EDIT: I was incorrect in my thinking around remote vs local MinKnow. Both instances are able to run locally (and allow remote access).

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 12, 2020

Thanks for your suggestions! One important lesson I learned is to keep all deb files. I have inquired to get the debs for another older x86_64 release of MinKNOW which we are using in our diagnostic routine. So far, I have not gotten a copy. If you installed it with apt install xyz the deb file is not kept on ubuntu (16.04). I have verified that if you make your installations with apt-get install xyz (note the -get), you preserve them in /var/cache/apt/archives/. Make sure to keep a copy in a safe place if you do not want to go through the "modding" process with every upgrade.

Do you think I should exchange the existing h5py that came with the current MinKNOW for the one you compiled? I will try it today; keeping the original version somewhere else. Thank you very much for compiling, it saves a lot of time and effort!

Regarding the permissions issue: The sudo chmod 777 was the first thing I tried. Nothing changed. I still do not understand the reason for the error message.

Regarding guppy and live basecalling during the run: I recall that ONT recommended using the same (or almost the same) version of guppy compared to the one that came with the MinKNOW install (i.e. not the latest). If there are binaries for arm64 around for other versions, it might be worthwhile trying these out.

We have 2 PCs with RTX2070 GPUs in use and have been able to patch MinKNOW with the respective GPU-aware guppy versions. There is a thread on this on the ONT forum: https://community.nanoporetech.com/posts/enabling-gpu-basecalling-f. This has worked out fairly well. If not already done maybe the configuration needs an adaptation as described here (point 5). Unfortunately, I am not yet there to give that a try. First, I need to get MinKNOW to see the MinION.

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 13, 2020

I have a question - could it be that some of ONT's configure scripts assume a particular user/group ID for the user minit:minit? If so, what is the ID? 1000? I guess I made a fatal mistake then when setting up our Xavier. As usual in our lab, I created a first admin user called "user" which got the ID 1000. Could this user account be the "root of all evil" with my USB port permission problem? What is the user/group ID of minit:minit on the MinIT device? If 1000 is reserved I will probably need to remove the "user:user" and "minit:minit", then re-create "minit:minit" and re-run the configuration of the MinKNOW packages.

Any hint would be highly appreciated.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 14, 2020

@juhench - some interesting comments that I will try and address below.

But before I do that I'll say that I'm very close to getting the newest version of MinKnow fully working on a Xavier NX. I was able to get it to recognise the minion today, and successfully run the hardware check. Guppy seems to be detecting correctly as well. It failed at the flow cell check, but that's due to h5py again - apparently the version I compiled from source isn't quite right. I think tomorrow I should be able to report back with some successful news and a more reproducible set up procedure.

deb files: yes you have an interesting point about caching deb files. I don't really use Ubuntu (Debian Unstable is my daily driver) so I took for granted the fact that they would be cached as usual. But at the end of the day Nanopore software is moving and improving so fast that we do really want to be running the latest versions and not constantly going back to archived deb files. I'm convinced now that the latest version of MinKnow will work and that I was just very 'lucky' with the nature of package combination on my first Xavier AGX install.

I was incorrect about Minknow running only in remote access mode, it does indeed work locally - so that's great, meaning that we can ditch the older python 2.7 version of MinKnow and associated scripts.

On the topic of python versions: (I mentioned earlier) it seems that the 3.7 version that I've compiled isn't working, so tomorrow's job is to get that sorted. Once I can confirm it's working I'll upload that to the GitHub repository. So, I'm sorry if you have downloaded and tried the lib I compiled, and please don't if you haven't. :)

Permission issues: I don't believe that having multiple users on the device would be causing the issues you are seeing. Both the Xavier AGX and Xavier NX I am using have 3 different user accounts each, with 'minit' being the last I added - I was too lazy to start from fresh installs! I mean it wouldn't hurt starting from scratch if you haven't got anything to lose, but I don't think it's the cause of the usb permissions issues. Have you checked which user the MinKnow service is running as? Mine was running as 'root' initially but I changed it to 'minit'. Again, I'm waiting to confirm this step on a completely different install, but on the system that is working the service is running as the 'minit' user. After I get the Xavier NX up and running I plan to start from a clean install on our second NX in an attempt to replicate the process. I will keep a keen eye out for 'issues' around these usb permissions when I'm doing so and report back.

In terms of Guppy: I've downloaded the version from the ONT community forum section, it's currently 4.2.2. This is a precompiled ARM64 version that needs to be uncompressed and then you need to provide the location of the executables to MinKnow (via app_conf). I just checked out the link you provided and this is essentially the process that I'm using, but there are actually a few more paths to modify (I will include these with my notes tomorrow). This set up seems to be working in that everything is detected and running, but the live base calling is causing an error when running a new experiment. At the moment I am putting this down to the well and truly hacked together version of MinKnow on the Xavier AGX, and hope to be able to draw a line under it when running the latest MinKnow. If I do continue to have problems with guppy after then I will try as you suggest and work through a few of the archived versions that I have available.

I hope some of this was useful, and I hope to have good news tomorrow with a much more useful and detailed set of instructions.

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 14, 2020

@sirselim - Thanks a lot. This sounds really good. If possible, I would like to use ubuntu 18.04; it is currently the default OS on our Linux computers. I can confirm that many of the MinKNOW components, in particular mk_manager_svc, are launched upon boot as "minit". There might be differences between Ubuntu and Debian regarding access to USB hardware.

It might be possible to solve the issue by trying to get the thing running as root:

root@meqneuropat08:~# service minknow status
\u25cf minknow.service - MinKNOW Instrument Software for MinIT (daemon)
   Loaded: loaded (/lib/systemd/system/minknow.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2020-10-12 10:07:15 CEST; 2 days ago
  Process: 12124 ExecStartPre=/bin/sleep 15 (code=exited, status=0/SUCCESS)
 Main PID: 12151 (mk_manager_svc)
    Tasks: 121 (limit: 4915)
   CGroup: /system.slice/minknow.service
           \u251c\u250012151 /opt/ont/minknow/bin/mk_manager_svc
           \u251c\u250012164 /opt/ont/minknow/bin/crashpad_handler --database=/data/core-dump-db --metrics-dir=/data/core-dump-db --url=https://submit.backtrace.io/nanoporetech/5487a506174a008b2ccee45c559eac77f8b7
           \u251c\u250012195 /usr/bin/guppy_basecall_server --config dna_r9.4.1_450bps_fast.cfg --port 5555 --log_path /var/log/minknow/guppy --ipc_threads 1 --max_queued_reads 5000 --chunks_per_runner 48 --num_ca
           \u251c\u250012196 /opt/ont/minknow/bin/grpcwebproxy --server_tls_cert_file=/opt/ont/minknow/conf/rpc-certs/localhost.crt --server_tls_key_file=/opt/ont/minknow/conf/rpc-certs/localhost.key --backend_add
           \u251c\u250012197 /opt/ont/minknow/bin/basecall_manager --guppy-address 5555 --conf /opt/ont/minknow/conf 127.0.0.1:9504
           \u251c\u250012204 /opt/ont/minknow/bin/control_server --config /tmp/minknow_34fd6751-89db-45e2-b022-3c1e2db56ca3/conf-MN26891
           \u251c\u250012212 /opt/ont/minknow/bin/crashpad_handler --database=/data/core-dump-db --metrics-dir=/data/core-dump-db --url=https://submit.backtrace.io/nanoporetech/5487a506174a008b2ccee45c559eac77f8b7
           \u251c\u250012223 /opt/ont/minknow/bin/grpcwebproxy --server_tls_cert_file=/opt/ont/minknow/conf/rpc-certs/localhost.crt --server_tls_key_file=/opt/ont/minknow/conf/rpc-certs/localhost.key --backend_add
           \u251c\u250012234 /opt/ont/minknow/bin/crashpad_handler --database=/data/core-dump-db --metrics-dir=/data/core-dump-db --url=https://submit.backtrace.io/nanoporetech/5487a506174a008b2ccee45c559eac77f8b7
           \u251c\u250012269 /opt/ont/minknow/bin/analyser --config /tmp/minknow_34fd6751-89db-45e2-b022-3c1e2db56ca3/conf-MN26891 --shmem-prefix mk12204
           \u251c\u250012270 /opt/ont/minknow/bin/grpcwebproxy --server_tls_cert_file=/opt/ont/minknow/conf/rpc-certs/localhost.crt --server_tls_key_file=/opt/ont/minknow/conf/rpc-certs/localhost.key --backend_add
           \u2514\u250012282 /opt/ont/minknow/bin/crashpad_handler --database=/data/core-dump-db --metrics-dir=/data/core-dump-db --url=https://submit.backtrace.io/nanoporetech/5487a506174a008b2ccee45c559eac77f8b7

Okt 12 10:07:15 meqneuropat08 systemd[1]: Started MinKNOW Instrument Software for MinIT (daemon).
Okt 12 10:10:30 meqneuropat08 minknow[12151]: libusb: error [_get_usbfs_fd] libusb couldn't open USB device /dev/bus/usb/001/020: Permission denied
Okt 12 10:10:30 meqneuropat08 minknow[12151]: libusb: error [_get_usbfs_fd] libusb requires write access to USB device nodes.
Okt 12 10:10:35 meqneuropat08 minknow[12151]: libusb: error [_get_usbfs_fd] libusb couldn't open USB device /dev/bus/usb/001/021: Permission denied
Okt 12 10:10:35 meqneuropat08 minknow[12151]: libusb: error [_get_usbfs_fd] libusb requires write access to USB device nodes.
Okt 12 10:15:09 meqneuropat08 minknow[12151]: libusb: error [linux_netlink_read_message] error receiving message from netlink (105)
Okt 12 10:15:16 meqneuropat08 minknow[12151]: libusb: error [_get_usbfs_fd] libusb couldn't open USB device /dev/bus/usb/001/022: Permission denied
Okt 12 10:15:16 meqneuropat08 minknow[12151]: libusb: error [_get_usbfs_fd] libusb requires write access to USB device nodes.
Okt 12 10:15:22 meqneuropat08 minknow[12151]: libusb: error [_get_usbfs_fd] libusb couldn't open USB device /dev/bus/usb/001/023: Permission denied
Okt 12 10:15:22 meqneuropat08 minknow[12151]: libusb: error [_get_usbfs_fd] libusb requires write access to USB device nodes.

I have found this post: http://seqanswers.com/forums/showthread.php?t=75136

It seems that the problem is indeed bound to mk_manager_svc and its permissions; I will try to modify the minknow service script if I can so that mk_manager_svc will run as root and report back.

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 14, 2020

@sirselim I have been able to get past the USB permission problems. After launching the MinKNOW service, I have killed the mk_manager_SVC from the root shell, verified that it is no longer running, and then relaunched it from the root shell. It continues to do its job, as it seems (see logs below) and recognizes the MinION.

When I launch the MinKNOW GUI as the user "minit", the GUI does not detect any device. Same thing as user "user". When I enter "localhost" or the IP into the field that pops up when clicking "add device manually", I get the error message "MinKNOW service is not running". When I check the mk_manager_svc log files and the ports that are open on the Xavier, this looks quite OK to me (log below).

I saw the error message "ERROR: failed_to_parse_identity_config (data)" in the mk_manager_svc log files and verified the file. The "fingerprint" is empty. PC-based installations do not contain this file (/etc/oxfordnanopore/configs/identity.config).

How did you install Debian on the Xavier? Anyway, it would seem attractive to work with the same system as you do, even though in the end, running ubuntu would be more consistent with our other devices.

user@meqneuropat08:~$ sudo -s
[sudo] password for user: 
root@meqneuropat08:~# killall mk_manager_svc
root@meqneuropat08:~# /opt/ont/minknow/bin/mk_manager_svc 
====================================
user@meqneuropat08:~$ cat /var/log/minknow/
basecall_manager_log-0.txt  mk_manager_svc_log-1.txt
basecall_manager_log-1.txt  mk_manager_svc_log-2.txt
basecall_manager_log-2.txt  mk_manager_svc_log-3.txt
basecall_manager_log-3.txt  mk_manager_svc_log-4.txt
basecall_manager_log-4.txt  mk_manager_svc_log-5.txt
basecall_manager_log-5.txt  mk_manager_svc_log-6.txt
basecall_manager_log-6.txt  mk_manager_svc_log-7.txt
basecall_manager_log-7.txt  mk_manager_svc_log-8.txt
basecall_manager_log-8.txt  mk_manager_svc_log-9.txt
guppy/                      MN26891/
mk_manager_svc_log-0.txt    MN35026/
user@meqneuropat08:~$ cat /var/log/minknow/mk_manager_svc_log-0.txt 
2020-10-14 18:15:12.026348    INFO: guppy_server_version (util)
    version: 4.2.2+effbaf8
2020-10-14 18:15:12.030287   ERROR: failed_to_parse_identity_config (data)
    path: /etc/oxfordnanopore/configs/identity.config
2020-10-14 18:15:12.032944    INFO: mk_manager_starting (mk_manager)
    system: User-provided system running on ubuntu 18.04
    version: 4.0.5
2020-10-14 18:15:12.051486    INFO: guppy_server_version (util)
    version: 4.2.2+effbaf8
2020-10-14 18:15:12.054135    INFO: mk_manager_initialised (mk_manager)
    pid: 9573
2020-10-14 18:15:12.057867    INFO: common_process_started (host)
    name: guppy
2020-10-14 18:15:12.061955    INFO: common_process_started (host)
    name: grpcwebproxy
2020-10-14 18:15:12.066954    INFO: common_process_started (host)
    name: basecall_manager
2020-10-14 18:15:12.196167    INFO: rpc_delegate_is_listening (host)
    executable: basecall_manager
    security: none
    port: 9504
2020-10-14 18:15:12.200009    INFO: common_process_started (host)
    name: basecall_manager:grpcwebproxy
2020-10-14 18:15:12.225621    INFO: rpc_delegate_proxy_is_listening (host)
    executable: basecall_manager
    tls_port: 9505
    insecure_port: 9506
2020-10-14 18:15:29.348442    INFO: loading_firmware (usb)
    device_path: 2a1d:0000@001:002
2020-10-14 18:15:30.718371    INFO: instance_started (host)
    grpc_port: 8000
    instance: MN35026
    grpc_secure_port: 8001
    grpcweb_insecure_port: 8003
    grpcweb_tls_port: 8002
    jsonrpc_port: 0
    shmem_prefix: mk9683
user@meqneuropat08:~$ 

user@meqneuropat08:~$ sudo nmap localhost
Starting Nmap 7.60 ( https://nmap.org ) at 2020-10-14 18:25 CEST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000024s latency).
Not shown: 988 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
25/tcp   open  smtp
111/tcp  open  rpcbind
631/tcp  open  ipp
5555/tcp open  freeciv
5901/tcp open  vnc-1
6001/tcp open  X11:1
8000/tcp open  http-alt
8001/tcp open  vcom-tunnel
8002/tcp open  teradataordbms
9502/tcp open  unknown
9503/tcp open  unknown

Nmap done: 1 IP address (1 host up) scanned in 1.81 seconds
user@meqneuropat08:~$ 


minit@meqneuropat08:~$ cat /etc/oxfordnanopore/configs/identity.config
Serial=meqneuropat08
Fingerprint=

small addition - device check also fails, no matter if executed as root or as minit:

minit@meqneuropat08:/$ /opt/ont/minknow/bin/device_control_testing 
Looking for device...
2020-10-14 18:48:39.157365 WARNING: reattempting_fpga_config_load (device)
    device_path: 2a1d:0001@001:003
    device_id: 
2020-10-14 18:48:39.408690 WARNING: reattempting_fpga_config_load (device)
    device_path: 2a1d:0001@001:003
    device_id: 
2020-10-14 18:48:39.909995 WARNING: reattempting_fpga_config_load (device)
    device_path: 2a1d:0001@001:003
    device_id: 
2020-10-14 18:48:40.911587 WARNING: reattempting_fpga_config_load (device)
    device_path: 2a1d:0001@001:003
    device_id: 
2020-10-14 18:48:42.913092 WARNING: reattempting_fpga_config_load (device)
    device_path: 2a1d:0001@001:003
    device_id: 

It seems that I am unable to replicate with Ubuntu 18.04 what you were able to do under Debian.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 14, 2020

@juhench - sorry for the confusion, I meant that I use Debian on my personal machines, the Jetson boards are all running Ubuntu 18.04.

I'm in the process of compiling h5py I'll report back shortly.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 15, 2020

Update 15th Oct 2020

We finally have success! Here is a flow cell check running on a Nvidia Jetson Xavier NX:

image

The big 'hold up' was compiling h5py, it was not the simple procedure that it was on python2.7, but that's a story for another day. In the end I was able to create a virtual python environment using python3.7, build h5py from source, move it over to the ONT directory and get up and running. I have uploaded the compiled version of h5py to the GitHub repo here. Here is the code to build from source in a virtual environment:


WARNING: this is very specific to the versions of python that are packaegd by ONT and that are available on the Jetson Xavier via JetPack 4.4 (Ubuntu 18.04). I was unable to get h5py to compile using the python3.7 version packaged by ONT (3.7.4), so I gambled on the version in the Ubuntu 18.04 repository in JetPack 4.4 (3.7.5). I think this small difference is OK, but it's something to be wary of. So obviously it goes without saying "here be dragons!" Also, unless you really know what you are doing, DO NOT update the default python3 version on the Xavier. As of the time of writing the python3 version set to default was 3.6.X, setting to 3.7.X has the potential for many unwanted and undesired results - this is why I went the virtual environment route. You have been fore warned.


  1. install python3.7 and virtual environment, and then pip for 3.7:
sudo apt install python3.7-dev python3.7-venv
python3.7 -m pip install pip
  1. set up virtual env and build h5py:
# create virtual env
python3.7 -m venv envs/h5py
# start venv
source envs/h5py/bin/activate
# grab set up packages
pip3.7 install cython
pip3.7 install wheel
# build h5py and depends
pip3.7 install --no-binary=h5py h5py
# leave venv
deactivate
  1. create a backup of the old ont h5py (this is effectively useless, but just in case):
sudo mv /opt/ont/minknow/ont-python/lib/python3.7/site-packages/h5py /opt/ont/minknow/ont-python/lib/python3.7/site-packages/h5py.bkup
  1. copy the newly compiled h5py across to the ont directory:
sudo cp -r ~/envs/h5py/lib/python3.7/site-packages/h5py /opt/ont/minknow/ont-python/lib/python3.7/site-packages/
  1. restart minknow services:
service minknow restart
  1. launch MinKnow UI and proceed to flow cell check.

All going well MinKnow should now successfully start and complete the flow cell checking and further sequencing experiments.

We're currently loading a test DNA prep on the device so I'll keep updating. Fingers crossed the live base calling works.
Update: live base calling broke, but I think it's a simple fix (a flag or two in app_conf). I'm away now until Monday but will report back then. For now we're sequencing HMW DNA without base calling on the Xavier NX.

The next step is to containerise this - it's going to be horrible but it means we can quickly test and release an image that others can deploy without jumping through all the hoops. In the long term we plan to streamline the whole process.

Edit: a few quick pics as I was leaving the office. This flow cell is a trooper, it's on it's forth use and still has enough pores available for us to test our HMW DNA prep. Excited by the fact that we had numerous reads over 100kb (not pictured below as I didn't take another pic) before I left for the day. Fingers crossed it looks good in the morning.

image

image

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 15, 2020

Congratulations! Will try out hopefully tomorrow!

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 16, 2020

I have tried your instructions. Everything complied as prepared - thanks a lot! Unfortunately, my problem with USB access persists. This is probably linked to my initial install of the system.

When I kill mk_manager_svc by killall mk_manager_svc , wait until it has quit and double-check with ps -Ae | grep mk_manager_svc and then relaunch the program as root with sudo /opt/ont/minknow/bin/mk_manager_svc & I am getting the following information in the respective log file

user@meqneuropat08:~$ cat /var/log/minknow/mk_manager_svc_log-0.txt
2020-10-16 09:22:12.257707    INFO: guppy_server_version (util)
    version: 4.2.2+effbaf8
2020-10-16 09:22:12.260659   ERROR: failed_to_parse_identity_config (data)
    path: /etc/oxfordnanopore/configs/identity.config
2020-10-16 09:22:12.262990    INFO: mk_manager_starting (mk_manager)
    system: User-provided system running on ubuntu 18.04
    version: 4.0.5
2020-10-16 09:22:12.285074    INFO: guppy_server_version (util)
    version: 4.2.2+effbaf8
2020-10-16 09:22:12.287957    INFO: mk_manager_initialised (mk_manager)
    pid: 15910
2020-10-16 09:22:12.291173    INFO: common_process_started (host)
    name: guppy
2020-10-16 09:22:12.294659    INFO: common_process_started (host)
    name: grpcwebproxy
2020-10-16 09:22:12.298540    INFO: common_process_started (host)
    name: basecall_manager
2020-10-16 09:22:12.397729    INFO: rpc_delegate_is_listening (host)
    executable: basecall_manager
    security: none
    port: 9504
2020-10-16 09:22:12.401822    INFO: common_process_started (host)
    name: basecall_manager:grpcwebproxy
2020-10-16 09:22:12.422651    INFO: rpc_delegate_proxy_is_listening (host)
    executable: basecall_manager
    tls_port: 9505
    insecure_port: 9506
2020-10-16 09:22:34.865614    INFO: loading_firmware (usb)
    device_path: 2a1d:0000@001:006
2020-10-16 09:22:36.180674    INFO: instance_started (host)
    grpc_port: 8000
    instance: MN35026
    grpc_secure_port: 8001
    grpcweb_insecure_port: 8003
    grpcweb_tls_port: 8002
    jsonrpc_port: 0
    shmem_prefix: mk16018

The permission errors are gone. Still, when I launch the MinKNOW GUI locally, I can't see the MinION and if I enter "localhost or even the respective IP of the device, I get the error message that MinKNOW service is not running. I then realized from your screenshots that you are probably running another (current) version of MinKNOW on your PC (not on the Xavier that I am running through a VNC session) and just tried that with a laptop hooked up to the same subnet in which the Xavier is connected. I started MinKNOW (current version with the "new" GUI" just as on your screen and entered the IP of the Xavier. That did the job!

Then I rebooted the Xavier and let it run "as normal", i.e. with letting it have the USB permission errors. In fact, it does recognize the MinION (through MinKNOW running on a PC, not the Xavier!) and it also lets me perform a hardware check. A subsequent flow cell check was performed correctly.

I started a run - getting a "run error" - basecalling was turned on - it seems I am replicating this error, too. Might need to adapt the basecalling options inside MinKNOW as the version of the basecaller that would come with this (old MinIT MinKNOW) does not match the more recent version of guppy we installed manually (4.2.2+effbaf8). I will have a look at that - similar issues exist when installing on the PC with guppy GPU versions. When launching the thing to sequence, the system complains about too little space after the mux scan - that can be fixed for the moment by attaching a SATA drive or probably by mounting a network drive. I can confirm that I am also getting reads (during the MUX scan)!

Thanks again, Miles, for this excellent work. It might be worthwhile to try and create an installer/configuration script that starts from the basic installation of the Xavier, even though an image would probably easier to test in the end.

I will now try to get our data analysis software to work on the Xavier - could be possible! I will also try older versions of guppy.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 16, 2020

Fantastic work @juhench! I'm glad you're up and running sequencing, and I'm very glad the python compilation instructions worked - that was a massive headache!

A couple of comments:

I then realized from your screenshots that you are probably running another (current) version of MinKNOW on your PC (not on the Xavier that I am running through a VNC session) and just tried that with a laptop hooked up to the same subnet in which the Xavier is connected. I started MinKNOW (current version with the "new" GUI" just as on your screen and entered the IP of the Xavier. That did the job!

I'm glad that process worked for you, but I am running MinKnow locally on the Xavier itself and not connected from another PC. When I'm back at work next week I will start documenting the setup process from start to finish.

The version of MinKnow running is the very latest version, this is compatible with Guppy 4.2.2. I'm very close to having the base calling working and think I have the fix sorted. Again I just need to return from my trip away and have the device in front of me (I was going to bring it but don't think my wife would have appreciated it!).

It might be worthwhile to try and create an installer/configuration script that starts from the basic installation of the Xavier, even though an image would probably easier to test in the end.

A nice set of installation scripts is the ultimate goal. The image is solely for internal testing, making sure we have it running on the multiple Xavier AGX and NX units we have. I'm getting access to a Xavier NX with a fresh image on it which I will use to start building the system up from scratch, making sure I can follow my notes and steps. I can then improve the process and ensure a reproducible method for others to follow.

You mention a SATA hard drive. Don't you have an nvme solid state drive in the Xavier itself? If not I highly recommend you put a very high speed >=1Tb ssd in there. You'll get much better performance for base calling etc which benefits from good I/O. You could then transfer the fast5 and fastq files off to the connected SATA drive. Just a suggestion, this is what I'm doing. You can adjust the location of Minknow output to the ssd by modifying the path found in the user_conf file.

I'm really excited that more and more of us are getting up and running, it's been a great community effort!

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 16, 2020

Dear @sirselim - I can now confirm that also our Jetson is running a local instance of MinKNOW (ont-kingfisher-ui-minit_4.0.21-1~xenial_all) which is - thanks to your h5py compiled with the python 3.7, installed from the ubuntu jetpack default repo - recognizing and running the MinION with basecalling turned off.

With regard to the minit user: Our Xavier in Basel has a user called "user" and a user called "minit" both of which are members of all groups that exist on the device. I did that to make sure that the minit user has all required permissions. It might look a bit sloppy, but it potentially saves a lot of time.

That said, I have determined the list of packages that currently needs to be installed and I can only advise to install them with apt-get install to get copies of the respective deb files:

minknow-core-minit-offline_4.0.5-xenial_arm64.deb
ont-bream4-minit_6.0.10-1~xenial_all.deb
ont-configuration-customer-minit_4.0.16-1~xenial_all.deb
ont-kingfisher-ui-minit_4.0.21-1~xenial_all.deb
ont-minknow-static-frontend_3.6.17-1~xenial-stable-minit_all.deb
ont-remote-support_3.0.5-0~xenial-minit_all.deb
ont-system-identification_3.5.1-0~xenial-minit_all.deb

With these installed, one can produce a working local system.

After installation - they can also be installed from the directory where one keeps these debs with dpkg -i *.deb without the repo from ONT - h5py needs to be compiled and to be moved as @sirselim described a few posts above (Update 15th Oct 2020).

Then, either local storage or network storage (for testing purposes needs to be added - my leftover 8GB are insufficient according to MinKNOW and the device complains about low storage, them immediately stopped the run.

It seems to be essential to stick to /data and mount the additional local or network storage into this directory, or as @sirselim suggests, build in a larger NVMe module.

For testing purposes, I can confirm that one can use sshfs:

sudo apt-get install sshfs
sudo service minknow stop
sudo modprobe fuse
sudo sshfs -o reconnect -o allow_other REMOTEUSER@SSHFILESERVER:/path/to/target/directory/on/fileserver /data
sudo service minknow start

I then started the MinKNOW UI as user "user" while I can confirm that most MinKNOW processes within the service are launched as "minit".

Within the MinKNOW GUI, I turned basecalling off and get the result below (from an old library that has been in a cell for weeks, we had actually intended to discard this cell). It comes in handy for testing now. This is human genomic tumor DNA prepared with RAD-RBK004.

Screenshot_2020-10-16_18-53-04

Dear @sirselim - THANKS a lot for all your effort. It was really worth it and we have a good base to continue.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 16, 2020

Nice work, congratulations!

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 19, 2020

Dear @sirselim, I have a general question about ont-python. Since we're always running after ONTs changes to the built-in python, do you think it could be possible to make this the main python implementation? While it might seem to be a lot of a hassle, it could be beneficial to create a working environment using this implementation, maybe in the form of a copy of it in order not to alter basic MinKNOW functionality?

I have not played around with this yet, but trying to get readfish from the Loose lab to work seems to become tricky if there are python differences between the readfish virtual environment (e.g., in Anaconda) and those parts that run from inside MinKNOW.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 19, 2020

Hi @juhench, it's something that I've thought about. We're currently talking about how we can strip back the default Jetson Jetpack install to something a little more minimal, both in terms of size and amount of resources being used. This would potentially mean changing the default python version installed would be a little easier as there would hopefully be less dependencies.

The version of python I ended up using isn't the exact version used in the arm64 repo version of minknow (ont-python version) currently available, they're both 3.7 but from memory the default 3.7 in the 18.04 repos is few point versions higher. Seems to work but I do caveat that.

I'm not familiar with read fish yet, but obviously not having conda on arm makes things trickier. Have you tried plain old python virtual environments to see if you can get it to play nicely with ont-python builds?

A question of my own: in the software section of the ONT community forum there is a post that instructs how to download archived versions of guppy for Linux (amd64), Windows and Mac (https://community.nanoporetech.com/posts/how-to-get-older-version-s#comment_25267). Do we know if we can get the same archived versions of guppy for arm64? I've only got the latest available, 4.2.2 and an older arm binary (3.4.x). I'm wanting to test an idea I have about live base calling.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 20, 2020

Some notes to myself to continue documenting live basecalling issues, various logs below.

Guppy log:

minit@xavnx:/var/log/minknow$ cat guppy/guppy_basecall_server_log-2020-10-19_14-30-38.log
2020-10-20 10:12:23.684218 [guppy/message] ONT Guppy basecall server software version 4.2.2+effbaf8, client-server API version 3.2.0
config file:         /opt/ont/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:          /opt/ont/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
log path:            /var/log/minknow/guppy
chunk size:          2000
chunks per runner:   48
max queued reads:    5000
num basecallers:     3
num socket threads:  2
max returned events: 50000
gpu device:          cuda:all
kernel path:         
runners per device:  2

2020-10-20 10:12:23.684903 [guppy/info] crashpad_handler not supported on this platform.
2020-10-20 10:12:23.686055 [guppy/info] Listening on port 5555.
2020-10-20 10:12:24.242732 [guppy/info] CUDA device 0 (compute 7.2) initialised, memory limit 8147730432B (1037365248B free)

2020-10-20 10:12:27.251491 [guppy/message] Starting server on port: 5555
minit@xavnx:/var/log/minknow$ cat mk_manager_svc_log-0.txt
2020-10-20 10:12:23.611346    INFO: guppy_server_version (util)
    version: 4.2.2+effbaf8
2020-10-20 10:12:23.619825    INFO: mk_manager_starting (mk_manager)
    system: MinION Mk1C (MIN-101C) XAVNX running on ubuntu 18.04 with 1 minion_pci flow cell positions
    version: 4.0.5
2020-10-20 10:12:23.646829    INFO: guppy_server_version (util)
    version: 4.2.2+effbaf8
2020-10-20 10:12:23.650486    INFO: mk_manager_initialised (mk_manager)
    pid: 8768
2020-10-20 10:12:23.654984    INFO: common_process_started (host)
    name: guppy
2020-10-20 10:12:23.659992    INFO: common_process_started (host)
    name: grpcwebproxy
2020-10-20 10:12:23.665050    INFO: common_process_started (host)
    name: basecall_manager
2020-10-20 10:12:23.797877    INFO: rpc_delegate_is_listening (host)
    port: 9504
    executable: basecall_manager
    security: none
2020-10-20 10:12:23.803387    INFO: common_process_started (host)
    name: basecall_manager:grpcwebproxy
2020-10-20 10:12:23.845283    INFO: rpc_delegate_proxy_is_listening (host)
    insecure_port: 9506
    tls_port: 9505
    executable: basecall_manager
2020-10-20 10:12:23.944182    INFO: instance_started (host)
    grpcweb_insecure_port: 8003
    instance: MN34702
    grpc_port: 8000
    grpc_secure_port: 8001
    grpcweb_tls_port: 8002
    jsonrpc_port: 0
    shmem_prefix: mk8822
minit@xavnx:/var/log/minknow$ cat basecall_manager_log-0.txt
2020-10-20 10:12:23.740521    INFO: starting_up (basecall_manager)
    version: 4.0.5
    hostname: xavnx
2020-10-20 10:12:23.769106    INFO: guppy_server_version (util)
    version: 4.2.2+effbaf8
minit@xavnx:/var/log/minknow$ cat MN34702/analyser_log-0.txt
2020-10-20 10:12:24.037723    INFO: starting_up (control)
    version: 4.0.5
    hostname: xavnx
2020-10-20 10:12:24.057388    INFO: guppy_server_version (util)
    version: 4.2.2+effbaf8
2020-10-20 10:12:24.070307 WARNING: partially_raised_file_limits (analyser)
    target: 4096
    actual: 1024
minit@xavnx:/var/log/minknow$ cat MN34702/control_server_log-0.txt
2020-10-20 10:12:23.811047    INFO: starting_up (control)
    version: 4.0.5
    hostname: xavnx
2020-10-20 10:12:23.842094    INFO: guppy_server_version (util)
    version: 4.2.2+effbaf8
2020-10-20 10:12:23.891082    INFO: machine_description (control_server)
    cpu_logical_core_count: 6
    memory_physical_bytes: 8147730432
    cpu_has_avx: false
    cpu_vendor: 
    cpu_brand: 
    cpu_physical_core_count: 6
    cpu_has_sse42: false
2020-10-20 10:12:23.891438    INFO: sending_telemetry_message (ping)
    data: {"machine":{"arch":"x64","cpu":{"data":"","ext_data":"","logical_cores":6,"physical_cores":6},"memory":{"phys":8147730432},"os":"ubuntu 18.04"}}
2020-10-20 10:12:23.893118    INFO: jsonrpc_interface_disabled (control_server)
2020-10-20 10:12:23.959325    INFO: minion_disconnected (engine)
2020-10-20 10:12:24.094454    INFO: active_device_set (mgmt)
    minion_id: MN34702
2020-10-20 10:12:24.109498    INFO: device_without_flowcell_discovered (engine)
    device_id: 
2020-10-20 10:12:24.129854    INFO: asic_id_changed (mgmt)
    old_asic_id: 0
    new_asic_id: 203450368
2020-10-20 10:12:25.484676    INFO: read_eeprom_id (mgmt)
    eeprom_id: 1974805
2020-10-20 10:12:25.510814    INFO: flowcell_discovered (engine)
    asic_id: 203450368
    asic_id_eeprom: 1974805
2020-10-20 10:12:27.490888    INFO: asic_id_changed (mgmt)
    old_asic_id: 203450368
    new_asic_id: 203448384
@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 20, 2020

I have looked into the deb files for the cuda9 minit guppy that are currently present in the ONT deb repo:
apt-get download ont-guppy-for-minit ont-guppy-for-minknow
I verified that it is impossible to run them on the Xavier with CUDA10: definitively impossible. Then I check ed the original install location on the MinIT. It is /opt/ont/guppy . I have then copied the guppy 4.2.2 into this location (entire directory structure).

I then adapted the MinKNOW configuration:

sudo /opt/ont/minknow/bin/config_editor --conf application --filename /opt/ont/minknow/conf/app_conf \
--set guppy.server_executable="/opt/ont/guppy/bin/guppy_basecall_server" \
--set guppy.client_executable="/opt/ont/guppy/bin/guppy_basecaller" \
--set guppy.gpu_calling=1 \
--set guppy.num_threads=3 \
--set guppy.ipc_threads=2

Same thing as before: If one turns on basecalling in MinKNOW, sequencing stops immediately before the pore check. i.e. still no luck and so far not even any helpful error messages.

user@meqneuropat08:~/scripts$ cat /var/log/minknow/guppy/guppy_basecall_server_log-2020-10-21_01-00-18.log 
2020-10-21 01:00:18.323910 [guppy/message] ONT Guppy basecall server software version 4.2.2+effbaf8, client-server API version 3.2.0
config file:         /opt/ont/guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:          /opt/ont/guppy/data/template_r9.4.1_450bps_fast.jsn
log path:            /var/log/minknow/guppy
chunk size:          2000
chunks per runner:   48
max queued reads:    5000
num basecallers:     3
num socket threads:  2
max returned events: 50000
gpu device:          cuda:all
kernel path:         
runners per device:  2

2020-10-21 01:00:18.324621 [guppy/info] crashpad_handler not supported on this platform.
2020-10-21 01:00:18.325847 [guppy/info] Listening on port 5555.
2020-10-21 01:00:18.738053 [guppy/info] CUDA device 0 (compute 7.2) initialised, memory limit 33477578752B (19724750848B free)

2020-10-21 01:00:20.378887 [guppy/message] Starting server on port: 5555
user@meqneuropat08:~/scripts$ cat /var/log/minknow/*-0.txt
2020-10-21 01:00:18.377440    INFO: starting_up (basecall_manager)
    version: 4.0.5
    hostname: meqneuropat08
2020-10-21 01:00:18.436899    INFO: guppy_server_version (util)
    version: 4.2.2+effbaf8
2020-10-21 01:00:18.265231    INFO: guppy_server_version (util)
    version: 4.2.2+effbaf8
2020-10-21 01:00:18.270190   ERROR: failed_to_parse_identity_config (data)
    path: /etc/oxfordnanopore/configs/identity.config
2020-10-21 01:00:18.274085    INFO: mk_manager_starting (mk_manager)
    system: User-provided system running on ubuntu 18.04
    version: 4.0.5
2020-10-21 01:00:18.295843    INFO: guppy_server_version (util)
    version: 4.2.2+effbaf8
2020-10-21 01:00:18.299283    INFO: mk_manager_initialised (mk_manager)
    pid: 8264
2020-10-21 01:00:18.302714    INFO: common_process_started (host)
    name: guppy
2020-10-21 01:00:18.307075    INFO: common_process_started (host)
    name: grpcwebproxy
2020-10-21 01:00:18.311545    INFO: common_process_started (host)
    name: basecall_manager
2020-10-21 01:00:18.480600    INFO: rpc_delegate_is_listening (host)
    executable: basecall_manager
    security: none
    port: 9504
2020-10-21 01:00:18.491864    INFO: common_process_started (host)
    name: basecall_manager:grpcwebproxy
2020-10-21 01:00:18.531368    INFO: rpc_delegate_proxy_is_listening (host)
    executable: basecall_manager
    tls_port: 9505
    insecure_port: 9506
2020-10-21 01:00:18.643079    INFO: instance_started (host)
    grpc_secure_port: 8001
    instance: MN35026
    grpc_port: 8000
    grpcweb_insecure_port: 8003
    grpcweb_tls_port: 8002
    jsonrpc_port: 0
    shmem_prefix: mk8619

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 21, 2020

@juhench - yes, I made sure that the directory structure for the pre-compiled version of guppy 4.2.2 was in the 'correct' location (apologies, I may not have documented that step fully). I've also played around with the exec paths in the app_conf file, i.e. changing them to a hard location and also sym linking them in /usr/local/bin. Either way seems to be fine for guppy to be recognised by minknow, but both still lead towards live base calling causing an error on run start up.

I just noticed that I haven't included one of the logs that states that there is an error contacting the guppy base calling server, I will add that information tomorrow. In light of this it would be interesting to try and send a guppy run from another machine over to a network attached Xavier running minknow and guppy to see if remote GPU base calling works as expected. If this comes back as server not contactable then I think we're drilling in closer to the issue. If it works then we have confirmation that the base calling server is working as intended and can cross that off the list of things to troubleshoot.

Anyway, just a few late night thoughts before heading to bed. I have a little bit of time tomorrow to do some more digging so let's see how that goes.

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 21, 2020

I wanted to test whether the guppy_basecall_server 4.2.2 can be accessed from a guppy_basecaller in client mode, either locally, or from a remote PC on the same network. I then noticed that the "--port SERVER:5555" option has vanished in 4.2.2 which is what others also reported on the ONT forum. I have posted a request and asked how to specify the former "--port" argument in 4.2.2; if not possible, it would explain why there is a problem with live basecalling in 4.2.2.

In 4.0.14 (x86_64), the port option is still available, and they remain downloadable:
ont-guppy_4.0.11_linux64.tar.gz
ont-guppy_4.0.11_linuxaarch64.tar.gz
ont-guppy_4.0.14_linux64.tar.gz
ont-guppy_4.0.14_linuxaarch64_cuda9.tar.gz
ont-guppy_4.0.14_linuxaarch64_cuda10.tar.gz

Will now try with 4.0.14.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 21, 2020

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 21, 2020

Success! Live basecalling on the Xavier is working with guppy version 4.0.14!

Not only does it allow live basecalling, but also allows acting as a remote basecalling server to PCs and alike that do not have a GPU etc. That said, the Xavier also resolves the need for an eGPU-like environment when a laptop computer should be used as the primary system.

I have posted the respective request about 4.2.2 on the ONT forum and with customer support - since there is a basecall_server in 4.2.2,
there should be a way to specify client and port. This, unfortunately, breaks scripts and alike which count on the --port option.

Here are the detailed instructions:

sudo service minknow stop
Download the cuda10 version ont-guppy_4.0.14_linuxaarch64_cuda10.tar.gz (the URL can be completed from the downloads section when logged in at ONT)
Unzip the file on the Xavier
move the whole thing to /opt/ont/guppy, e.g.
sudo mv ~/Downloads/ont-guppy /opt/ont/guppy
Make sure your configuration file looks correct by executing

sudo /opt/ont/minknow/bin/config_editor --conf application --filename /opt/ont/minknow/conf/app_conf \
--set guppy.server_executable="/opt/ont/guppy/bin/guppy_basecall_server" \
--set guppy.client_executable="/opt/ont/guppy/bin/guppy_basecaller" \
--set guppy.gpu_calling=1 \
--set guppy.num_threads=3 \
--set guppy.ipc_threads=2

sudo service minknow start

Then, you should be able to use MinKNOW with live basecalling as on the PC. Dear @sirselim, thanks a lot for making all of this possible!

That way, it should also be able to get readfish (https://pypi.org/project/readfish/0.0.5a2/) which is what I will try out next.

Screenshots below from working live basecalling.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 21, 2020

Excellent work @juhench! I wish it wasn't 3am here now as I can't wait to try this. πŸ™‚

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 21, 2020

Screenshot_2020-10-21_15-32-40
Screenshot_2020-10-21_15-35-02
Screenshot_2020-10-21_15-39-27
Screenshot_2020-10-21_15-42-14

The result (subfolders with barcodes for RBK004) on the hard drive is also as usual, and remote basecalling from other computers works pretty fast on the Xavier (with quite some heating up).

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 21, 2020

I bet its hot, you haven't got the fan going! I've posted scripts to do that, let me dig them up...

Edit: here you go.

# enter root
sudo su
# turn to 100%
echo 255 | tee /sys/devices/pwm-fan/target_pwm
# turn off
echo 0 | tee /sys/devices/pwm-fan/target_pwm
# about 30%
echo 77 | tee /sys/devices/pwm-fan/target_pwm
@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 21, 2020

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 21, 2020

Fan commands just in the above comment. I recommend setting it manually while doing basecalling. I'm currently writing a more automated function to do this but the above work just fine for now.

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 21, 2020

Perfect, thanks a lot! I also noted that some cpu cores are not working by default:

This can be changed with the chcpu command, e.g. in the form a shell script:

#!/bin/bash

# taken from https://forums.developer.nvidia.com/t/why-are-4-cores-in-jetson-agx-xavier-offline-is-there-a-way-to-enable-all-8/68273/2

sudo chcpu -e 0
sudo chcpu -e 1
sudo chcpu -e 2
sudo chcpu -e 3
sudo chcpu -e 4
sudo chcpu -e 5
sudo chcpu -e 6
sudo chcpu -e 7

And the threads still need some tweaking; I will probably look at the older posts to adapt for 4.0.14

sudo /opt/ont/minknow/bin/config_editor --conf application --filename /opt/ont/minknow/conf/app_conf \
--set guppy.server_executable="/opt/ont/guppy/bin/guppy_basecall_server" \
--set guppy.client_executable="/opt/ont/guppy/bin/guppy_basecaller" \
--set guppy.gpu_calling=1 \
--set guppy.num_threads=16 \
--set guppy.ipc_threads=2

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 21, 2020

readfish

I decided to give ont-python a try and created a venv with ont-python (currently Python 3.7.4 (default, Oct 7 2019, 15:48:32)
[GCC 5.4.0 20160609] on linux)

I then replicated what works on a PC with RTX2070 and Ubuntu 18.04:
https://pypi.org/project/readfish/0.0.5a2/

I adapted the script not to use the system's python (works nicely on the PC)

ontpython=/opt/ont/minknow/ont-python/bin/python3.7
cd ~/
$ontpython -m venv readfish
source ./readfish/bin/activate
# upgrade pip
pip install --upgrade pip
pip install git+https://github.com/nanoporetech/read_until_api
pip install --pre readfish

and the result is, that it does not install spontaneously on the Xavier (I was almost sure it wouldn't, but it was worth the hope):

Collecting pip
  Downloading https://files.pythonhosted.org/packages/cb/28/91f26bd088ce8e22169032100d4260614fc3da435025ff389ef1d396a433/pip-20.2.4-py2.py3-none-any.whl (1.5MB)
    100% |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 1.5MB 6.0MB/s 
Installing collected packages: pip
  Found existing installation: pip 19.0.3
    Uninstalling pip-19.0.3:
      Successfully uninstalled pip-19.0.3
Successfully installed pip-20.2.4
Collecting git+https://github.com/nanoporetech/read_until_api
  Cloning https://github.com/nanoporetech/read_until_api to /tmp/pip-req-build-lo1iluti
Collecting grpcio
  Downloading grpcio-1.32.0.tar.gz (20.8 MB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 20.8 MB 103 kB/s 
Collecting requests
  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 61 kB 101 kB/s 
Collecting minknow-api
  Downloading minknow_api-4.0.4.tar.gz (188 kB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 188 kB 13.4 MB/s 
Collecting six>=1.5.2
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting certifi>=2017.4.17
  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 156 kB 21.7 MB/s 
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Downloading urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 127 kB 21.5 MB/s 
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 58 kB 1.1 MB/s 
Collecting chardet<4,>=3.0.2
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 133 kB 15.6 MB/s 
Collecting numpy~=1.11
  Downloading numpy-1.19.2-cp37-cp37m-manylinux2014_aarch64.whl (12.2 MB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 12.2 MB 15.6 MB/s 
Collecting protobuf~=3.11
  Downloading protobuf-3.13.0-py2.py3-none-any.whl (438 kB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 438 kB 16.7 MB/s 
Collecting packaging>=14.1
  Downloading packaging-20.4-py2.py3-none-any.whl (37 kB)
Requirement already satisfied: setuptools in ./readfish/lib/python3.7/site-packages (from protobuf~=3.11->minknow-api->read-until==2.1.0) (40.8.0)
Collecting pyparsing>=2.0.2
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 67 kB 3.3 MB/s 
Using legacy 'setup.py install' for read-until, since package 'wheel' is not installed.
Using legacy 'setup.py install' for grpcio, since package 'wheel' is not installed.
Using legacy 'setup.py install' for minknow-api, since package 'wheel' is not installed.
Installing collected packages: six, grpcio, certifi, urllib3, idna, chardet, requests, numpy, protobuf, pyparsing, packaging, minknow-api, read-until
    Running setup.py install for grpcio ... done
    Running setup.py install for minknow-api ... done
    Running setup.py install for read-until ... done
Successfully installed certifi-2020.6.20 chardet-3.0.4 grpcio-1.32.0 idna-2.10 minknow-api-4.0.4 numpy-1.19.2 packaging-20.4 protobuf-3.13.0 pyparsing-2.4.7 read-until-2.1.0 requests-2.24.0 six-1.15.0 urllib3-1.25.11
Collecting readfish
  Downloading readfish-0.0.5a1-py3-none-any.whl (58 kB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 58 kB 1.1 MB/s 
Collecting watchdog
  Downloading watchdog-0.10.3.tar.gz (94 kB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 94 kB 1.3 MB/s 
Collecting google
  Downloading google-3.0.0-py2.py3-none-any.whl (45 kB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 45 kB 1.6 MB/s 
Requirement already satisfied: requests in ./readfish/lib/python3.7/site-packages (from readfish) (2.24.0)
Collecting jsonschema
  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 56 kB 423 kB/s 
Requirement already satisfied: grpcio in ./readfish/lib/python3.7/site-packages (from readfish) (1.32.0)
Collecting mappy
  Downloading mappy-2.17.tar.gz (199 kB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 199 kB 17.4 MB/s 
Collecting toml
  Downloading toml-0.10.1-py2.py3-none-any.whl (19 kB)
Collecting biopython
  Downloading biopython-1.78.tar.gz (16.9 MB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 16.9 MB 12.1 MB/s 
Requirement already satisfied: protobuf in ./readfish/lib/python3.7/site-packages (from readfish) (3.13.0)
Collecting pyguppyclient>=0.0.7a1
  Downloading pyguppyclient-0.0.7a1.tar.gz (17 kB)
Collecting pandas
  Downloading pandas-1.1.3-cp37-cp37m-manylinux2014_aarch64.whl (9.4 MB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 9.4 MB 23.2 MB/s 
Requirement already satisfied: numpy in ./readfish/lib/python3.7/site-packages (from readfish) (1.19.2)
Collecting pathtools>=0.1.1
  Downloading pathtools-0.1.2.tar.gz (11 kB)
Collecting beautifulsoup4
  Downloading beautifulsoup4-4.9.3-py3-none-any.whl (115 kB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 115 kB 20.7 MB/s 
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./readfish/lib/python3.7/site-packages (from requests->readfish) (1.25.11)
Requirement already satisfied: idna<3,>=2.5 in ./readfish/lib/python3.7/site-packages (from requests->readfish) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in ./readfish/lib/python3.7/site-packages (from requests->readfish) (2020.6.20)
Requirement already satisfied: chardet<4,>=3.0.2 in ./readfish/lib/python3.7/site-packages (from requests->readfish) (3.0.4)
Collecting attrs>=17.4.0
  Downloading attrs-20.2.0-py2.py3-none-any.whl (48 kB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 48 kB 3.2 MB/s 
Requirement already satisfied: six>=1.11.0 in ./readfish/lib/python3.7/site-packages (from jsonschema->readfish) (1.15.0)
Collecting pyrsistent>=0.14.0
  Downloading pyrsistent-0.17.3.tar.gz (106 kB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 106 kB 17.9 MB/s 
Collecting importlib-metadata; python_version < "3.8"
  Downloading importlib_metadata-2.0.0-py2.py3-none-any.whl (31 kB)
Requirement already satisfied: setuptools in ./readfish/lib/python3.7/site-packages (from jsonschema->readfish) (40.8.0)
Collecting pyzmq==17.1.2
  Downloading pyzmq-17.1.2.tar.gz (1.1 MB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 1.1 MB 18.9 MB/s 
Collecting flatbuffers==1.11
  Downloading flatbuffers-1.11-py2.py3-none-any.whl (15 kB)
Collecting ont-fast5-api>=3.0.1
  Downloading ont_fast5_api-3.1.6-py3-none-any.whl (2.0 MB)
     |Ò–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆÒ–ˆ| 2.0 MB 18.2 MB/s 
ERROR: Could not find a version that satisfies the requirement ont-pyguppy-client-lib==4.0.11 (from pyguppyclient>=0.0.7a1->readfish) (from versions: none)
ERROR: No matching distribution found for ont-pyguppy-client-lib==4.0.11 (from pyguppyclient>=0.0.7a1->readfish)

That means I'll need to have a look into ont-pyguppy and probably talk to ONT.

Given the success with ont-python for readfish on the PC platform, I will also try get h5py to work with ont-python. This might be a good way to stick to current ONT releases.

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 21, 2020

Just got a reply on the ONT forum about the missing --port option:
https://community.nanoporetech.com/posts/enabling-gpu-basecalling-f

From 4.2.2 onward, there is a separate binary called guppy_basecall_client. Hence, scripts (or maybe even the binary's name or a symlink) will need to change. The guppy_basecaller binary is standalone. This explains why it won't work with MiniIT MinKNOW and probably readfish will also need adaptation to make this work again.

@lfaino

This comment has been minimized.

Copy link

@lfaino lfaino commented Oct 21, 2020

Dear,
for it works the 4.2.2. I think that in a later version of the ui, the option is not there anymore

cheers
Luigi

@lfaino

This comment has been minimized.

Copy link

@lfaino lfaino commented Oct 21, 2020

Which version of minknow-ui do you use?

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 21, 2020

Which version of minknow-ui do you use?

That's an interesting question @lfaino. From the screenshots you posted @juhench it does look like there are some differences in the UI between what you are running to what I am seeing.

@lfaino: I'm going to test the downgrade to 4.0.14 when I get into work in a couple of hours and if it works the same as it has for @juhench then I think we'll settle on this as a method until we can figure out how to replicate your success with 4.2.2. :)

@lfaino

This comment has been minimized.

Copy link

@lfaino lfaino commented Oct 21, 2020

btw, I had problems as well today (a student had) i think that when the ui update, thinks get changed and we should be careful. I will keep you posted if i can solve tomorrow

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 21, 2020

Dear all, what a great collaborative success!
I think we should create a version table what goes together and what does not. I am not in the lab right now.

I did not upgrade the ONT packages since the installation of these (with apt-get; not apt):

minknow-core-minit-offline_4.0.5-xenial_arm64.deb
ont-bream4-minit_6.0.10-1~xenial_all.deb
ont-configuration-customer-minit_4.0.16-1~xenial_all.deb
ont-kingfisher-ui-minit_4.0.21-1~xenial_all.deb
ont-minknow-static-frontend_3.6.17-1~xenial-stable-minit_all.deb
ont-remote-support_3.0.5-0~xenial-minit_all.deb
ont-system-identification_3.5.1-0~xenial-minit_all.deb

The guppy versions that are intended to work with this MinIT system are 3.x and 4.0.x. The closest currently available was 4.0.14.

I have to check the command line arguments for guppy_basecall_client. Currently, to get 4.2.2 to work (I can maybe try tomorrow), I would first try to modify the config:

sudo /opt/ont/minknow/bin/config_editor --conf application --filename /opt/ont/minknow/conf/app_conf \
--set guppy.server_executable="/opt/ont/guppy_4.2.2/bin/guppy_basecall_server" \
--set guppy.client_executable="/opt/ont/guppy_4.2.2/bin/guppy_basecall_client" \
--set guppy.gpu_calling=1 \
--set guppy.num_threads=16 \
--set guppy.ipc_threads=2

Note the change in the 3rd line: we would refer to the binary that is intended to basecall "externally", i.e. in the guppy_basecall_server. I would, for the sake of completeness, keep the various guppy versions in parallel within respectively named directories. I have been doing so with the x86_64 architecture incl. those with GPUs. It helps in keeping the overview.

As for the UI: The UI I am using has a nice addition for simulation runs, which allows switching TOML files by a (new) pull-down menu. This is, in particular, useful for setting up readfish but also for benchmarking guppy. The pulldown menu appears if multiple TOML files named with the same flow cell prefix exist, i.e. from duplication of a particular TOML file into XXX_simulation.toml.

To avoid unwanted upgrading in a working system, I suggest commenting out the ONT repos after the initial install and, in addition, use apt-clone to create a backup of the currently installed packages (alternatively, copy the deb files after installation with apt-get (not apt)) into a safe place (from /var/cache/apt). This will be very useful if you need to flash the Xavier for some reason. You can then apt-clone everything back into running state (hopefully) or, at least, install the debs with dpkg.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 22, 2020

Amazing collaborative effort all round! A quick downgrade to guppy 4.0.14 did the trick, here are a lot of screenshots...

This run was using an old flow cell that had a COVID ARTIC prep on it, as can be seen below it's working rather well even after being sat for a long time.

Screenshot from 2020-10-22 12-37-46

Screenshot from 2020-10-22 12-38-26

Screenshot from 2020-10-22 12-38-38

Screenshot from 2020-10-22 12-38-49

Screenshot from 2020-10-22 12-39-03

Screenshot from 2020-10-22 12-39-47

Screenshot from 2020-10-22 12-40-04

Screenshot from 2020-10-22 12-41-09

... and the wider view of everything running (including my brand new Logitech mouse and keyboard!):

IMG_20201022_134219

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 22, 2020

@sirselim: cool :) / hot :) no matter how you look at it, it's great!

Below is a picture of the "Basel" setup. As mentioned above, we're running all of our Nanopore equipment as remote desktops (vnc4server); also, to be able to help our technical staff that does the routine testing when we're out of the office through a remote connection. The devices are hooked up to our LAN. Just fits nicely in a corner next to my desk between the document scanner (left) and typewriter (right; we use that for quickly printing on sticky labels). Also, note the advanced dust cover on the PCI slot :) and our brand new MinION device; unfortunately, one can not easily obtain many of them; in part understandable, but bad for our cutting-edge development. I find it amazing that a flow cell that had reached its end of life (see the notes) is still producing reads that are correct. The sequencing chemistry seems to be quite robust.

IMAG0922

Enough with the joking - back to work now.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 22, 2020

@juhench - very nice! Thanks for the break down of your set up, really neat idea to be able to easily provide remote support. I also love the PCIe dust cover. ;)

I'm starting the arduous process of compiling all of my notes and writing up a 'proper' guide to setting up a MinION/MinKnow/Guppy on Nvidia Jetson Xavier AGX/NX boards. Hopefully you @juhench and @lfaino will be able to chip in here and there if you feel so inclined. It would be great to confirm that the process also works for the TX2 @lfaino, we also need to get to the bottom of the fact that you can run guppy 4.2.2 just fine as well...

I've started by pulling together everything from the top of my head (and our collective notes) that was needed in terms of 'ONT' specific packages/libraries. This is what it currently looks like:

xavier_minit_build/
β”œβ”€β”€ debs
β”‚Β Β  β”œβ”€β”€ minknow-core-minit-offline_4.0.5-xenial_arm64.deb
β”‚Β Β  β”œβ”€β”€ ont-bream4-minit_6.0.10-1~xenial_all.deb
β”‚Β Β  β”œβ”€β”€ ont-configuration-customer-minit_4.0.16-1~xenial_all.deb
β”‚Β Β  β”œβ”€β”€ ont-kingfisher-ui-minit_4.0.21-1~xenial_all.deb
β”‚Β Β  β”œβ”€β”€ ont-minknow-frontend-minit_3.6.16-1~xenial_arm64.deb
β”‚Β Β  β”œβ”€β”€ ont-minknow-static-frontend_3.6.17-1~xenial-stable-minit_all.deb [*]
β”‚Β Β  └── ont-remote-support_3.0.5-0~xenial-mk1c_all.deb
β”œβ”€β”€ guppy
β”‚Β Β  β”œβ”€β”€ guppy_arm64_cuda10_urls.txt
β”‚Β Β  β”œβ”€β”€ ont-guppy_4.0.14_linuxaarch64_cuda10.tar.gz
β”‚Β Β  β”œβ”€β”€ ont-guppy_4.0.15_linuxaarch64_cuda10.tar.gz
β”‚Β Β  └── ont-guppy_4.2.2_linuxaarch64_cuda10.tar.gz
└── python
    β”œβ”€β”€ libs
    β”‚Β Β  β”œβ”€β”€ python2.7
    β”‚Β Β  β”‚Β Β  └── h5py.zip
    β”‚Β Β  └── python3.7
    β”‚Β Β      └── h5py.zip
    └── readme.txt

6 directories, 15 files

Note: I decided to keep the python2.7 h5py lib as some people may be interested in running legacy GUI (ont-minknow-frontend-minit_3.6.16-1~xenial_arm64.deb), which is working completely fine on one of our Xavier AGX boxes. I'm going to upgrade that machine to the newer Kingfisher UI but it has been interesting seeing that currently both versions will run in this environment. I HAVEN'T as yet tested live base calling and it may not work as the versions are quite different between MinKnow and Guppy.

@juhench - for fun I had another quick look and found another archived version of guppy 4.0.X: ont-guppy_4.0.15_linuxaarch64_cuda10.tar.gz. I haven't installed this yet, but I imagine it will work just fine when replaced either on top of or parallel to 4.0.14. I had a very quick play today with your ideas around playing with the app_conf paths, but didn't have any joy. Interested to see how you get on. I also think you have an interesting point about commenting out the ONT repo once installed. I'm very much intrigued to see for how long ONT will update software specifically for the MinIT now that it has been discontinued for a few months. On the one hand there must be a decent amount of these devices out there so people will be expecting support and updates. On the other the Mk1c is the 'replacement' device in this space, although there must be a Xavier variant on the horizon...

Another thing we tried for fun, was installing all the Mk1c debs - by using this repo deb http://mirror.oxfordnanoportal.com/apt xenial-stable-mk1c non-free - we got some very interesting results. I think it may be possible, but the biggest issue is that when running the Mk1c GUI it takes complete control of the OS, i.e. you have to drop to tty and kill the process. I was thinking that if the MinIT loses support then the Mk1c is essentially the same internal hardware, so we'd have a backup to turn to for loading on to our '3rd party' devices. This needs a lot more exploration, but was a fun thought experiment.

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 22, 2020

I just took the freedom to write to ONT and suggested that they could make a repo dedicated to the NVIDIA Xavier. The next thing I will try to implement is nanoDx (Philipp Euskrichen's [brain] AI-based tumor classification system) and merge that with our UMAP-based system (www.epidip.org). All of this could, in my opinion, run alongside MinKNOW and guppy on the Xavier and would deliver tumor classification during [brain] surgery. This would be game-changing for surgical procedures, and ONT might be interested in supporting it. While it is a hassle to set up PC hardware to run all these tasks at once (it is actually a hassle, I am remotely setting up a colleague's system somewhere else who has no IT background or staff), a Xavier, even a preconfigured one or better, through a repo, would really become distributable. Let's see what ONT thinks. I see their point in both, dumping MinIT soon and taking control of the GUI on Mk1c. Unfortunately, one of the systems currently offered hardware systems by ONT fits the need of our tumor diagnostics while the Xavier seems almost perfectly suited. Also, the Xavier will disappear as quickly as it came but probably will have a comparable successor. Then, in particular, it would be good to have a (development) repo from ONT so that all that was achieved now by you (@miles) during the past months, would come as one "apt-add repo" for the different hardware solutions produced by NVIDIA (and maybe others? well rather not).

I will give guppy 4.2.2 a short try and report back. I would love to see a GitHub repo that would sum up the efforts from this page, however, I fear it will be very short-lived as we're not allowed to package the non-free software (which is sad, but it is how it is). Every new version will, again, not be adapted to the Xavier, so things like with guppy 4.2.2 and h5py can be expected to come daily routine. This makes 3 things impossible:

  • run the same system in many institutions
  • get it accredited as a diagnostic system
  • support non-IT professionals (i.e., the majority of my colleagues, to date)
@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 22, 2020

Quick note: I suggest outcommenting the ONT repo for a moment.
Dear all, it seems that we are currently running (+/-) the same software versions, i.e. a MinKNOW 20.0.X and guppy 4.0.x. I just out-commented the ONT repo for a while. I would make things a bit more consistent (here, for us). For the NVIDIA-Things, I observed GPU temperature and fan operation during the run, and also during post-hoc basecalling (GPU load 100% for a while. At least for me, the fan operation is as expected, and the components are being cooled appropriately (without modification of fan control).

I also saw that the current MinKNOW offers (live?) alignment and I will try it out soon.

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 22, 2020

A brief annoyance after upgrading to the latest Xavier packages (Ubuntu only, 2020-10-22). The MinKNOW service did not launch on boot.

I did this:

user@meqneuropat08:~$ sudo /opt/ont/minknow/bin/mk_manager_svc 
^C
user@meqneuropat08:~sudo service minknow start
user@meqneuropat08:~$ sudo service minknow status

After hearing that the MinION's fan slowed down (mk_manager_svc) I killed the process and started MinKNOW again. Status messages before that (from the service minknow status) were really misleading. I guess it might again be some sort of permission problem with the USB (see posts above). However, the above sequence of command provides a simple fix for the moment. Need to figure out what goes wrong here.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 22, 2020

You've been busy @juhench! A few comments in reply:

  • Did you hear back from ONT regarding ont-pyguppy-client-lib? I was looking into this as well as we have quite a few readuntil experiments to run. The issue is that the lib isn't available compiled for arm64: https://pypi.org/project/ont-pyguppy-client-lib/4.0.15/#files The good news is that python version doesn't seem to matter as long as it's between 3.5-3.8. I wonder what the chances are of getting this built on arm...?

  • Interesting about the update, I haven't seen anything come through in the repos on my end yet (maybe I already updated?). I'll have to check the versions of the installed packages out and compare them to yours.

  • Love the thinking around your Xavier use in real-time for surgery. Nvidia highlighted this very thing (using Xaviers) recently at the Nvidia GTC conference, meet the Nvidia Clara AGX developer kit: https://developer.nvidia.com/clara-agx-devkit In one of the talks they showed footage of a surgeon virtually marking regions internally so that they could easily relocate to the exact spot upon returning. I got very excited! I have a PhD student working on deep learning in medical images/videos who also has a passion for genomics, and my mind went straight to "you could easily have sequencing running alongside this on the same device. So I'm trying to get my hands on one of these little beasts with the goal of getting the minion workflow ported across (should be easy as it's a Xavier at heart) but then having the power of a RTX6000 really opens up possibilities. I look forward to your feedback on how something like nanoDX runs on the Xavier. It might be that the 'little' GPU starts to get a bit stressed running that and GPU base calling, but only one way to find out. That actually reminds me, I need to see what happens when you plug a pcie GPU into the Xavier... anyone tried yet?

@thor-johnson

This comment has been minimized.

Copy link

@thor-johnson thor-johnson commented Oct 22, 2020

Hi all, very much appreciate this community effort. I'm working on setup of MinIT software on Xavier NX. Our end goal is to have a 'headless' system for automated DNA processing. Running the MinKNOW gui is a check on whether we have all the underlying pieces in place and then our next step is to script the underlying utility apps. I am not yet seeing MinKNOW recognize the MinION but I'm likely missing a package. I did figure out something with mk_manager_svc which is registering it with systemd for proper startup.

// below are the steps to register mk_manager_svc with systemd

nano minknow.service (nano is a text editor, steps assume the file is created in ~)

//insert the following lines:
[Unit]
Description=MinKNOW Instrument Software for MinIT (daemon)

[Service]
ExecStart=/opt/ont/minknow/bin/mk_manager_svc
WorkingDirectory=/opt/ont/minknow
KillMode=mixed
User=minit
Group=minit
SyslogIdentifier=minknow
LogsDirectory=minknow
LimitCORE=infinity
LimitNICE=40
NoNewPrivileges=true
ExecStartPre=/bin/sleep 15
CPUQuota=300%

[Install]
WantedBy=multi-user.target

// save and exit text editor
// copy the minknow.service file to the systemd/system directory and change permissions
sudo cp minknow.service /etc/systemd/system/minknow.service
sudo chmod 644 /etc/systemd/system/minknow.service

sudo systemctl start minknow
sudo systemctl status minknow

//Use the enable command to ensure that the service starts whenever the system boots:
sudo systemctl enable minknow

//[Restart XavierNX] Shutdown -> Restart

//open a terminal and check minknow
sudo systemctl status minknow

//output should be something like:
minknow.service - MinKNOW Instrument Software for MinIT (daemon)
Loaded: loaded (/lib/systemd/system/minknow.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2020-10-12 14:20:09 EDT; 1 weeks 3 days ago
Process: 968 ExecStartPre=/bin/sleep 15 (code=exited, status=0/SUCCESS)
Main PID: 2206 (mk_manager_svc)
Tasks: 132
Memory: 88.2M
CPU: 1d 1h 29min 16.474s
CGroup: /system.slice/minknow.service
β”œβ”€2206 /opt/ont/minknow/bin/mk_manager_svc
β”œβ”€2216 /opt/ont/minknow/bin/crashpad_handler --database=/data/core-dump-db --metrics-dir=/data/core-dump-db --url
β”œβ”€2236 /usr/bin/guppy_basecall_server --config dna_r9.4.1_450bps_fast.cfg --port 5555 --log_path /var/log/minknow
β”œβ”€2237 /opt/ont/minknow/bin/grpcwebproxy --server_tls_cert_file=/opt/ont/minknow/conf/rpc-certs/localhost.crt --s
β”œβ”€2238 /opt/ont/minknow/bin/basecall_manager --guppy-address 5555 --conf /opt/ont/minknow/conf 127.0.0.1:9504
β”œβ”€2248 /opt/ont/minknow/bin/crashpad_handler --database=/data/core-dump-db --metrics-dir=/data/core-dump-db --url
β”œβ”€2260 /opt/ont/minknow/bin/grpcwebproxy --server_tls_cert_file=/opt/ont/minknow/conf/rpc-certs/localhost.crt --s
β”œβ”€4506 /opt/ont/minknow/bin/control_server --config /tmp/minknow_5bdd88af-4446-4d7b-8ac7-f462521ff2ee/conf-MN3214
β”œβ”€4509 /opt/ont/minknow/bin/crashpad_handler --database=/data/core-dump-db --metrics-dir=/data/core-dump-db --url
β”œβ”€4544 /opt/ont/minknow/bin/analyser --config /tmp/minknow_5bdd88af-4446-4d7b-8ac7-f462521ff2ee/conf-MN32140 --sh
β”œβ”€4545 /opt/ont/minknow/bin/grpcwebproxy --server_tls_cert_file=/opt/ont/minknow/conf/rpc-certs/localhost.crt --s
└─4554 /opt/ont/minknow/bin/crashpad_handler --database=/data/core-dump-db --metrics-dir=/data/core-dump-db --url

// here are other commands for the minknow service (which launches mk_manager_svc)
// Note that the MinION fan will come on after boot and then quiet down once the mk_manager_svc daemon backplane is doing it's job
sudo systemctl stop minknow
sudo systemctl restart minknow

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 22, 2020

Hi @thor-johnson, this reminds me of what I experienced this afternoon after the system upgrades and reboot, but also before. It might have to do with USB device permissions (but I am not 100% sure there). Does the MinION fan spin fast indefinitively when connecting it? This happens if mk_manager_svc is not running properly. If so, try this:

user@meqneuropat08:~$ sudo /opt/ont/minknow/bin/mk_manager_svc 
^C
user@meqneuropat08:~sudo service minknow start
user@meqneuropat08:~$ sudo service minknow status

Some seconds after issuing launching mk_manager_svc as root, the MinION fan should slow down to normal (silent) levels. Once that happens, you can CTRL-C the application and try launching MinKNOW again. Interestingly, the mk_manager_svc application then runs under the user "minit". Did you create a user called "minit"? I also added that user to all groups on the system. I am not sure in which groups it is on a real MinIT (I don't have one).

I still have to figure out why this problem occurs. It did not happen during the last reboot, but there was a system (ubuntu jetpack) upgrade in between. The error message within service status output once contained a statement about problems with writing the log (without specifying which log or whether it was a permissions error).

I am operating the Xavier in headless mode. This works both through a vnc4server-based desktop environment (I am using xfce4, personal preference) but also through the MinKNOW GUI on another computer (tested with 4.0.x, current release). I personally like it better with the virtual desktop, as you can secure the device more easily on a LAN (e.g., use ssh). I put the device behind an openWRT router that acts as a ssh login server and to connect I do xvnc4viewer -via root@myrouter myXavier:1. You can now use anything that runs an X server and has a mouse/keyboard to operate your Xavier remotely. You can also specify the -alwaysshared option in vnc4server so that mulitple people can interact with the Xavier desktop at the same time. That way, I can help our lab technicians from anywhere.

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 22, 2020

@sirselim

  • Did you hear back from ONT regarding ont-pyguppy-client-lib? I was looking into this as well as we have quite a few readuntil experiments to run. The issue is that the lib isn't available compiled for arm64: https://pypi.org/project/ont-pyguppy-client-lib/4.0.15/#files The good news is that python version doesn't seem to matter as long as it's between 3.5-3.8. I wonder what the chances are of getting this built on arm...?

I hinted at that in my mail today. ONT should have some interest in supporting ARM but, in the end, it remains up to them. I still don't have a reply, but such answers need some processing. I guess these will go through internal discussions first. Btw: what is your average fragment read length? This is one of our problems with readuntil. Our fragments are still too short to really see selectivity (average 2.5kb, N50 6-7 kb). We're working on optimization of the DNA extraction procedure, but since it's about brain tumors for us, we have very limited amounts of tissue.

  • Interesting about the update, I haven't seen anything come through in the repos on my end yet (maybe I already updated?). I'll have to check the versions of the installed packages out and compare them to yours.

The Jetpack Ubuntu did update quite some packages today for me... In the end, it said that I was up to date now. After the reboot, mk_manager_svc did not launch until after executing it as root once. I then even ran a sequencing on my "scrap" flow cell and it worked again.

I need to see what happens when you plug a pcie GPU into the Xavier... anyone tried yet?

I was hesitant. Just ordered two more Xaviers today (which is a problem for Switzerland); hope I will get them. Once received, I will try with a PC PSU for powering both the Xavier and the GPU. That system is going to look funny. Using 2 PSUs can be dangerous for the components. I have once fried a USB port on an old PC by using two PSUs on an experimental system due to a voltage difference between the PSUs. I am not sure whether The addon GPU can work. If the GPU architectures of the Xavier and the PCI GPU are the same, then it might work. If not, I think it won't. I have not been successful in installing two different GPU architectures in a single PC even though both were theoretically supported by the respective driver (package). The Nvidia driver installers sense the type of GPU and then compile the kernel modules (at least on the PCs). This requires the GPU to be present in the system. It could end up having either GPU not working any longer. I strongly agree that if it would work, it would be a useful: More GPU cores and extra (separate) GPU RAM.

Love the thinking around your Xavier use in real-time for surgery. Nvidia highlighted this very thing (using Xaviers) recently at the Nvidia GTC conference, meet the Nvidia Clara AGX developer kit: https://developer.nvidia.com/clara-agx-devkit

Thanks for the hint - I also subscribed; let's see. This looks like the pro version of adding a GPU to the PCI slot. Still, I am hoping that I can get NanoDx to run (and maybe readuntil, but if binaries are involved, we depend on ONT)

@Tetrakis

This comment has been minimized.

Copy link

@Tetrakis Tetrakis commented Oct 22, 2020

All of this looks really exciting! Great work everyone! I can’t wait to unbury myself from my teaching load and get back to this. I looked into a PCI GPU for he Xavier a while ago and I don’t believe it’s supported yet because of power distribution issues. https://forums.developer.nvidia.com/t/graphic-card-plug-into-xavier-solved/65310/11

@thor-johnson

This comment has been minimized.

Copy link

@thor-johnson thor-johnson commented Oct 23, 2020

@juhench Yes, I created a the user 'minit'. Here is my procedure:

Step 0: XavierNX hardware setup and connections

  • Install SSD on underside of Xavier NX. Samsung EVO Plus 970 2TB Gen3 PCI is a decent choice, Samsung 860 should work also. Philips screwdriver and care should be taken with regards to static / ESD. Remove screw, insert SSD flat to the back of the XavierNX, press into connector, then align with screw hole, thumb on back side of connecotr finger on the edge of SSD squeeze firmly as needed to seat and align with hole, screw into place.

Connections (power disconnected)

Step 1: setup user minit

  • Apply power to the XavierNX, it should boot to Ubuntu 18.04 with NVidia logo

  • Create user β€˜minit’ with password β€˜Bigminit555’ with Admin priv in the Ubuntu setup process.

  • Open a terminal (Alt-Ctrl-T) and change the password
    minit@xaviernx1:~$ sudo passwd minit
    [sudo] password for minit:
    Enter new UNIX password: minit
    Retype new UNIX password: minit
    passwd: password updated successfully

// Not sure if this step is necessary, but I did it after some iterations

  • change password from Bigminit555 to minit
    minit@xaviernx1:$ su -
    Password:
    root@xaviernx1:# passwd
    Enter new UNIX password: minit
    Retype new UNIX password: minit
    passwd: password updated successfully
    root@xaviernx1:# exit
    logout

  • From the ubuntu desktop, use β€˜Brightness and Lock’ to disable screen/power save which is defaulted to 5min.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 23, 2020

@thor-johnson - Welcome to our little corner of Jeston hackery! πŸ˜‰

In response to your first point, that's rather odd. Are you saying you created the service file? For me it was already there and registered correctly with systemd from the get go.

I also believe you and @juhench might want to check this file and change the user and group both to root. That's what mine is set to and I haven't had any issues with permissions since, plus I've updated my system (via apt upgrade) numerous times now with zero breakage. Just a suggestion. πŸ€·β€β™‚οΈ

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 23, 2020

@thor-johnson: Looks OK to me. I still did not upgrade the SSD; not sure if I will or if this is even necessary. I have network-mounted (sshfs) storage into /data. Then, of course, network failure will terminate sequencing. I guess I will stick to a 16TB HDD eSATA storage in the long run. The Xavier is attached to a gigabit router (NBG6515 with openWRT, nothing fancy). Another setup could be a router with USB3-NAS capability. Will test this as well; it would make the setup independent of a LAN and allow simple connection of, e.g., a laptop device.

If you are using the GPU for desktop rendering you are increasing the load on the GPU, even though not by much.

What does your log of mk_manager_svc look like?

user@meqneuropat08:~$ cat /var/log/minknow/mk_manager_svc_log-0.txt 
2020-10-22 18:54:26.417754    INFO: guppy_server_version (util)
    version: 4.0.14+8d3226e
2020-10-22 18:54:26.428129   ERROR: failed_to_parse_identity_config (data)
    path: /etc/oxfordnanopore/configs/identity.config
2020-10-22 18:54:26.431785    INFO: mk_manager_starting (mk_manager)
    system: User-provided system running on ubuntu 18.04
    version: 4.0.5
2020-10-22 18:54:26.456838    INFO: guppy_server_version (util)
    version: 4.0.14+8d3226e
2020-10-22 18:54:26.460243    INFO: mk_manager_initialised (mk_manager)
    pid: 21426
2020-10-22 18:54:26.464241    INFO: common_process_started (host)
    name: guppy
2020-10-22 18:54:26.469379    INFO: common_process_started (host)
    name: grpcwebproxy
2020-10-22 18:54:26.473254    INFO: common_process_started (host)
    name: basecall_manager
2020-10-22 18:54:26.584333    INFO: rpc_delegate_is_listening (host)
    executable: basecall_manager
    security: none
    port: 9504
2020-10-22 18:54:26.588978    INFO: common_process_started (host)
    name: basecall_manager:grpcwebproxy
2020-10-22 18:54:26.610279    INFO: rpc_delegate_proxy_is_listening (host)
    executable: basecall_manager
    tls_port: 9505
    insecure_port: 9506
2020-10-22 18:54:26.719188    INFO: instance_started (host)
    grpc_secure_port: 8001
    instance: MN35026
    grpc_port: 8000
    grpcweb_insecure_port: 8003
    grpcweb_tls_port: 8002
    jsonrpc_port: 0
    shmem_prefix: mk21496

I have been running everything as both, "minit" and "user"; no difference.

If you're getting permission errors about USB, try this:

sudo service minknow status # probably not runnΓ­ng anway
sudo service minknow stop # if it is running, otherwise skip
sudo /opt/ont/minknow/bin/mk_manager_svc # wait a while; MinION fan speed should become silent
^C # CTRL-C
sudo service minknow start
sudo service minknow status

Maybe you can paste your output here.

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 23, 2020

@sirselim: Do you mean the ownership of the service file? Which path exactly? I also did not modify anything here.

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 23, 2020

@Tetrakis: Thanks for the link. Too bad with the 2nd GPU. In addition, I just checked the Xavier's PSU: It's not 12V DC but 19V DC, hence it will not be possible to use a generic PC PSU top power both a PCIe GPU and the Xavier from that. What might work to resolve the power issue for a PCIe GPU is the use of a riser board. Our 2nd testing setup was a small form factor HP desktop machine (we have a pile of them laying around here unused) and a 2nd power supply connected to GPU (one 8-pin connector on the RTX2070 and one 6-pin connector to the riser board (e.g., https://www.amazon.de/gp/product/B073TQ1D69/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1). That way, you would not put any load on the Xavier's PCIe. Of course, driver issues will remain.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 23, 2020

@juhench:

I still did not upgrade the SSD; not sure if I will or if this is even necessary

I'm really interested to start getting together some collaborative benchmarking. In my experience having high speed local I/O gives a significant boost to performance with Guppy basecalling. I would suggest even putting a lower capacity (512Gb) high speed solid state nvme drive in there and then have scripts move the runs to your network attached storage when they're done.

In terms of the minknow service file I mean replace minit with with root under the flags USER and GROUP. I believe it fixes a few niggles, i.e. Permissions etc. I noticed you have errors in your logs which I don't get either, so might sort that out as well?

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 23, 2020

@Tetrakis: Thanks for the link. Too bad with the 2nd GPU. In addition, I just checked the Xavier's PSU: It's not 12V DC but 19V DC, hence it will not be possible to use a generic PC PSU top power both a PCIe GPU and the Xavier from that. What might work to resolve the power issue for a PCIe GPU is the use of a riser board. Our 2nd testing setup was a small form factor HP desktop machine (we have a pile of them laying around here unused) and a 2nd power supply connected to GPU (one 8-pin connector on the RTX2070 and one 6-pin connector to the riser board (e.g., https://www.amazon.de/gp/product/B073TQ1D69/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1). That way, you would not put any load on the Xavier's PCIe. Of course, driver issues will remain.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 23, 2020

@juhench - regarding GPU in the PCIe slot. I've got a passive Quadro P400 which I replaced with a Titan RTX. So I plan on chucking that in there and seeing what I can see. The power thing can't be that much of an issue right? πŸ˜‰ In the Clara AGX Nvidia have incorporated a Xavier alongside an RTX 6000 and a small form factor PSU. I'm sure we can solve something together!

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 23, 2020

@sirselim: I have not been basecalling other than "live" yet. The live mode is very mild on the GPU. I fully agree (I noticed from our server) that posthoc basecalling is much faster when copying the data from a network share (on 10GBit glass fiber LAN) to the Optane PCIe memory in our X86_64 120 cores server; even with prefetching to optane memory and basecalling on an external PC (Gigabit Ethernet only) on the RTX2070 (I can't put a GPU into that server, unfortunately, since I have no power connector and can't find it on the market for this model), it remains faster, prefetching included.

I have also tried remote basecalling "to" the Xavier, i.e., from the X86_64 PC server through Gigabit LAN (PC in client mode, Xavier in basecall_server mode). This runs a bit slower than an RTX2070 in a desktop PC; it was an unfair comparison, however, the guppy version on the RTX2070 is older (3.x) than on the Xavier 4.0.x.

Maybe we can share a fast5 set for benchmarking. Do you have some with microbial or cell culture data (devoid of human DNA). I don't; ours are all patient's data? Otherwise, I could go ahead with Matt Loose's demo file and "playback" for the first 10 minutes. The resulting fast5 could be placed here at GitHub so we would benchmark on the same file.

Quick question: where exactly is the service file? Would be you so kind as to post the path and the file contents?

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Oct 23, 2020

@sirselim: I would not plug that into the Xavier. Even though the Xavier PSU says 65W and probably the PSU components on the Xavier board that provide 3.3V/5V/12V might be designed for more than 25W, I would get a riser board. You will then be on the safe side. I was also thinking of connecting one of the many passive GPUs we have around here (I kept many for tinkering purposes) but decided against it. We have also seen a desktop PC tower fail when the GPU draws too much PCIe power. It boots normally, but as soon as one puts a load on the GPU - ZAP... and reboots. From the watts indicated on the PSU, it should have worked.

The key is the circuitry for the PCIe power "part", i.e. the long edge connector. Here, as you can see from the riser board picture, no pins are connected.; only the data bus (the small part of the connector).

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 23, 2020

Quick question: where exactly is the service file? Would be you so kind as to post the path and the file contents?

I'm not in easy access to any devices unfortunately. If you do a find from root looking for 'minknow.service' you'll track it down in no time. Actually, @thor-johnson posted details further up the thread, which is why I noticed they were using minit as USER and GROUP. The path is also mentioned in that post but I believe @thor-johnson created that file and moved it so I would try a find first to confirm it's location.

In terms of GPUs, thanks for the warning, I'll be careful. I've got an eGPU enclosure to play with as well.

In terms of benchmarking, yes I can find plenty of non human data. While I'm in the human genomics team we have lots of people working on bacteria and viruses. I could put together some COVID data. I'll look at getting that together next week, we have a long weekend here in NZ so I'll be a bit quiet over the next few days.

@thor-johnson

This comment has been minimized.

Copy link

@thor-johnson thor-johnson commented Oct 23, 2020

@sirselim Thanks for the welcome and I fully appreciate the forum for discussion and self help. As I go, I'm documenting the procedure and leveraging what yourself and others have done.

I have two XavierNX, and a TX2, as well as access to a MinIT. Please let me know if you need help documenting the procedure. I got all snarled up on h5py source build yesterday and need to start over on one of the XavierNXs.

Here's what I'm doing for install, and please note that systemd service is NOT setup. I suspect I'm missing the installation of one of the ONT packages. I noticed a statement above that you mentioned installing other packages

"Frustrated and on a whim, I installed a couple of other ont packages (currently trying to determine which because I didn't write them down! EDIT: may have been ont-minknow-frontend-minion) and then opened the minknow-ui on the Xavier itself."

I get an error, E: Unable to locate package ont-minknow-frontend-minion...

Here's the procedure I'm using. I do the upgrade of ubuntu/nvidia first, then apt-get to save the ont packages.

// Turn up performance to 15W 4core (minimap2 is best with 3 cores)
sudo /usr/sbin/nvpmodel -m 1

// update ubuntu and nvidia packages and add few utilities
sudo apt update
sudo apt install apt-utils
sudo apt install curl
sudo apt install hdparm //useful for SSD
sudo apt install nano //optional step to install a simple text editor.

sudo apt upgrade (15 to 20 min)

// setup ONT package repo
sudo nano /etc/apt/sources.list.d/nanoporetech.list
Add the line:
deb [trusted=yes] http://mirror.oxfordnanoportal.com/apt xenial-stable-minit non-free
Save and exit

// add cert
wget -O- https://mirror.oxfordnanoportal.com/apt/ont-repo.pub | sudo apt-key add -

// ONT packages, note apt-get for downloading the packages to be saved
sudo apt update
sudo apt-get upgrade

// This is the subset package list recommended by sirselim
// https://gist.github.com/sirselim/2ebe2807112fae93809aa18f096dbb94
// Note: there are still some issues with install and setup and this is a work in progress,
// likely due to some packages / installation scripts not installed
// Note: apt-get, pkgs to /var/cache/apt/archives/

sudo apt-get install minknow-core-minit-offline ont-bream4-minit ont-configuration-customer-minit ont-minknow-frontend-minit ont-minknow-static-frontend

Note: in posting this, I realize I may need to redo the install from scratch, I think I've been thrashing a bit on apt installs... I will edit this after I do that.

// Verify the packages installed
"minit@xaviernx2: apt list --installed | grep 'ont-'"
"ont-bream4-minit/stable,now 6.0.10-1 xenial all [installed]"
"ont-configuration-customer-minit/stable,now 4.0.16-1 xenial all [installed]"
ont-kingfisher-ui-minit/stable,now 4.0.21-1 xenial all [installed]
ont-minknow-static-frontend/stable,now 3.6.17-1 xenial-stable-minit all [installed]
ont-remote-support/stable,now 3.0.5-0 xenial-minit all [installed]
ont-system-identification/stable,now 3.5.1-0 xenial-minit all [installed]

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 26, 2020

@thor-johnson - that's strange that you have had to create the minknow.service file. I modified the file that is located here: /lib/systemd/system/minknow.service, it was there by "default". I just replaced USER and GROUP with 'root' (it was 'minit') and then restarted the service using service minknow restart. Contents of minknow.service below:

minit@xavnx:/$ cat /lib/systemd/system/minknow.service 
[Unit]
Description=MinKNOW Instrument Software for MinIT (daemon)

[Service]
ExecStart=/opt/ont/minknow/bin/mk_manager_svc
WorkingDirectory=/opt/ont/minknow
KillMode=mixed
User=root
Group=root
SyslogIdentifier=minknow
LogsDirectory=minknow
LimitCORE=infinity
LimitNICE=40
NoNewPrivileges=true
ExecStartPre=/bin/sleep 15
CPUQuota=300%


[Install]
WantedBy=multi-user.target

I suspect I'm missing the installation of one of the ONT packages. I noticed a statement above that you mentioned installing other packages

"Frustrated and on a whim, I installed a couple of other ont packages (currently trying to determine which because I didn't write them down! EDIT: may have been ont-minknow-frontend-minion) and then opened the minknow-ui on the Xavier itself."

I get an error, E: Unable to locate package ont-minknow-frontend-minion...

Ignore my past self, that was a red herring - I was missing a few steps at that point in time and didn't recall exactly what I had tried. This is why I am putting together a more refined set of notes (a set up guide if you will) - this gist is a great place to document, troubleshoot and collaborate, BUT it can provide outdated/misleading information as we refine our processes.

The 6 specific ONT packages that are required to get up and running with the latest Kingfisher UI are are:

minknow-core-minit-offline
ont-bream4-minit
ont-configuration-customer-minit
ont-kingfisher-ui-minit
ont-remote-support
ont-system-identification

NOTE: in an above post I mention legacy UI versions and specific deb files. I just want to make it clear here that the above 6 packages are required for the Kingfisher UI of MinKnow.

I believe you have all of these packages installed based on your notes that you have posted. Could you check to see if /lib/systemd/system/minknow.service is present on your install?

EDIT: actually, @thor-johnson do you have the minknow-core-minit-offline package installed?

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 29, 2020

Update 29th October 2020

Nothing 'major' or new to report really, other than I've added a 7 inch touch screen to further our efforts towards portability, and a story about trying to get it set up. First here's a picture:
image

A little story...

It's an extremely cheap and cheerful no-brand thing (link), but the display is nice enough and MinKnow is actually very easy to read and use. The one current issue I'm having is that the touch functionality works perfectly on the gdm3 login screen and then for a minute or so after login, it then stops working. The screen/display is fine, just not touch input. Looking through the logs give some hints, I currently believe it's an issue with power/suspending of the usb ports.

I spent a decent amount of time trying to fix this, to the point where I may have been a little too tired and tried a system update after messing with various kernel modules etc etc. This ended in a boot loop that I didn't want to spend time digging into. I was happy in my knowledge that my mirrored/back up sd card was ready^... until it also turned out to be corrupted! So it was back to the beginning with a fresh JetPack install and what I hoped was a decent grasp on all the steps I needed to replicate the previous set up... Turns out I could go from the fresh OS image to fully ready for MinION sequencing in roughly 20 mins, so that's a win. It's also spurred me to structure the process in terms of ordered steps below, mainly so I can use them as decent notes to continue to flesh out a nice user friendly set up guide.

^ a nice feature of the Xavier NX is the fact that you can easily have multiple sd cards with JetPack OS installed and ready to go, meaning you don't have to fuss around with the Nvidia SDK.

The process...

So in light of the above, while I was going through the procedure of starting from scratch, I quickly constructed the steps below.

WARNING - Now I want to stress that there is a lot still missing in terms of details below. Most of the commands might not work on other systems (i.e. working dirs, paths, etc.), but the general order of steps and what is being performed is what is needed to get up and running. There are a few steps that can be shuffled about, but currently the below order makes sense and works nicely, at least in my hands on a Xavier NX. I imagine that the same procedure will work on the Xavier AGX, but I've yet to test that. Additionally, step 1. is still a work in progress. What I plan to develop is this GitHub repository that contains the required files and some small scripts to aid the set up process. This will contain things like the pre-compiled h5py library, ONT repository list and minknow.service file, BUT it won't contain actual ONT binaries. Where able I will try and provide the easiest solution to obtain things such as the correct version of guppy.

OK, so here it is as it currently stands:

# 0. pre-setup
# fresh inistall of Nvidia Jetson JetPack
# create user: minit with pw: minit


# 1. get everything together before we start
# ensure in /home/minit dir
cd ~/
# clone repo
git clone https://github.com/sirselim/jetson_nanopore_sequencing.git
cd ./jetson_nanopore_sequencing
# download guppy from ONT servers
wget https://mirror.oxfordnanoportal.com/software/analysis/ont-guppy_4.0.15_linuxaarch64_cuda10.tar.gz
# extract required packages
tar -xzvf ont-guppy_4.0.15_linuxaarch64_cuda10.tar.gz
unzip ./libs/python3.7/h5py.zip 


# 2. full system update and upgrade
# shouldn't take long with latest JetPack
sudo apt update
sudo apt upgrade


# 3. mount ssd (this should be properly done, place holder for now)
sudo mkdir /xavier_ssd
sudo mkdir /xavier_ssd/minknow
sudo mount /dev/nvme0n1p1 /xavier_ssd/


# 4. add nanopore repo to system and update
sudo cp nanoporetech.list /etc/apt/sources.list.d/
sudo apt update


# 5. install ont packages
# 6 packages required:
# - ont-bream4-minit 
# - ont-kingfisher-ui-minit 
# - ont-remote-support 
# - ont-system-identification 
# - ont-configuration-customer-minit 
# - minknow-core-minit-offline
# BUT we can use the predefined list of packages
cat ont-package-list-xavier.txt | xargs sudo apt-get install


# 6. copy guppy to correct location, create sym links and check links and guppy version
sudo cp -r ont-guppy /opt/ont/
sudo ln -s /opt/ont/ont-guppy/bin/guppy_basecaller /usr/bin/guppy_basecaller
sudo ln -s /opt/ont/ont-guppy/bin/guppy_basecall_server /usr/bin/guppy_basecall_server
sudo ln -s /opt/ont/ont-guppy/bin/guppy_barcoder /usr/bin/guppy_barcoder
sudo ln -s /opt/ont/ont-guppy/bin/guppy_aligner /usr/bin/guppy_aligner
guppy_basecaller --version
guppy_basecall_server --version
guppy_aligner --version
guppy_barcoder --version


# 7. edit the minknow conf files to change the location of data and check guppy paths and parameters
cd /opt/ont/minknow/conf
sudo nano user_conf 
sudo nano app_conf


# 8. install libhdf5-dev and copy across pre-compiled h5py (after removing old version)
# move back to git repo dir 
cd ~/jetson_nanopore_sequencing
sudo apt install libhdf5-dev
sudo rm -rf /opt/ont/minknow/ont-python/lib/python3.7/site-packages/h5py/
sudo cp -r h5py/ /opt/ont/minknow/ont-python/lib/python3.7/site-packages/


# 9. copy minknow.service for systemd set up and restart the service
sudo cp minknow.service /etc/systemd/system/
service minknow restart
# check
service minknow status
# can check that the services are running as root with htop

Hopefully that is helpful to some. It's really helpful for me to have it out of my head as well as not being spread across multiple posts. :)

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Oct 30, 2020

Update 30th October 2020 - working touch screen... no thanks to Unity!

So that was frustrating! After pouring over logs, testing many many different things, and walking away from it for a few hours, I finally got to the bottom of the touch screen issues on the Jetson Xavier NX. Before I talk about what was going on here are a few pics of the 'new' interface (the observant will know the solution before reading below...):

image
image
image
image
image
image

Ahh, full screen MinKnow with a pop up keyboard (onboard) - nice! Apologies for the poor lighting in the above, a bright screen in a dark environment is never a good combination.

The problem?

So to get to this stage we have to step back a little. At the end of the day yesterday I was convinced it was a power issue over the usb port(s). So this morning I tested the screen on my Linux laptop (Dell Precision 5520) and a Raspberry Pi 3, both worked perfectly plug'n'play. OK, so that meant that there was nothing wrong with the touch screen, and if the Pi could power it then the Xavier should easily do so. Back to the drawing board.

A solution

I was then delving into syslog and Xorg log files and noticed that udev was always unloading the libinput module after login. Turning to Google with the various messages from the log files wasn't really proving fruitful, until I stumbled across this post from YEARS ago: https://bugs.launchpad.net/ubuntu/+source/unity/+bug/1099289#24

Huh, in 2013 there was a bug in Unity that seemed to be causing some issues with various connected devices after logging in. A quick install of Xfce4 later and everything is running perfectly, thanks Unity!

So looks like we're ditching that. An added benefit is that by not running Unity as the desktop environment things are quite a bit snappier. Whether I stick with Xfce or 'shop' around a little is yet to be seen, but it is very nice to have a fully functioning touch panel and a nice, fairly user friendly environment.

Virtual keyboards

The onboard virtual keyboard is working really well, including automatic pop-ups. The one down side is that if you put an app like Chrome or MinKnow into true full screen mode, the onboard button gets lost underneath. This is again a known and long standing bug, but not from the onboard side. Sigh. I might try a few different virtual keyboards, but at this stage I am extremely happy with the set up. Hopefully we can get it doing some more sequencing next week.

@thor-johnson

This comment has been minimized.

Copy link

@thor-johnson thor-johnson commented Nov 2, 2020

@sirselim thanks very much for steering me to kingfisher ui. I had followed a path that installed a directory, opt/ont/minknow-ui from another package (static frontend?) but could not get the minion to be recognized as a connection. With the instructions you provided above, I have now installled, setup and have it running on Xavier NX. I will post those minor tweaks.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Nov 2, 2020

@thor-johnson - fantastic news! I'm glad that the basic set of steps was useful. As I mentioned earlier, there is a fair bit of conflicting information out there, including in this discussion thread, that comes from the collective effort of getting things optimised. Now I think we're honing in on a robust approach (for the current software stack at least). So in line with this I've started to put some more effort into the GitHub repository, here.

If anyone would like to contribute it is much appreciated. Please feel to contribute anything that you think might be useful either as issues, pull requests or even comments here. Awesome work team!

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Nov 2, 2020

Slight aside from the usual content

In terms of Xavier AGX and GPUs, I alluded to the Nvidia Calra AGX that's under development in a previous post. I just got around to going through my notes from the GTC conference and decided to post a question back to the Nvidia forums to see if there is any updates. I've added the post below for those that are interested. (here's the post - link)


Is there any update on this topic? I ask in light of watching the Clara AGX talks at GTC last month.

I got quite excited and took a few notes, here's one in particular that struck a chord with me in regards to Xavier, full PCIe cards and a potential interface (sorry for my rough diagram, I was trying to listen and jot down notes):
image

I didn't get my question answered, so I'm still unsure if the RTX6000 is interfacing through the Xavier PCIe or if it's some other solution. But a little more insight would be awesome if it can be shared.

For those interested there is a little bit of information here - it's still in beta so info is light.

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Nov 6, 2020

I am still experincing some sort of permission error after rebooting the Xaver board, even though my service startup file now looks like this

user@meqneuropat08:~$ cat /lib/systemd/system/minknow.service
[Unit]
Description=MinKNOW Instrument Software for MinIT (daemon)

[Service]
ExecStart=/opt/ont/minknow/bin/mk_manager_svc
WorkingDirectory=/opt/ont/minknow
KillMode=mixed
User=root
Group=root
SyslogIdentifier=minknow
LogsDirectory=minknow
LimitCORE=infinity
LimitNICE=40
NoNewPrivileges=true
ExecStartPre=/bin/sleep 15
CPUQuota=300%


[Install]
WantedBy=multi-user.target

sudo service minknow status returns that the service is dead, which is correct. As soon as I do sudo service minkow start after the GUI shows up, everything works fine. It seems that on my system, I am having two problems: one has been solved - it's the permission issue that changing minit to root has eliminated. The other issue seems to be something during the boot. Somehow, the service dies.

I also have still no success towards pyguppy, I tried a couple of things meanwhile. It might be possible to build a workaround using a RAMdisk directory and polling that with gruppy. The results may be similar to what pyguppy is supposed to do.
https://pypi.org/project/ont-pyguppy-client-lib/4.2.2/ won't install on arm64, as we already found out :(

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Nov 30, 2020

Adaptive sampling might move a step closer soon. We have now 3 AGXs in Basel and that allowed me to start out with a "clean" device, ubuntu 18.04 jetpack.

After installing the MinKNOW kingfisher UI as described above, I noticed that the adaptive sampling dialog has now been included (screenshot below); this is version MinKnow 4.1.22. It is the release that has been online for short while only. Still, the MinIT bionic repo does not contain any files, so I installed the Xenial versions. I also manually installed guppy 4.0.14 and tried to replicate live basecalling. It failed; this time bream (in /var/log/minknow/MINION-ID/bream...-0.log) had error messages that arose from guppy communication.

I then checked on an x86_64 PC (Xenial, 16.04) which I prepared with the current MinKNOW and Cuda 10 GPU enabled, and found that the version of Guppy coming with the current MinKNOW release is 4.2.x, i.e. it has the new separation into standalone basecaller, server, and client. This leads to different command line argument and executable layout, which is incompatible with older releases of guppy.

I stopped the MinkKNOW service (sudo service minknow stop). I then deleted guppy 4.0.14 and copied 4.2.2 over it (as root). I then restarted the MinKNOW service (sudo service minknow start). I verified that the processes are running, in particular, guppy_basecall_server, with the GPU-enabled cmd line argument. This was the case. I then set out for a playback run (MIN-106) and bingo - live basecalling now works again! I am getting pretty excited; will now head home, but will try what happens when I turn on adaptive sampling on the playback run. Maybe there is working adaptive sampling software in the current release or, at least, some of it.

pastedImage

I will keep posting the results here.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Dec 1, 2020

Update 1st December 2020

Awesome detective work @juhench! I was exploring the newer MinKnow and Kingfisher packages but gave up when they spat errors, nice that you've done the hard work here. :)

I've started a little ARM-based (MinIT focused at this stage, but probably holds true for other archs) software compatibility table, here.

MinKnow Core version Kingfisher UI version Guppy version working
4.0.5 4.0.21 4.0.11, 4.0.14, 4.0.15
4.1.2 4.1.22 4.2.2, 4.2.3

Also, a little while ago I also minted that repo with a DOI in case we want to start citing it as a resource: https://github.com/sirselim/jetson_nanopore_sequencing

Adaptive Sequencing

In terms of adaptive sequencing, I can confirm the same as you @juhench, it is now incorporated into the Kingfisher UI (alongside samplesheets and the ability to interact with MinKnow from Android/iOS devices - more on this below).

I have tried a couple of different approaches to adaptive sequencing using some simulation data. While everything is 'working' (sequencing, live-base calling using Guppy 4.2.3) it doesn't appear that enrichment or depletion is occurring. This is in line with what I was observing when I was trying to get both ReadUntil and Readfish working on both Xavier NX and AGX.

Definitely exciting that the options are arising, but I think there might be a bit more work needed to determine if this is something that is going to run on these Jetson boards. Apart from a tweet from Clive Brown alluding to adaptive sequencing on Mk1c, I have yet to see anything confirmed. If we see it working on Mk1c it will easily work on the Xaviers. I've heard from several people extremely well versed in the adaptive sequencing 'world' (Matt Loose being one of them) that they have had zero success with the minIT and Mk1c - but lets hold out hope and keep trying!

I've got a few more tests to run before I report back, but I can make a PSA that one should not provide the human genome as a reference to MinKnow... that runs the system out of RAM in seconds! I'm sure I read somewhere on the ONT forum that it's really only suited to smaller reference genomes or targeted regions, but I obviously forgot. Don't make that mistake. :) I've also noticed that the Xavier AGX isn't able to keep up with basecalling, even after optimising - this isn't going to be conducive to good adaptive sampling.

MinKnow on Android/iOS

Another new feature that comes with the latest MinKnow is the ability to be able to access running services from Android/iOS devices. You can find out more here and download the app.

I've had a play and it's great, here are a few shots:
image
image
image

A really cool and useful feature that we will be using a lot.

Samplesheets in MinKnow

This is exciting if you are potentially running multiple MinIONs. You can now fill out all required run information in a samplesheet and upload to MinKnow. I haven't tested this yet but it's a nice feature.


Edit: so it may be sort of working... I was trying to enrich for chr21 and chr22. The highest mean number of reads was for chr21, but it's not great across the board. We might call this adaptive sequencing kind of working - it's a start!

contig number sum min max std mean median N50
chr1 314 4620764 218 391059 35161 14,716 2166 75415
chr2 314 4445177 222 288636 33725 14,157 1796 82616
chr3 252 3890723 200 293666 36823 15,439 3456 80439
chr4 236 4587572 206 257351 40414 19,439 2602 86265
chr5 246 4709994 242 367721 43713 19,146 2652 108991
chr6 224 2687510 235 149167 23796 11,998 2489 63648
chr7 218 3211821 138 317145 36828 14,733 1274 74378
chr8 179 2709674 216 327988 40578 15,138 3285 57398
chr9 179 3360063 269 245513 40120 18,771 4818 82520
chr10 177 2923455 241 287256 36791 16,517 1306 74566
chr11 142 2396419 292 304231 37866 16,876 4950 64810
chr12 174 2283273 264 145604 25030 13,122 2868 60625
chr13 84 1132072 220 204096 30736 13,477 1655 60591
chr14 99 2149931 276 151188 37798 21,716 5894 101773
chr15 172 2366617 259 187957 30379 13,759 1292 69523
chr16 130 1615172 244 239234 27986 12,424 1146 63205
chr17 93 1500748 267 398833 48521 16,137 2230 87689
chr18 94 1673496 276 377165 45158 17,803 1306 80974
chr19 102 1734998 256 215416 38318 17,010 4747 104383
chr20 88 1783803 138 209664 40224 20,270 1320 89407
chr21 37 1209274 308 305186 64391 32,683 6672 158897
chr22 42 279471 240 36666 8422 6,654 3280 13684
chrM 34 188797 272 16556 5973 5,553 1618 12658
chrX 204 3168832 209 250428 37425 15,533 1470 101809
@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Dec 1, 2020

@sirselim πŸ‘ great job; now you did it before me! Great.

I can confirm your results and have identified two processes that are actually taking up a processor core one each: a process called "control_server / control_main" and "read_until_script.py". Given the so-so CPU performance of the AGX, it might be worthwhile to use a laptop computer in conjunction with the Xavier and, perhaps, run the alignment with GASAL on the AGX's GPU. There should be a sufficient amount of memory. I will also try to optimize the processes - now that it is "kind of" working. I think at this point, thanks also go to ONT for a) integrating adaptive sequencing nicely and b) for also including everything in the MinIT / ARM64 version.

It seems to be a valid strategy to have a GPU-equipped PC running Xenial as a reference system where installs work out of the box. From there, one can figure out what does not install due to Xenial/Bionic incompatibilities. Recall that AGX is running Ubuntu 18.04 and the MinIT Ubuntu 16.04.

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Dec 2, 2020

Good news: It turns out - the Xavier AGX can do adaptive sampling and performance against the human genome reference is comparable to an i7 system running at 3.8 GHz (with network GPU basecalling via port forwarding).

Utilization is about 50% of everything if the cores run all at 2.3 GHz.

I will post a detailed set of instructions on how to configure it. The reason it does not work well is actually playback mode: Here, the process called "control_server / control_main" runs at 100% CPU while it consumes about 50% with a new run (fresh MIN-106).

Hurra!

The screenshots with the old MinKNOW software is the same sample on the flow cell but just run before the read-until run. I am targeting a single gene and will check whether I got it enriched to maybe a single read. This was a 1h run only. Note the difference in read length histograms. The same experiment was also possible in playback mode on the i7, but not the AGX (due to lacking of CPU clock freq). Overall, it does not matter and we still got about 50% of CPU/GPU leftover for other analyses.

Screenshot_2020-12-02_16-54-23
Screenshot_2020-12-02_16-53-48
Screenshot_2020-12-02_16-53-27
Screenshot_2020-12-02_16-52-56
Screenshot_2020-12-02_16-48-39
Screenshot_2020-12-02_16-48-14
Screenshot_2020-12-02_16-47-52
Screenshot_2020-12-02_16-47-19
Screenshot_2020-12-02_16-46-53
Screenshot_2020-12-02_16-46-05

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Dec 2, 2020

Amazing work @juhench, congratulations!! Very exciting to see this working. I was wondering what the impact of playback mode was, knowing that it was not a direct comparison, but having this evidence is fantastic.

I look forward to your write up.

P.S. I see you have the new 32Gb Xavier AGX, very nice. We need to get a couple of those.

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Dec 2, 2020

Here is the result: 2 long reads on target after 1 hour of sequencing for the proof-of-principle experiment, actually at the beginning of the target region:

chr10:129467190-129770983 (in the minimap reference I used). I will need to play a bit with the settings and run more samples. It seems that adaptive sampling can even work with our rather short biopsy DNA (1-2 kb pieces). In addition, routine methylome analysis using the remaining short (~500bp) reads (using P. Euskirchen's NanoDx pipeline) keeps working like a charm. Short reads all across the genome improve copy number profile resolution a lot; what a nice side effect for my purpose.

Screenshot_2020-12-02_22-52-14

Thank you all for this wonderful collaboration and, in particular, @sirselim, for setting this up.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Dec 2, 2020

@juhench Very nice, very nice indeed! Exciting times all round. :)

Thank you all for this wonderful collaboration and, in particular, @sirselim, for setting this up.

No thanks necessary, but you're most welcome. A massive thanks to you @juhench and the rest of the community contributors - this has been an amazing demonstration of what happens when we can all share experiences and work together.

We've got a LOT more going on in this space, so I'm hoping to keep this discussion active and I'm always looking forward to posts from other as well.

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Dec 3, 2020

It can be replicated (I got comparable results with 2 more runs). I now adjusted the target region and will need to see what works best for us. Still the same library from yesterday. It runs like a charm, with very little pore loss in 5 hours of accumulated run time. Even a partial 2x coverage for short reads (ok, that might just be by chance). But there are several full-length reads (2-5 KB) on target while the majority of other reads (like 99.9%) have been truncated. I will need to adapt the sampling range even closer to my target. Let's see.

Screenshot_2020-12-03_17-47-31

@lfaino

This comment has been minimized.

Copy link

@lfaino lfaino commented Dec 4, 2020

Hi All,

I have a question. Finally, I got the developer agreement license signed and i got access to the Nanopore github branch. I was trying to install guppy on a Jetson Nano but if fails and i can not understand why. Does anyone want to help?

thanks
Luigi

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Dec 4, 2020

Hi @lfaino,

I just checked the memory consumption of guppy on our AGX and that this might be a key problem. Have you received sources to compile guppy yourself? In order to run the current (4.2.2) version of guppy, more than 4GB RAM (currently, 15GB) are being consumed for guppy during a regular read-until run on the Jetson Xavier AGX. I have been trying to run guppy on an RTX 2070 GPU in an x86_64 environment alongside other GPU applications I created with pycuda and had only a couple of GB GPU RAM left. It was impossible to run guppy in parallel if there was not enough RAM. I am furthermore not sure whether the number of GPU cores on the Nano would be sufficient to do anything useful in terms of basecalling.

The other key reason that I think, makes it impossible to even compile guppy on the Nano is its compute capability (in case you would work around all memory issues) - I recall guppy requires >=6, the nano is 5.2 (https://developer.nvidia.com/cuda-gpus). I ran into the same problems when joining this gist - I wanted to run guppy on google collaboratory, which failed at exactly this point.

While I agree with you that it would be a dream to have a running setup on a jetson nano (by stripping out everything not essentially required), I think that at the current level of complexity (of the ONT platform), the nano is not an option. I could only envision functionality on a device like the nano if there was, e.g., an FPGA-addon that would compensate for some of the heavy lifting (like an FPGA-based neural network for basecalling).... just dreaming.

Given the current CPU/GPU/RAM requirements, I can only suggest obtaining an AGX device. It has been confirmed to work and it has the capacity for additions, such as further downstream data analysis.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Dec 4, 2020

Hi Luigi,

I'm not part of the developer agreement, my institutes legal department weren't happy to sign it unfortunately. So I'm unable to comment, however @juhench might have some insight.

I was initially interested in guppy on the Nano but the incompatible cuda compute put a stop to that. I think it's going to be more than a recompile of the source, you'll probably have to switch in some alternative libraries (I remember someone mentioning pytorch). How have you tried to get it working?

@lfaino

This comment has been minimized.

Copy link

@lfaino lfaino commented Dec 4, 2020

Hi,
i do agree that basecalling can not be done on a regular flowcell, but what about flongle? I think that if we can manage the error with memory, ampliseq on a flongle can be managed.

Cheers
Luigi

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Dec 4, 2020

Yes I had often wondered if it would work alongside a flongle. Have you tried running minknow by itself, without guppy? I imagine everything should install, but @juhench is right the memory reserved by minknow service is over 3gb on a Jetson xavier NX. That's not leaving much at all for gpu basecalling.

One thought I just had, what about CPU base calling?

Edit/update: I'm going to be submitting an issue to the Readfish-CPU repository next week to start working on arm issues with the CPU basecaller. We're not sure if it's going to work but it'll be interesting. The CPU on the Nano will likely be too busy running minknow and not have enough cores to dedicate to calling (or raw speed). Still an interesting experiment.

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Dec 4, 2020

@sirselim, @lfaino
Sure, CPU-based basecalling can be done for a flongle. But then, I see no point in using a Jetson Nano (since you are not using the GPU). CPU basecalling could negatively affect the responsiveness of the system if running "live". Why not use any old laptop computer? We have successfully worked with flongle and CPU basecalling before we switched to GPUs in early 2019. One older multicore x68_64 CPU was kind of able to keep up with guppy CPU basecalling. You would have to make sure (with the nano) that guppy is not using more than 2 cores. One remains for the system and one for "analyzer_main, control_main", etc.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Dec 4, 2020

@juhench yep, I completely take your point and sure an old laptop probably is fine. But that old laptop isn't super portable, power friendly or cheap ($99 Nano). Your use case I completely agree with in a lab environment, but imagine being able to supply extremely cheap, portable and accessible devices into local communities and developing countries. That's a big passion of mine and while the Xavier is the sweet spot I believe (the NX is incredible for the price), I really would love to see the Nano have a role. Maybe that is running Readfish-CPU to enrich for a particular virus/psthogen in the field on a flongle, which you then gpu base call after the run (shut down minknow and then use guppy/bonito). I might be dreaming, but it's a fun dream. πŸ˜‰

@lfaino

This comment has been minimized.

Copy link

@lfaino lfaino commented Dec 4, 2020

I agree with sirselim. This is my quest as well.
I want to use GPU based guppy basecalling for this i need some help. however, bonito can be a good idea as well.

I can try

@lfaino

This comment has been minimized.

Copy link

@lfaino lfaino commented Dec 4, 2020

To make the dream of Sirselim, I tested the miniPCR of URS (https://forum.openhardware.science/t/pocketpcr-low-cost-usb-powered-open-source-pcr/2187) and it can be used to prepare the library with RAP. all these devices can be run on a car battery...

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Dec 4, 2020

@lfaino and @sirselim,
that is interesting indeed. I recall the old versions of guppy (that are able to identify barcodes just fine) used much less RAM (around 1.7GB) and if you had access to the sources it might be possible to compile them on a nano. It should suffice for adaptive sampling. There is also GASAL2, which should (I have not tried yet) make use of the GPU. Using the ARM CPU cores as little as possible will probably lead to success. The MinKNOW GUI eats a lot but of resources but can be shut down. Maybe I will get a nano as well - not you both got me really interested. I will ask my wife for another X-mas present :). Is it possible to get guppy source code through the developer channel? I did not find it there. Maybe adapting an older version of guppy to run on older compute platforms is interesting (also, perhaps, google collaborator - another thing that could be useful for those now having a lot of resources at hand).

@lfaino

This comment has been minimized.

Copy link

@lfaino lfaino commented Dec 6, 2020

Hi Guys,

btw, my jetson has 4GB of ram...

Cheers
Luigi

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Dec 7, 2020

Hi @lfaino,

Yep, I assumed you'd be using the 4Gb model. The new 2Gb Nano's would really be strecthing it!

I'm trying to set up MinKnow on one of our Nano's for fun this afternoon. ;)

  • Miles
@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Dec 7, 2020

@juhench and @lfaino - check this out: link. The upcoming Jetson Mate - a unit that can house up to 4 Nano/Xavier NX modules (it can run on combinations from 1 - 4 installed). Populated with 4x Xavier NX that would in theory give you access to:

  • 24 ARM cores (4*6 cores)
  • 32Gb RAM (4*8Gb)
  • 1536 cuda cores (4*384)
  • 192 tensor cores (4*48 tensor cores)

If you populated with 4 Nano modules:

  • 16 ARM cores (4*4 cores)
  • 16Gb RAM (4*4Gb)
  • 512 cuda cores (4*128)

You can also mix and match modules, so you could have a Xavier NX as a 'main' module and then 3 worker Nano's.

All this is powered of a single 65W USB-C power adapter, so pretty accessible laptop level power bricks would work I imagine.

Talking to a few people it looks like the carrier board (Jetson Mate) will be released at ~$200 USD. So we can do some rough calculations here:

  • for a fully populated Xavier NX set up

    • (4 * $399) + $200 = $1796 USD
  • for a fully populated Nano set up

    • (4 * $99) + $200 = $596 USD
  • for a mixed Xavier NX / Nano set up

    • ((1 * $399) + (3 * $99)) + $200 = $896 USD

There are obviously lots of questions around whether this set up would lend itself to the workflows we're interested in, but I'm very interested to see and might try and get my hands on one when they are released.

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Dec 7, 2020

This is like a dream - but obviously becoming reality. Btw. I can confirm that adaptive sampling is working also with huge amounts of (small) targets (>400K!) on the AGX. CPU is at about 50% in that case and RAM is at 23GB (all included, with GUI Desktop). Thank you so much for sharing! I can also confirm (from the PC side) that through port forwarding it is very well possible to basecall on another computer making MinKNOW think it was calling locally on port 5555. It eliminates the need for reconfiguring MinKnow. Such a desktop blade server makes perfect sense.

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Dec 7, 2020

I can confirm that adaptive sampling is working also with huge amounts of (small) targets (>400K!) on the AGX. CPU is at about 50% in that case and RAM is at 23GB (all included, with GUI Desktop).

Fantastic news! :) We're hoping to get some 'real live' bacterial mixtures prepared in the next week or so, and then do some adaptive sampling testing with the AGX and NX - I'm interested to see if we are limited by the 16Gb of RAM on our AGX's. We won't be using human reference genome in this instance so we might be OK. We have a couple of the 32Gb models about to be ordered, but I'm going to keep pushing the 16Gb AGX and 8Gb NX boards to see exactly how much they can handle.

I can also confirm (from the PC side) that through port forwarding it is very well possible to basecall on another computer making MinKNOW think it was calling locally on port 5555. It eliminates the need for reconfiguring MinKnow. Such a desktop blade server makes perfect sense.

Exactly what I was thinking. There are a lot of possibilities for little clusters like this. I'm going to dub it the 'mini GridION' as a version with 4 NX modules would easily live base call 4 MinIONs at once, for less than $2000 USD! I'm very excited. You could run MinKnow on all modules and then just link them into the version running on the main board, so you'd easily be able to control it from a single display/VNC.

@lfaino

This comment has been minimized.

Copy link

@lfaino lfaino commented Dec 8, 2020

Hi Guys,

it works.... Guppy on jetson nano. I will check if MinKnow will start tomorrow

NanoJetson

NanoJetson1

@juhench

This comment has been minimized.

Copy link

@juhench juhench commented Dec 8, 2020

@lfaino: Congratulations! How did you overcome the problem with the compute version incompatibility? This is amazing! I guess this will do for a Flongle, as you initially suggested. I take back all I wrote earlier. May I ask which release of guppy you installed?

@sirselim

This comment has been minimized.

Copy link
Owner Author

@sirselim sirselim commented Dec 8, 2020

Yes, I reiterate the congratulations!

@juhench on twitter @lfaino included another screenshot from jtop. It shows RAM usage at ~3.7Gb for the system. I suggested dropping Unity and gdm3 if he hasn't already, as this can easily free up 1Gb of memory. This 'might' give the system some overhead to run Minknow...?

I'm also very interested to hear how this was achieved and what version of Guppy is being used. πŸ™‚

Edit: source for the above here.

@lfaino

This comment has been minimized.

Copy link

@lfaino lfaino commented Dec 8, 2020

the compute, I asked Nanopore and they pointed out where to change.

I compiled the 4.2.2 but it is roller coaster using 2 version of GCC and a version of rapidjson from another tool.

I can try to document on this page but I do not know if i can because the develop agreement

@lfaino

This comment has been minimized.

Copy link

@lfaino lfaino commented Dec 8, 2020

@sirselim, if you tell me how to do, I can try. I'm new to this computers and I do not know how to do what you suggested.

@sirselim