a collection of my notes while working on nanopore basecalling on the Jetson Xavier

basecalling_notes.md

Jetson Xavier basecalling notes

initial basecalling runs

'fast' flip-flop calling on the Jetson Xavier

guppy_basecaller --disable_pings --compress_fastq -c dna_r9.4.1_450bps_fast.cfg -i flongle_fast5_pass/ -s flongle_test2 -x 'auto' --recursive 

high-accuracy calling with base modifications on the Jetson Xavier

guppy_basecaller --disable_pings --compress_fastq -c dna_r9.4.1_450bps_modbases_dam-dcm-cpg_hac.cfg --fast5_out -i flongle_fast5_pass/ -s flongle_hac_fastq -x 'auto' --recursive 

$ guppy_basecaller --compress_fastq -c dna_r9.4.1_450bps_modbases_dam-dcm-cpg_hac.cfg -i flongle_fast5_pass/ -s flongle_hac_fastq -x 'auto' --recursive
ONT Guppy basecalling software version 3.4.1+213a60d0
config file:        /opt/ont/guppy/data/dna_r9.4.1_450bps_modbases_dam-dcm-cpg_hac.cfg
model file:         /opt/ont/guppy/data/template_r9.4.1_450bps_modbases_dam-dcm-cpg_hac.jsn
input path:         flongle_fast5_pass/
save path:          flongle_hac_fastq
chunk size:         1000
chunks per runner:  512
records per file:   4000
fastq compression:  ON
num basecallers:    1
gpu device:         auto
kernel path:
runners per device: 4

Found 105 fast5 files to process.
Init time: 2790 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 2493578 ms, Samples called: 3970728746, samples/s: 1.59238e+06
Finishing up any open output files.
Basecalling completed successfully.

So from the above we see in high accuracy mode it take the Xavier ~41 minutes to complete the base calling using the default configuration files. For reference the fast calling mode was ~8 minutes.

optimising settings for Jetson Xavier

• When performing GPU basecalling there is always one CPU support thread per GPU caller, so the number of callers (--num_callers) dictates the maximum number of CPU threads used.
• Max chunks per runner (--chunks_per_runner): The maximum number of chunks which can be submitted to a single neural network runner before it starts computation. Increasing this figure will increase GPU basecalling performance when it is enabled.
• Number of GPU runners per device (--gpu_runners_per_device): The number of neural network runners to create per CUDA device. Increasing this number may improve performance on GPUs with a large number of compute cores, but will increase GPU memory use. This option only affects GPU calling.

There is a rough equation to estimate amount of ram:

runners * chunks_per_runner * chunk_size < 100000 * [max GPU memory in GB]

For example, a GPU with 8 GB of memory would require:

runners * chunks_per_runner * chunk_size < 800000

some suggested settings from ONT

NVIDIA Jetson TX2

--num_callers 1
--gpu_runners_per_device 2
--chunks_per_runner 48

from hac config file (dna_r9.4.1_450bps_modbases_dam-dcm-cpg_hac.cfg)

chunk_size                          = 1000
gpu_runners_per_device              = 4
chunks_per_runner                   = 512
chunks_per_caller                   = 10000

modified testing

'fast' flip-flop calling on the Jetson Xavier

guppy_basecaller --disable_pings --compress_fastq -c dna_r9.4.1_450bps_fast.cfg -i flongle_fast5_pass/ \
  -s flongle_test2 -x 'auto' --recursive --num_callers 4 --gpu_runners_per_device 8 --chunks_per_runner 256

$ guppy_basecaller --disable_pings --compress_fastq -c dna_r9.4.1_450bps_fast.cfg \
-i flongle_fast5_pass/ -s flongle_test2 -x 'auto' --recursive --num_callers 4 \
--gpu_runners_per_device 8 --chunks_per_runner 256
ONT Guppy basecalling software version 3.4.1+213a60d0
config file:        /opt/ont/guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /opt/ont/guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass/
save path:          flongle_test2
chunk size:         1000
chunks per runner:  256
records per file:   4000
fastq compression:  ON
num basecallers:    4
gpu device:         auto
kernel path:
runners per device: 8

Found 105 fast5 files to process.
Init time: 880 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 428745 ms, Samples called: 3970269916, samples/s: 9.26021e+06
Finishing up any open output files.
Basecalling completed successfully.

I was able to shave a minute off the fast model on the Xavier (above) getting it down to ~7 minutes.

Update: (13th Dec 2019)

Just modifying the number of chunks per runner has allowed me to get the time down to under 6.5 mins (see table below).

chunks_per_runner	time
(160) default	~8 mins
256	7 mins 6 secs
512	6 mins 28 secs
1024	6 min 23 secs

It looks like we might have reached an optimal point here. Next I'll test some of the other parameters and see if we can speed this up further.

high-accuracy calling with base modifications on the Jetson Xavier

guppy_basecaller --disable_pings --compress_fastq -c dna_r9.4.1_450bps_modbases_dam-dcm-cpg_hac.cfg \
  --num_callers 4 --gpu_runners_per_device 8 --fast5_out -i flongle_fast5_pass/ \
  -s flongle_hac_basemod_fastq -x 'auto' --recursive

increased number of callers

$ guppy_basecaller --disable_pings --compress_fastq -c dna_r9.4.1_450bps_fast.cfg \
  -i flongle_fast5_pass/ -s flongle_test2 -x 'auto' --recursive --num_callers 8 \
  --gpu_runners_per_device 8 --chunks_per_runner 1024
ONT Guppy basecalling software version 3.4.1+213a60d0
config file:        /opt/ont/guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /opt/ont/guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass/
save path:          flongle_test2
chunk size:         1000
chunks per runner:  1024
records per file:   4000
fastq compression:  ON
num basecallers:    8 
gpu device:         auto
kernel path:
runners per device: 8 

Found 105 fast5 files to process.
Init time: 897 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 383865 ms, Samples called: 3970269916, samples/s: 1.03429e+07
Finishing up any open output files.
Basecalling completed successfully.

increased chunk size

$ guppy_basecaller --disable_pings --compress_fastq -c dna_r9.4.1_450bps_fast.cfg \
  -i flongle_fast5_pass/ -s flongle_test2 -x 'auto' --recursive --num_callers 4 \
  --gpu_runners_per_device 8 --chunks_per_runner 1024 --chunk_size 2000
ONT Guppy basecalling software version 3.4.1+213a60d0
config file:        /opt/ont/guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /opt/ont/guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass/
save path:          flongle_test2
chunk size:         2000
chunks per runner:  1024
records per file:   4000
fastq compression:  ON
num basecallers:    4
gpu device:         auto
kernel path:
runners per device: 8

Found 105 fast5 files to process.
Init time: 1180 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 503532 ms, Samples called: 3970269916, samples/s: 7.88484e+06
Finishing up any open output files.
Basecalling completed successfully.

increased runners per device and number of callers

$ guppy_basecaller --disable_pings --compress_fastq -c dna_r9.4.1_450bps_fast.cfg \
  -i flongle_fast5_pass/ -s flongle_test2 -x 'auto' --recursive --num_callers 8 \
  --gpu_runners_per_device 16 --chunks_per_runner 1024 --chunk_size 1000
ONT Guppy basecalling software version 3.4.1+213a60d0
config file:        /opt/ont/guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /opt/ont/guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass/
save path:          flongle_test2
chunk size:         1000
chunks per runner:  1024
records per file:   4000
fastq compression:  ON
num basecallers:    8
gpu device:         auto
kernel path:
runners per device: 16

Found 105 fast5 files to process.
Init time: 1113 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 383466 ms, Samples called: 3970269916, samples/s: 1.03536e+07
Finishing up any open output files.
Basecalling completed successfully.

current 'optimal' parameters

The below parameters seem to provide the 'optimal' speed increase with a resultant run time of 6 mins and 23 secs.

$ guppy_basecaller --disable_pings --compress_fastq -c dna_r9.4.1_450bps_fast.cfg \
  -i flongle_fast5_pass/ -s flongle_test2 -x 'auto' --recursive --num_callers 4 \
  --gpu_runners_per_device 8 --chunks_per_runner 1024 --chunk_size 1000
ONT Guppy basecalling software version 3.4.1+213a60d0
config file:        /opt/ont/guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /opt/ont/guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass/
save path:          flongle_test2
chunk size:         1000
chunks per runner:  1024
records per file:   4000
fastq compression:  ON
num basecallers:    4
gpu device:         auto
kernel path:
runners per device: 8

Found 105 fast5 files to process.
Init time: 926 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 382714 ms, Samples called: 3970269916, samples/s: 1.0374e+07
Finishing up any open output files.
Basecalling completed successfully.

---

exploring portable batteries and power modes

We are currently using a 27000mAh AC Portable Charger from Ravpower.

Below: Ravpower Xtreme Series 27000mAh AC Portable Charger This battery bank/charger has a built in 220V AC outlet and 1 usb-c and 2 usb 3.1 outputs.

Below: powerbank charging from the wall. Ravpower claim this powerbank will charge a smartphone 11 times, a tablet 4 times or a laptop 3 times.

Below: running our first portable Xavier GPU basecalling of nanopore data!

changing power modes

Running the Xavier in different power states obviously influences the amount of run time on the battery.

Power mode	Time
10W	33.4 mins
15W	14.3 mins
30W 2 cores	10.8 mins
30W 4 cores	10.8 mins
30W MAX (8 cores)	7.5 mins

The above benchmarks were performed on data generated from a flongle run (~0.5 Mb of sequence or 5.5 Gb of actual data).

---

potential V100 examples

V100 config example for high accuracy model

guppy_basecaller \
--disable_pings \
--compress_fastq \
-c dna_r9.4.1_450bps_modbases_dam-dcm-cpg_hac.cfg \
--ipc_threads 16 \
--num_callers 8 \
--gpu_runners_per_device 4 \
--chunks_per_runner 512 \
--device "cuda:0 cuda:1" \ # this parameter should now scale nicely across both cards, I haven't checked though
--recursive \
--fast5_out \
-i fast5_input \
-s fastq_output

V100 config example for fast calling model

guppy_basecaller \
--disable_pings \
--compress_fastq \
-c dna_r9.4.1_450bps_fast.cfg \
--ipc_threads 16 \
--num_callers 8 \
--gpu_runners_per_device 64 \
--chunks_per_runner 256 \
--device "cuda:0 cuda:1" \
--recursive \
-i fast5_input \
-s fastq_output

---

Guppy basecalling benchmarking on a Titan RTX

There has been some discussion about the recent release of Guppy (3.4.1 and 3.4.2) in terms of speed. I was interested in running some benchmarks across different versions. I had a hunch it may have been something to do with the newly introduced compression of the fast5 files...

Test parameters

The only things I am changing are the version of Guppy being used, and in the case of 3.4.3 I am trying with and without vbz compression of the fast5 files. Everything else is as below:

System:

• Debian Sid (unstable)
• 2x 12-Core Intel Xeon Gold 5118 (48 threads)
• 256Gb RAM
• Titan RTX
• Nvidia drivers: 418.56
• CUDA Version: 10.1

Guppy GPU basecalling parameters:

• --disable_pings
• --compress_fastq
• --dna_r9.4.1_450bps_fast.cfg
• --num_callers 8
• --gpu_runners_per_device 64
• --chunks_per_runner 256
• --device "cuda:0"
• --recursive

For each Guppy version I ran the basecaller three times in an attempt to ensure that results were consistent^*.

Note: I chose the fast basecalling model as I wanted to do a quick set of benchmarks. If I feel up to it I may do the same thing for the high accuracy caller...

^* Spoiler, I didn't originally do this and it proved misleading...

Results

guppy version	time (seconds)	samples/s
3.1.5#	93.278	4.25638e+07
3.2.4#	94.141	4.21737e+07
3.3.0#	94.953	4.1813e+07
3.3.3#	95.802	4.14425e+07
3.4.1 (no vbz compressed fast5)	79.913	4.96824e+07
3.4.1 (vbz compressed fast5)	90.895	4.36797e+07
3.4.3 (no vbz compressed fast5)	90.674	4.37862e+07
3.4.3 (vbz compressed fast5)	82.877	4.79056e+07

^# these versions of Guppy did not support vbz compression of fast5 files (pre 3.4.X from memory).

Summary to date

I initially thought that there was something off with the compression imlementation in 3,4,3 as my first run on uncompressed data was ~3x slower than the run on the compressed data. When I grabbed 3.4.1 to perform the same check I noticed that it was fairly consistent between compressed and not. So I went back and was more rigorous and performed 3 iterations of each run for each version, ditto for versions 3.4.X compressed and not. This proved that the initial run was an anomaly and should be disregarded.

What was quite interesting is that running on vbz compressed fast5 data appears to be in the range of 8-10 seconds faster than uncompressed. So there is a slight added speed benefit on top of the nice reduction in file size - which is a little nicer for the SSD/HDD.

So at this stage I can't confirm any detrimental speed issues when using Guppy version 3.4.X, but this needs to be caveated with all the usual disclaimers:

• all systems are different (I'm not on Ubuntu for instance).
• drivers are different (I need to update).
• GPUs are very different, i.e. many people (including me) are using 'non-supported' GPUs - in my case a Titan RTX which is no slouch.
  • For what it's worth, I can add a comment here saying that I haven't had any speed issues with basecalling on our Nvidia Jetson Xaviers using Guppy 3.4.1.
• our 'little' Linux server isn't exactly a slouch either - so laptop/desktop builds could be very different.
• I ran the fast basecaller (I'm currently flat out and can't wait for the high accuracy caller) - I may take a subset of data and revist with hac at some stage.

You can view the 'raw' results/output for each run below:

Guppy 3.1.5

~/Downloads/software/guppy/3.1.5/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    --num_callers 8 \
    --gpu_runners_per_device 64 \
    --chunks_per_runner 256 \
    --device "cuda:0" \
    --recursive \
    -i flongle_fast5_pass \
    -s testrun_fast_3.1.5

ONT Guppy basecalling software version 3.1.5+781ed57
config file:        /home/miles/Downloads/software/guppy/3.1.5/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /home/miles/Downloads/software/guppy/3.1.5/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass
save path:          testrun_fast_3.1.5
chunk size:         1000
chunks per runner:  256
records per file:   4000
fastq compression:  ON
num basecallers:    8
gpu device:         cuda:0
kernel path:        
runners per device: 64

Found 105 fast5 files to process.
Init time: 1000 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 93278 ms, Samples called: 3970269916, samples/s: 4.25638e+07
Finishing up any open output files.
Basecalling completed successfully.

Guppy 3.2.4

~/Downloads/software/guppy/3.2.4/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    --num_callers 8 \
    --gpu_runners_per_device 64 \
    --chunks_per_runner 256 \
    --device "cuda:0" \
    --recursive \
    -i flongle_fast5_pass \
    -s testrun_fast_3.2.4

ONT Guppy basecalling software version 3.2.4+d9ed22f
config file:        /home/miles/Downloads/software/guppy/3.2.4/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /home/miles/Downloads/software/guppy/3.2.4/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass
save path:          testrun_fast_3.2.4
chunk size:         1000
chunks per runner:  256
records per file:   4000
fastq compression:  ON
num basecallers:    8
gpu device:         cuda:0
kernel path:        
runners per device: 64

Found 105 fast5 files to process.
Init time: 836 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 94141 ms, Samples called: 3970269916, samples/s: 4.21737e+07
Finishing up any open output files.
Basecalling completed successfully.

Guppy 3.3.0

~/Downloads/software/guppy/3.3.0/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    --num_callers 8 \
    --gpu_runners_per_device 64 \
    --chunks_per_runner 256 \
    --device "cuda:0" \
    --recursive \
    -i flongle_fast5_pass \
    -s testrun_fast_3.3.0

ONT Guppy basecalling software version 3.3.0+ef22818
config file:        /home/miles/Downloads/software/guppy/3.3.0/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /home/miles/Downloads/software/guppy/3.3.0/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass
save path:          testrun_fast_3.3.0
chunk size:         1000
chunks per runner:  256
records per file:   4000
fastq compression:  ON
num basecallers:    8
gpu device:         cuda:0
kernel path:        
runners per device: 64

Found 105 fast5 files to process.
Init time: 722 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 94953 ms, Samples called: 3970269916, samples/s: 4.1813e+07
Finishing up any open output files.
Basecalling completed successfully.

Guppy 3.3.3

~/Downloads/software/guppy/3.3.3/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    --num_callers 8 \
    --gpu_runners_per_device 64 \
    --chunks_per_runner 256 \
    --device "cuda:0" \
    --recursive \
    -i flongle_fast5_pass \
    -s testrun_fast_3.3.3

ONT Guppy basecalling software version 3.3.3+fa743a6
config file:        /home/miles/Downloads/software/guppy/3.3.3/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /home/miles/Downloads/software/guppy/3.3.3/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass
save path:          testrun_fast_3.3.3
chunk size:         1000
chunks per runner:  256
records per file:   4000
fastq compression:  ON
num basecallers:    8
gpu device:         cuda:0
kernel path:        
runners per device: 64

Found 105 fast5 files to process.
Init time: 726 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 95802 ms, Samples called: 3970269916, samples/s: 4.14425e+07
Finishing up any open output files.
Basecalling completed successfully.

Guppy 3.4.1 (not compressed)

~/Downloads/software/guppy/3.4.1/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    --num_callers 8 \
    --gpu_runners_per_device 64 \
    --chunks_per_runner 256 \
    --device "cuda:0" \
    --recursive \
    -i flongle_fast5_pass \
    -s testrun_fast_3.4.1

ONT Guppy basecalling software version 3.4.1+ad4f8b9
config file:        /home/miles/Downloads/software/guppy/3.4.1/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /home/miles/Downloads/software/guppy/3.4.1/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass
save path:          testrun_fast_3.4.1
chunk size:         1000
chunks per runner:  256CUDA Version: 10.1
fastq compression:  ON
num basecallers:    8
gpu device:         cuda:0
kernel path:        
runners per device: 64

Found 105 fast5 files to process.
Init time: 728 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 90895 ms, Samples called: 3970269916, samples/s: 4.36797e+07
Finishing up any open output files.
Basecalling completed successfully.

Guppy 3.4.1 (compressed)

~/Downloads/software/guppy/3.4.1/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    --num_callers 8 \
    --gpu_runners_per_device 64 \
    --chunks_per_runner 256 \
    --device "cuda:0" \
    --recursive \
    -i flongle_compressed \
    -s testrun_fast_3.4.1

ONT Guppy basecalling software version 3.4.1+ad4f8b9
config file:        /home/miles/Downloads/software/guppy/3.4.1/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /home/miles/Downloads/software/guppy/3.4.1/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_compressed
save path:          testrun_fast_3.4.1
chunk size:         1000
chunks per runner:  256
records per file:   4000
fastq compression:  ON
num basecallers:    8
gpu device:         cuda:0
kernel path:        
runners per device: 64

Found 105 fast5 files to process.
Init time: 725 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 79913 ms, Samples called: 3970269916, samples/s: 4.96824e+07
Finishing up any open output files.
Basecalling completed successfully.

Guppy 3.4.3 (not compressed)

~/Downloads/software/guppy/3.4.3/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    --num_callers 8 \
    --gpu_runners_per_device 64 \
    --chunks_per_runner 256 \
    --device "cuda:0" \
    --recursive \
    -i flongle_fast5_pass \
    -s testrun_fast_3.4.3_uncompressed

first run (it looks like this is anomaly)

ONT Guppy basecalling software version 3.4.3+f4fc735
config file:        /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass
save path:          testrun_fast_3.4.3_uncompressed
chunk size:         1000
chunks per runner:  256
records per file:   4000
fastq compression:  ON
num basecallers:    8
gpu device:         cuda:0
kernel path:
runners per device: 64

Found 105 fast5 files to process.
Init time: 738 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 270953 ms, Samples called: 3970269916, samples/s: 1.4653e+07
Finishing up any open output files.
Basecalling completed successfully.

second run

ONT Guppy basecalling software version 3.4.3+f4fc735
config file:        /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_fast5_pass
save path:          testrun_fast_3.4.3_uncompressed
chunk size:         1000
chunks per runner:  256
records per file:   4000
fastq compression:  ON
num basecallers:    8
gpu device:         cuda:0
kernel path:        
runners per device: 64

Found 105 fast5 files to process.
Init time: 705 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 90674 ms, Samples called: 3970269916, samples/s: 4.37862e+07
Finishing up any open output files.
Basecalling completed successfully.

third run

ONT Guppy basecalling software version 3.4.3+f4fc735 config file: /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg model file: /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/template_r9.4.1_450bps_fast.jsn input path: flongle_fast5_pass save path: testrun_fast_3.4.3_uncompressed3 chunk size: 1000 chunks per runner: 256 records per file: 4000 fastq compression: ON num basecallers: 8 gpu device: cuda:0 kernel path:
runners per device: 64

Found 105 fast5 files to process. Init time: 719 ms

0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----|

---

Caller time: 94516 ms, Samples called: 3970269916, samples/s: 4.20063e+07 Finishing up any open output files. Basecalling completed successfully.

Guppy 3.4.3 (compressed)

~/Downloads/software/guppy/3.4.3/ont-guppy/bin/guppy_basecaller \
    --disable_pings \
    --compress_fastq \
    -c dna_r9.4.1_450bps_fast.cfg \
    --num_callers 8 \
    --gpu_runners_per_device 64 \
    --chunks_per_runner 256 \
    --device "cuda:0" \
    --recursive \
    -i flongle_compressed \
    -s testrun_fast_3.4.3

ONT Guppy basecalling software version 3.4.3+f4fc735
config file:        /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
model file:         /home/miles/Downloads/software/guppy/3.4.3/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
input path:         flongle_compressed
save path:          testrun_fast_3.4.3
chunk size:         1000
chunks per runner:  256
records per file:   4000
fastq compression:  ON
num basecallers:    8
gpu device:         cuda:0
kernel path:        
runners per device: 64

Found 105 fast5 files to process.
Init time: 721 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 82877 ms, Samples called: 3970269916, samples/s: 4.79056e+07
Finishing up any open output files.
Basecalling completed successfully.

Hi all, thanks for the feedback. I restarted the system previously (always gotta try the turn on and off). I am getting this error when I try to launch using the terminal:
minit@epbarnhart-desktop:~/jetson_nanopore_sequencing$ /opt/ont/ui/kingfisher/MinKNOW
No protocol specified
The futex facility returned an unexpected error code.Aborted (core dumped)
So clearly something is amiss. Should I just start from scratch here? Thanks again.

@epbarnhart: The futex facility is like a timing system to ensure asynchronous functions do not try to operate on something before its value is set or its memory has been allocated, and a locking/control system to ensure a shared value or object is not altered before it is done being used. It sounds like something is misconfigured or being executed from an unexpected initial condition. Incidentally, I can produce the "... futex facility returned an unexpected error ..." message and core dump by calling /opt/ont/ui/kingfisher/MinKNOW from a local Ubuntu terminal on the Jetson Xavier NX when logged in as minit in an ssh session to the localhost (sometimes, other times it just gives a segmentation fault). I'm assuming that you are not indirectly logged in as user minit, but I wonder if something is misconfigured in the system itself. Can you launch other gui apps (e.g. gedit or chromium-browser) from the terminal? Those fail more gracefully when the display cannot be opened properly.

Success! We did the reinstall as suggested, but I think the main issue was that I had switched users via terminal rather than logging from the startup in as minit. Such a small thing, and I am super excited to get this going. Thank you so much for your help and patience, I really appreciate it.

@epbarnhart: Good to hear it is working! I also found out the hard way that GUI apps won't run correctly when launched from an indirect terminal login, even on the local system (they can't get a display correctly). Most apps catch it and provide a message, but MinKNOW, possibly its electron framework, apparently just crashes without a useful message.

@thalljiscience: Thanks for the stopping protocol. We have so far tested it in our x86_64 environment and it works! I apologize for the delay, I was out of the office. My colleague had tried it meanwhile! Great job!

@juhench: Great to hear! Thanks for letting me know. I'm glad to have that AGX Xavier up and running now too. Everything works great on it except I have this pesky problem of having to control the fan manually for some reason or it gets really hot. I need to search around to see if that is a common thing and something just needs to be installed or if I have to write a pid service to run the fan.

@thalljiscience: The fan is indeed a bit evil on the AGX. We have had heat-induced reboots. On the development devices, I am running it at around 80% continuously and on the diagnostic system, we run it at 70% when launching a run (manually). I hope to put his into our automatic start/end script; fixing the details for the live data evaluation and decision-making right now.

I have played around with the fan. It somehow starts too late. When It should turn on when the system reaches 40C but instead it turns on around reaching 50C. I have gone through the forums and found that the fan's PWM has been addressed in different ways throughout jetpack's evolution. Still, the "cool" mode does not do what it is supposed to. A workaround might be adapting the fan to CPU/GPU load and immediately increase speed according to load. Without load, fan speed could be decreased according to the present temperature.

@juhench: That is interesting to hear. I thought it was something I did in setting up the system. I have been setting the PWM at 125 (50%) by default. I still think my AGX is a bit confused though, because most of the time after booting the fan does not turn on by itself no matter how hot it gets (it gets painful to the touch), but once in a while after reboot it will kick on. I wonder if a simple service that runs a pid algorithm to modulate the fan up and down with a slightly lower-than-default temperature profile could just be written. I'd probably just try to borrow the general pid algorithm off the Arduino controller used in this fun little open-source PocketPCR device for controlling heating/cooling of the aluminum block, then empirically modify parameters and eliminate the heating side of the equation: https://github.com/thalljiscience/PocketPCR_Programmable. I've done this for fan control to keep a larger aluminum block or a heated PocketPCR lid at a set temp using Elegoo (cheap Arduino) microcontroller boards. First I'd need to learn how to read the temperature on the Jetson and write a background service though. I haven't done that on a Linux device before.

@thalljiscience: One thing that might help in understanding the AGX's fan is the fact that the jetson_stats background component, which is started as a service, takes a moment after booting to activate. If you create a service for custom fan control, it will only have an effect if you add a delay of a minute (or a condition?); otherwise, your PWM settings, etc. will be overwritten with the non-working jetson_stats defaults. If you have jtop running in the terminal and set the fan manually, the value will last. If you set the fan via the command line, etc. elsewhere, I have observed that the system sometimes overrides your setting. For our routine use, the lab techs always have jtop open in the fan view and make sure it is running 70% when launching.

@jhuench: Interesting. Thank you for that information. If a service were written to read the current temp on a timer and set the fan PWM directly according to a standard proportional–integral–derivative feedback, I think that should work, since the PWM can be set at any time manually and appears to remain stable thereafter once set. If it is sometimes overridden by the system, that should not matter if the PID responds in a few seconds by setting the PWM according to the current temperature, as long as the temperature can be read.

@jhuench: With a small C++ program (compiled with g++), it seems pretty easy to control the fan speed according to the thermal reading from tegrastats (which is effectively just a weighted average of CPU, GPU and AUX temps at 3:3:4). It turns out to be easier to just use a simple linear response to temperature over some target parameter than a pid because the intent is not to minimize time to a target from both sides, but just to keep the temperature down overall.

The little program I wrote below simply:

1. Runs tegrastats once every 3 seconds (can be modified to be less frequent), then stops tegrastats to get a single reading.
2. Loads the tegrastats output and deletes it
3. Parses output to get the "thermal" temp
4. Calculates a PWM setting and sets fan PWM with /sys/devices/pwm-fan/target_pwm at some value over a minimum (20 now, since that is around the lowest setting that will actually move the fan).

It seems to work fine if process is started with sudo (required for the system call to target_pwm).
I'm testing it with a sequencing run with basecalling. Parameters can be adjust for the adjustFan method to have the fan kick up slower or faster and at a lower or higher temperature.

It's a bit little clunky, since there is a transient file intermediate.

I slowed it down to an 8 second interval (shouldn't really impact resources), set the adjustFan parameters (targetTemp, rangeOver, and minPWM) to 30, 30, 30 , commented out the console outputs, and set it up as a service. It starts up OK and I'm going to monitor it for a a few days to see if there is any apparent interference with anything else.

fancontroller.cpp:

#include <iostream>
#include <fstream>
#include <unistd.h>
#include <stdio.h>
#include <string.h>

using namespace std;

double extractAveTemp(char *tegraBuffer)
{
  if (char* thermal = strstr(tegraBuffer, " thermal@")) // find thermal temp in buffer 
  {
    thermal+=9;  // move up to temp number in buffer
    int endAt=0;  
    while (endAt < strlen(thermal) && thermal[endAt] != 'C') endAt++;  // find end of temp
    if (endAt < strlen(thermal) && thermal[endAt]== 'C') 
    {
      thermal[endAt]='\0';  // terminate buffer string
      return std::stod(thermal); // return value as a double -- this value is actually ((3 * GPU) + (3 * CPU) + (4 * AUX)) / 10), but provided as thermal in tegrastats
    }   
  } 
}

void adjustFan(double currTemp, double targetTemp, double rangeOver, int minPWM)
{
  int pwmRange = 255 - minPWM;  // determine effective PWM range
  int PWM = minPWM;  // initialize PWM target at minimum
  if (currTemp > targetTemp) 
  {
    double overage = currTemp - targetTemp; 
    double percentRange = overage / rangeOver;
    int pwmOffset = (int)(percentRange * (double)pwmRange);
    PWM = pwmOffset + minPWM;  // set PWM to percentage along range plus minimum PWM setting
  }
  char commandStr[55];
  int len = sprintf(commandStr, "sh -c 'echo %i > /sys/devices/pwm-fan/target_pwm'", PWM);  // run standard fan PWM override and set to new PWM setting
  commandStr[len] = '\0';
  system(commandStr);
  
  cout << "fan PWM set to " << PWM << endl;  // Delete or comment out for a service
}

void monitorTemp() 
{
  int i=0;
  while (i<1) { // loop indefinitely until killed.  If killed, also kill tegrastats with tegrastats --stop in case it was killed with tegrastats running
    system("tegrastats --interval 3000 --logfile /home/minit/temp_control/currStats.txt &"); // Start tegrastats in background mode on 3 second timer
    usleep(4500000);  // wait 4.5 seconds to allow time for output
    system("tegrastats --stop");  // stop tegrastats
    usleep(500000);  // sleep a half second
    ifstream statFile("/home/minit/temp_control/currStats.txt", std::ios::binary | std::ios::ate);  // read the stats file
    streamsize size = statFile.tellg();
    if (size > 0) {  // if the file has content (if not, just skip and let it try again)
      statFile.seekg(0, std::ios::beg);
      char* buf = new char[size+1];
      if (statFile.read(buf, size))
      {
        buf[size]='\0';
        cout << buf << endl;  // check output
        double thermalTemp = extractAveTemp(buf); // Get the current thermal temperature - actually = ((3 * GPU) + (3 * CPU) + (4 * AUX)) / 10)
        cout << "thermal = " << thermalTemp << endl; // Delete or comment out for a service
        adjustFan(thermalTemp, 30, 40, 20);  // set the fan speed
      }
      statFile.close();
      delete []buf; // clear buffer memory
    }
    system("rm currStats.txt");  // delete the stats file to clear it out for next round
    usleep(1000000); // wait a second
  }
}

int main()
{
  cout << "starting temp monitor" << endl;
  monitorTemp();	
}

@thalljiscience: Thanks a lot! I wonder if one could replace the way through the file (SSD wear). I think I read somewhere on the NVIDIA forum where we could grab the thermal values. Give me some time; I will try to find that again.

Hi @sirselim

I just update my PC to MinION 21.11.07 and manually setup Guppy-5.1.12 as your suggestions. But there are some changes in app-conf of MinION 21.11.07 that are different from your instructions for manual installation guppy-5.1.11. When I tried stop and start minknow service, I cannot see guppy_basecaller running. Could you please check it out?

Thanks
Han

Hi @hanfan1803

Yes, the recent release of MinKNOW (21.11.7) has changed a few things. ONT for some time have been starting to move the management of Guppy out from under the control of MinKNOW, that is what we saw with the release of the systemd service guppyd. The most recent release is starting to push this transition further, namely that guppyd.service now overrides most of the information contained in MinKNOWs app_conf. The message I got from within ONT is that they ultimately don't want the majority of users to have to config settings with app_conf.

With this in light I went into the update with the goal of trying not to tweak any parameters in app_conf. The most important thing is to ensure guppyd.service is correctly configured, mainly that it's pointing towards the correct version of Guppy - if that's the GPU version that you manually download then you have to ensure that that path is reflected in guppyd.service. The other thing to note is the port being served to. This needs to be the same between guppyd.service and app_conf.

I actually wrote a response in the community forum that I'll post below that reiterates the above with more detail:

---

I think the major problem here is that there are two versions of Guppy, CPU and GPU. By default the CPU version is installed alongside MinKNOW, which means that the paths to the Guppy binaries (basecaller, client, server, etc) in the app_conf are all pointing to the CPU version. If the guppyd.service is installed at this time it will also be pointing to the CPU Guppy basecall server executable. When you install GPU Guppy you have to overide this path to the CPU version in guppyd.service with the path to the GPU version. BUT (from my testing) you also have to make sure all the paths in the app_conf are also pointing to the GPU version of the respective binaries.

As I mentioned earlier in the thread ONT are moving towards Guppy managing itself, meaning that we shouldn't really ever need to touch/modify the app_conf. So I went in to my "testing" in this mind set. I first did a fresh install of all ONT software, including the GPU version of Guppy. The first thing I modified was guppyd.service to make sure it pointed towards the GPU server binary. It's important to note that the --port option must represent the same value that is in the app_conf (or that should be the other way around, the app_conf should reflect what is used by the server). Here's my modified line:

ExecStart=/usr/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --device cuda:all --port /tmp/.guppy/5555

NOTE: I sym link the latest GPU version to /usr/bin/ on my systems. Make sure to update the path to where you have installed GPU Guppy.

Before you edit the service it's worth stopping it if it's running:

sudo systemctl stop guppyd.service

Edit the service file and restart:

sudo systemctl restart guppyd.service

At this point I restarted the minknow.service even though I hadn't modified it (it can never hurt), and tried to do a run with basecalling in MinKNOW. No dice, it errored out. Checking the logs the error was that MinKNOW couldn't find the Guppy binaries. Turns out that the paths in the app_conf are still respected. So I just edited them to point to the GPU Guppy binaries, restarted the MinKNOW service again, as well as guppyd, and then tried another run. This time everything worked just fine. Live GPU basecalling as well as re-basecalling from within MinKNOW.

I have now repeated the same steps across 5 different devices (a 21.04 server, a 20.04 server and 3 ARM64 Nvidia Jetson devices) and all are up and running nicely.

I don't know whether this will work for everyone but I'm happy to help troubleshoot - I understand how frustrating this can be!

---

I recently had a bit of an overhaul of the Jetson Github repository in light of the above, so if you are looking for some working examples (albeit for the ARM architecture) feel free to take a look there.

• Miles

Thank you so much for detailed reply.

When waiting for your reply, I tried to reinstall minion-nc for few times. It ended up with a new issue regarding setting up minion-nc during installation with apt-get. The log is below:

ntt-7@ntt-7:~$ sudo apt-get install minion-nc
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libboost-atomic1.65.1 libboost-chrono1.65.1 libboost-log1.65.1
  libboost-program-options1.65.1 libboost-regex1.65.1 libhdf5-cpp-100
  ont-guppy-cpu-for-minion ont-guppyd-for-minion
The following NEW packages will be installed:
  libboost-atomic1.65.1 libboost-chrono1.65.1 libboost-log1.65.1
  libboost-program-options1.65.1 libboost-regex1.65.1 libhdf5-cpp-100
  minion-nc ont-guppy-cpu-for-minion ont-guppyd-for-minion
0 upgraded, 9 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/756 MB of archives.
After this operation, 1.307 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Selecting previously unselected package libboost-atomic1.65.1:amd64.
dpkg: warning: files list file for package 'minknow-core-minion-nc' missing; assuming package has no files currently installed
(Reading database ... 181322 files and directories currently installed.)
Preparing to unpack .../0-libboost-atomic1.65.1_1.65.1+dfsg-0ubuntu5_amd64.deb ...
Unpacking libboost-atomic1.65.1:amd64 (1.65.1+dfsg-0ubuntu5) ...
Selecting previously unselected package libboost-chrono1.65.1:amd64.
Preparing to unpack .../1-libboost-chrono1.65.1_1.65.1+dfsg-0ubuntu5_amd64.deb ...
Unpacking libboost-chrono1.65.1:amd64 (1.65.1+dfsg-0ubuntu5) ...
Selecting previously unselected package libboost-regex1.65.1:amd64.
Preparing to unpack .../2-libboost-regex1.65.1_1.65.1+dfsg-0ubuntu5_amd64.deb ...
Unpacking libboost-regex1.65.1:amd64 (1.65.1+dfsg-0ubuntu5) ...
Selecting previously unselected package libboost-log1.65.1.
Preparing to unpack .../3-libboost-log1.65.1_1.65.1+dfsg-0ubuntu5_amd64.deb ...
Unpacking libboost-log1.65.1 (1.65.1+dfsg-0ubuntu5) ...
Selecting previously unselected package libboost-program-options1.65.1:amd64.
Preparing to unpack .../4-libboost-program-options1.65.1_1.65.1+dfsg-0ubuntu5_amd64.deb ...
Unpacking libboost-program-options1.65.1:amd64 (1.65.1+dfsg-0ubuntu5) ...
Selecting previously unselected package libhdf5-cpp-100:amd64.
Preparing to unpack .../5-libhdf5-cpp-100_1.10.0-patch1+docs-4_amd64.deb ...
Unpacking libhdf5-cpp-100:amd64 (1.10.0-patch1+docs-4) ...
Selecting previously unselected package ont-guppy-cpu-for-minion.
Preparing to unpack .../6-ont-guppy-cpu-for-minion_5.1.12-1~bionic_amd64.deb ...
Unpacking ont-guppy-cpu-for-minion (5.1.12-1~bionic) ...
Selecting previously unselected package ont-guppyd-for-minion.
Preparing to unpack .../7-ont-guppyd-for-minion_5.1.12-1~bionic_all.deb ...
Unpacking ont-guppyd-for-minion (5.1.12-1~bionic) ...
Selecting previously unselected package minion-nc.
Preparing to unpack .../8-minion-nc_21.11.7-1~bionic_all.deb ...
Unpacking minion-nc (21.11.7-1~bionic) ...
Setting up libhdf5-cpp-100:amd64 (1.10.0-patch1+docs-4) ...
Setting up libboost-atomic1.65.1:amd64 (1.65.1+dfsg-0ubuntu5) ...
Setting up libboost-chrono1.65.1:amd64 (1.65.1+dfsg-0ubuntu5) ...
Setting up libboost-program-options1.65.1:amd64 (1.65.1+dfsg-0ubuntu5) ...
Setting up libboost-regex1.65.1:amd64 (1.65.1+dfsg-0ubuntu5) ...
Setting up libboost-log1.65.1 (1.65.1+dfsg-0ubuntu5) ...
Setting up ont-guppy-cpu-for-minion (5.1.12-1~bionic) ...
Setting up ont-guppyd-for-minion (5.1.12-1~bionic) ...
Setting up minion-nc (21.11.7-1~bionic) ...
/var/lib/dpkg/info/minion-nc.postinst: 8: cd: can't cd to /opt/ont/minknow/
dpkg: error processing package minion-nc (--configure):
 installed minion-nc package post-installation script subprocess returned error exit status 1
Processing triggers for libc-bin (2.27-3ubuntu1.4) ...
Errors were encountered while processing:
 minion-nc
E: Sub-process /usr/bin/dpkg returned an error code (1)

If you know about this issue, please let me know how to fix it. I also make a post on nanopore community about this.

Han

@hanfan1803 - that is a pretty generic error so I really don't know where to start. I personally stay away from the minion-nc package, solely because it packages the CPU version of Guppy and I don't have a need for it as I'm always installing the GPU version. The GPU version can also perform basecalling on the CPU, so I don't understand why ONT insist on maintaining different versions.

One thing I note in your output is that you have boost libs being installed/updated, that's always a fun time and a potential sign that you might be mixing and matching between OS releases. What version of Ubuntu is this computer running and what version of the ONT repo's are you installing MinKNOW etc from?

Hi @sirselim

My OS is Ubuntu 18.04, and ONT repo is http://mirror.oxfordnanoportal.com/apt bionic-stable non-free" | sudo tee /etc/apt/sources.list.d/nanoporetech.sources.list. The last version minion-nc (21.06) was also installed through apt-get. I just simply remove/purge remove it from apt-get, then install it again.

Han

Yes I know it's a generic error, but everytime, I try removing then installing new version of MinKnow I have this exactly issue. The only solution so far is that re-install OS. My PC also is setup with tons of other bioinformatic softwares. It's really annoying if I have to re-install the OS.

Hi,

I already fix this issue with apt-get. Thank you for the instruction to setup live basecalling with guppy.

Han

Hi @hanfan1803,

we have had this issue for a long time, i.e. things change in MinKNOW / guppy so frequently that it is impossible for me and my (small) scientific team to keep up. We have also developed software around MinKNOW and hence use "frozen" versions, i.e. we keep all required *.deb files in a safe place. For our applications, this is good enough. Our strategy also allows for exact setup replication on new computer systems. The only thing one must not perform are updates. One way (at least with the versions we use) that allowed disabling updates is to remove the ONT repo from the /etc/apt/sources... and in addition to remove the ubuntu update manager (functionally) by removing executable permissions.

This may not be a solution for staying up-to-date; otherwise, if you would like to produce reproducible data and be sure you obtain them through the exact same setup over periods of several years, this will be the only way for the time being. Versions freezes have been announced by ONT but were never made available to me.

Finally getting a chance to get back to this. First off I've got to say thanks so much Miles (and other contributors) for your dedication to this. Like I said I have not had time to dedicate to this and what has documented here and in the main Nanopore Sequencing with the Xavier page has saved my butt several times. So thanks!

Recently we started running into space problems, even though there was plenty of room on the SSD I'd installed (but the main boot SD only had 8GB of space left). I thought it was a misdirected swap file space because the sequencer would error out after 8GB worth of sequencing. Now looking over this gist again I wonder if it had something to do with some of the settings in the guppy.service file. My first try was redirecting the swap to the SSD, but no dice. So, I decided to try something that almost bricked the Jetson before and make the Xavier boot off of the SSD. As of JetPack 4.6 this should be possible. I found this wonderful site here (https://github.com/jetsonhacks) on github that had step by step how to do it. And without too much fiddling it actually worked! So now I wanted to reinstall of the new Mk1C update because I wanted to check out short fragment mode and play with adaptive sequencing. I dug around in the ONT forums and it looked like guppy 6.0.6 should work with software release 22.03.2 for Mk1C (in retrospect I probably should have picked 6.0.7, but I missed it). I changed the part of the mk1c setup instructions to reflect the version of guppy downloaded. Other than that I followed the instructions pretty much to the letter and right now I'm sequencing away with live basecalling!

Two things of note: 1) I think it's basecalling slower than it was before (averaging 50% on super-high accuracy with 25% of the pores sequencing). This may be the whole 6.0.6 vs. 6.0.7 problem. I want to explore more but my student may kill me if I don't get this setup back. 2) several of the features mentioned on ONT's site aren't there (the System overview, the basecall model in the fastq header and, sadly, the short fragment mode). I need to get the versions of all the components. Is there a simple command/place to look for getting version numbers for everything MinKNow is using?

Anyway I count the booting off the SSD and live basecalling with 6.0.6 as some kind of victory. I'll write again when I find out more. Thanks all.

-John

Edit: I'm an idiot. I installed the Xenial builds. So this is 21.11.7 running with guppy 6.0.6 not 22.0.3.2 as I previously said. I will fix and we'll see what happens.

Well one step forward and two steps back. First I removed guppy and got version 6.0.7. Then I removed the Xenial builds and set the repos to the bionic repos in /etc/apt/sources.list.d/nanopore.sources.list. That didn't work because of some certificate problems. I set them to trusted so the lines looked like this

deb [trusted-yes arch=arm64] https://cdn.oxfordnanoportal.com/apt bionic-stable-mk1c non-free
deb [trusted-yes arch=arm64] https://mirror.oxfordnanoportal.com/apt bionic-stable-mk1c non-free

after which things downloaded nicely. Then I went through the mk1c setup guide and everything seemed file. MinKNOW booted up fine, but now plugging in the Mk1b resulted in a high fan noise and not being recognized. I went back to the part of the setup where a permissions problem was mentioned for the minknow service and the result being the mk1b not being recognized. Sadly that didn't allow it to be recognized either. I restarted, still not recognizing the MinION. Interesting there was a program that crashed in the background called ont-get-product-info.

I remember having to deal with these connection problems way in the beginning of looking at using the Xavier as a MinIT like device, but for the life of me I can't remember how they were solved. Anyone got any ideas? Thanks.

-John

Just wanted to touch base here after a long hiatus. I'm about to update the Jetson Nanopore GitHub repo. This update will provide instructions for getting the latest Mk1C software running on the Jetson boards. The big issue was the large update from Xenial to Bionic and a large amount of code refactoring with recent updates to MinKNOW. Plus I haven't had the time or hardware since changing jobs to look into these issues.

So apologies to anyone that has been struggling, but hopefully updating to the latest versions of MinKNOW and Guppy will fix things. A nice benefit is that now that things are inline again (i.e. Mk1C running Bionic and the Jetsons running Bionic) Guppy can be pulled stright from the repos and doesn't require manual downloading and configuration. 🥳

I'm hoping to have the update live in the next few days (here).

It's great to hear you've almost completed the transition. Following your past instructions and hard work, I've had great success running the MinION with fast basecalling for all the 16S analyses I do. Unfortunately, last week something strange happened (MinKNOW GUI no longer shows the different kits), so I've been forced to use my M1 MacBook... and the time differences are tremendous.

It’s most likely the grpc certs expired 1/14/2023. It can be updated without updating minknow. I’ll send a forum link later today. Or you can look it up in the community forums.

…

-Tom

On Jan 26, 2023, at 7:20 AM, pablosoup1 ***@***.***> wrote: ***@***.*** commented on this gist. It's great to hear you've almost completed the transition. Following your past instructions and hard work, I've had great success running the MinION with fast basecalling for all the 16S analyses I do. Unfortunately, last week something strange happened (MinKNOW GUI no longer shows the different kits), so I've been forced to use my M1 MacBook... and the time differences are tremendous. — Reply to this email directly, view it on GitHub or unsubscribe. You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS or Android.

Thanks so much for pointing me in the right direction! Here's ONT's official response/fix:

https://community.nanoporetech.com/posts/certificate-expiry-for-min

And the first image they show is exactly what I saw in the GUI, with no real error codes showing up.

That is a different post than I first saw. Thanks. It shows the same procedure for updating the certs from the .deb package, and that did work for me. Just FYI, due to the skyrocketed cost and low availability last year of NVIDIA Jetson systems (I haven’t checked on that situation recently), however, I switched from Jetson to Intel Nuc gaming devices equipped with NVIDIA RTX-level GPUs and run minknow under Ubuntu. It works pretty well in case anyone needs an alternative that is not too far off from the original size (a little bigger) of the Jetsons. They also have more ports and can double as pretty decent and versatile general use computers. e.g.: https://www.amazon.com/Intel-Enthusiast-NUC11PHKi7C-Barebone-i7-1165G7/dp/B09DDJFSNY/ref=sr_1_3?crid=159ZQQ4B4G68&keywords=intel+nuc+11+enthusiast+barebones&qid=1674752926&sprefix=intel+nuc+11+enthusiast+barebones%2Caps%2C146&sr=8-3 The cost is now half of what I paid for barebones systems about a year ago. I put in 16Gb or 32Gb of memory and a 1TB NVMe for less than $200. Ubuntu OS is free and comes loaded - Libre Office and other desktop programs run better on these than on the Jetson device in my experience.

…

-Tom

On Jan 26, 2023, at 8:48 AM, pablosoup1 ***@***.***> wrote: ***@***.*** commented on this gist. Thanks so much for pointing me in the right direction! Here's ONT's official response/fix: https://community.nanoporetech.com/posts/certificate-expiry-for-min And the first image they show is exactly what I saw in the GUI, with no real error codes showing up. — Reply to this email directly, view it on GitHub or unsubscribe. You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS or Android.

BTW, I just downloaded the certs using the link provided for Mac/PC, then moved it into the appropriate spot on my Jetson. Worked like a charm (because I'd forced the system to quit updating the ONT packages, this was a quick workaround).

ONT just announced that they are removing the old authentication system.

https://community.nanoporetech.com/posts/retiring-old-minknow-authe

I take it this will probably crater the Jetson nano?

@pablosoup1 - not sure what you mean by "crater the Jetson nano"?

The authentication change shouldn't cause any issues as long as you are using the latest MinION Mk1C software branch, as this is up to date and includes the change. Now, if you are still using the old MinIT branch then yes this will stop working. But there is no reason to be on that software when the Mk1C software runs well and gives you all the latest features.

On that note, the next version of MinKNOW sees Guppy being replaced by Dorado (even on the Mk1C). This is good for many reasons, but mainly: 1) dorado has matured and is much faster than Guppy now, 2) dorado has native ARM builds, including for the newer Jetson Orin architecture. So for all those out there with Orin boards, you should be able to get a fully working sequencing stack in the next month or so, as soon as MinKNOW has Dorado integrated.