fast5 demultiplex by barcode

fast5 demultiplex by barcode

pip3 install ont-fast5-api

cat sequencing_summary.txt | cut -f 21 | grep barcode[0-9] | sort | uniq > barcodes.log

while read -r barcode
  do
    head -n 1 sequencing_summary.txt > $barcode.txt
    cat sequencing_summary.txt | grep $barcode >> $barcode.txt
  
    for f in *.txt
      do 
        fast5_subset -i folder/with/your/fast5s -s output/folder/for/demultiplexed_fast5/$barcode -l $f -t 14
      done
  done < barcodes.log

Hi Johannes,

thanks for providing this command pipeline. I am wondering is there any way to generate separated(for each barcode) fast5 files from fast5_skip. many thanks. have a great days

Hi, I'm not sure, I think the 'fast5_skip' assignment happens if the quality is too low, and therefore barcode assignment never happens. Since guppy can now properly demultiplex fast5s files I don't use this script anymore. What you could try is use the latest version of guppy and recall your fast5 files and lower the qscore to accept more reads. Therefore you should end up with fewer reads in the skip folder. Hope this helps. I can send you my guppy command when I'm back in the lab tomorrow.

…

On Tue, 15 Feb 2022, 21:07 Fatih Tiras, ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ Hi Johannes, thanks for providing this command pipeline. I am wondering is there any way to generate separated(for each barcode) fast5 files from fast5_skip. many thanks. have a great days — Reply to this email directly, view it on GitHub <https://gist.github.com/4af1e37ac96af3d9ed2104658993d944#gistcomment-4066085>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBHB2UQ7MZRRKOZERLKEO3U3JFZLANCNFSM5OOPDVPA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: <JWDebler/fast5 demultiplex by ***@***.***>

hi Johannes,

Firstly thanks for a quick reply. Fast5_skip file was generated from MinKNOWN when ı stopped the basecalling(while Minkonw was still working for basecall after sequence finished) after sequencing finished. then additional fast5 (fast5_skip) was generated from MinKNOWN and put all unbasecallled fast5 files into that file. so ı am trying to find a way to separate my fast5 files for each barcode from the fast5_skip. many thanks. Sincerely.

Hi, If it has been skipped due to not been basecalled, this should be easy. Also, if you are basecalling after the run, I would suggest using the 'sup' model for better basecalls. You might as well recall the whole run instead of only the 'skipped' folder. I hope you have a GPU in that case though :-) Otherwise substitute the 'sup' in the first line of the command with whatever basecalling model you used in your actual run ('fast' or 'hac'). You could do something like this: guppy_basecaller -c dna_r9.4.1_450bps_sup.cfg \ -i fast5_skip_folder \ --recursive \ -o output_folder \ --barcode_kits EXP-NBD104 \ --trim_barcodes \ --detect_mid_strand_barcodes \ --min_score_mid_barcodes 60 \ --compress_fastq \ --fast5_out \ -x cuda:all the last line (-x cuda:all) is only needed if you have a GPU. If you are using guppy 6 you need to change this line:

…

--min_score_mid_barcodes 6 to:

--min_score_barcode_mid Cheers

On Tue, 15 Feb 2022 at 21:47, Fatih Tiras ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ hi Johannes, Firstly thanks for a quick reply. Fast5_skip file was generated from MinKNOWN when ı stopped the basecalling(while Minkonw was still working for basecall after sequence finished) after sequencing finished. then additional fast5 (fast5_skip) was generated from MinKNOWN and put all unbasecallled fast5 files into that file. so ı am trying to find a way to separate my fast5 files for each barcode from the fast5_skip. many thanks. Sincerely. — Reply to this email directly, view it on GitHub <https://gist.github.com/4af1e37ac96af3d9ed2104658993d944#gistcomment-4066118>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBHB2QNVMGN6LHL36UWW23U3JKNNANCNFSM5OOPDVPA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: <JWDebler/fast5 demultiplex by ***@***.***>

Hi Johannes,

many thanks for sending your commands. ı am really appreciated it. but ı got some error. the first ı tried with version 6(guppy) and got it = Unexpected option '-o' found on command line.
Unexpected option '--min_score_mid_barcodes' found on command line.
then ı tried previous version 5.0.16(guppy) and got "Unexpected option '-o' found on the command line.
Missing required option 'save_path'." error. Also, ı don't have GPU facilities currently and are new in this field. manny thanks for your help.
used command= /cluster/lrcfs/ftiras/bin/ont-guppy-cpu-5.0.16/bin/guppy_basecaller -c dna_r9.4.1_450bps_sup.cfg -i fast5_skip/ --recursive -o output_folder/ --barcode_kits EXP-NBD104 --trim_barcodes --detect_mid_strand_barcodes --min_score_mid_barcodes 60 --compress_fastq --fast5_out

Sincerely

Hi Johannes,

when ı remove the -o output_folder command and added save_path command the basecalling is worked but into the work space all fast5 are the same area and they have same name with what is in fast5_skip folder.
Also, ı attached a jpeg for my file appearance, ı hape it gives better idea. manny thanks. have a great days.
best wishes

oh, my mistake, that should have been '-s', not '-o'. '--min_score_mid_barcodes' is for guppy 5, in guppy 6 this has been renamed to '--min_score_barcode_mid' (see screenshot below). I don't exactly remember if guppy 5 could do fast5 demultiplexing, that may have been an option that was implemented with guppy 6. Also, make sure to use the correct barcoding kit parameter. NBD104 is the barcoding kit for barcodes 1 to 12. Try again with guppy 6 and '--min_score_barcode_mid': guppy_basecaller -c dna_r9.4.1_450bps_sup.cfg -i fast5_skip/ --recursive -s output_folder/ --barcode_kits EXP-NBD104 --trim_barcodes --detect_mid_strand_barcodes --min_score_barcode_mid 60 --compress_fastq --fast5_out [image: image.png] Hope this helps :-)

…

On Thu, 17 Feb 2022 at 08:18, Fatih Tiras ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ Hi Johannes, when ı remove the -o output_folder command and added save_path command the basecalling is worked but into the work space all fast5 are the same area and they have same name with what is in fast5_skip folder. Also, ı attached a jpeg for my file appearance, ı hape it gives better idea. manny thanks. have a great days. best wishes [image: Slayt1] <https://user-images.githubusercontent.com/75265865/154379456-6f204b7d-b26a-46c1-b0ad-a4d8119faa01.JPG> — Reply to this email directly, view it on GitHub <https://gist.github.com/4af1e37ac96af3d9ed2104658993d944#gistcomment-4068317>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBHB2V7LYCG7H4GWV64Q7TU3Q5GXANCNFSM5OOPDVPA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: <JWDebler/fast5 demultiplex by ***@***.***>

Hi Johannes,

Now it looks like working. many thanks. but after basecalled, generated fast5 files are still all together in one file (in the workspace) and named like FAP...fast5_skip.....fast5. is there any way to separate that fast5 files according to each barcodes because ı wanna use deepsignal and tombo packages? many thanks. :)

Sincerely.

Hi, I don't know. Guppy 6 should be demultiplexing your barcodes with the mentioned command. If it is not happening you can try the code in the gist above. Have a look at the sequencing_summary.txt file created by guppy and check if there are any 'barcode0x' in field 21 or if they are all 'unclassified' cat sequencing_summary.txt | cut -f 21 If there are barcodes, you should be able to use the script in this gist, but also, guppy should have done it.... If that leads to nothing you will have to ask this in the nanopore community, as both guppy and this script work for my setup, sorry.

…

On Thu, 17 Feb 2022 at 22:38, Fatih Tiras ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ Hi Johannes, Now it looks like working. many thanks. but after basecalled, generated fast5 files are still all together in one file (in the workspace) and named like FAP...fast5_skip.....fast5. is there any way to separate that fast5 files according to each barcodes because ı wanna use deepsignal and tombo packages? many thanks. :) [image: Slayt1] <https://user-images.githubusercontent.com/75265865/154503922-16cb4b6e-ed48-4ef4-af4e-76fec4ad244e.JPG> Sincerely. — Reply to this email directly, view it on GitHub <https://gist.github.com/4af1e37ac96af3d9ed2104658993d944#gistcomment-4069063>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBHB2WSPLGAKVRUNIPBM6TU3UB7ZANCNFSM5OOPDVPA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: <JWDebler/fast5 demultiplex by ***@***.***>

Hi Johannes,

I am still working to fix my problem but many thanks for your help. I have really appreciated your effort for help. have a great days.:)

Sincerely.

No worries. Let me know once you succeed. I'm interested to find out why my suggestions didn't work. Good luck.

…

On Mon, 21 Feb 2022, 20:14 Fatih Tiras, ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ Hi Johannes, I am still working to fix my problem but many thanks for your help. I have really appreciated your effort for help. have a great days.:) Sincerely. — Reply to this email directly, view it on GitHub <https://gist.github.com/4af1e37ac96af3d9ed2104658993d944#gistcomment-4072902>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBHB2VN7RUE22EZ2WAB3JDU4IUCXANCNFSM5OOPDVPA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: <JWDebler/fast5 demultiplex by ***@***.***>

Hi Johannes,

I wanna let you know that your code has worked on my last run and ı have been able to separate fast5 for each barcode. I dont know why but ıt has not been working my previous run (still generate - folder and put all fast5 in there like here nanoporetech/ont_fast5_api#68) so ı decided to basecall all my previous raw data again and see how it come up. I wanna personally say many thanks for your effort. have a great days.:)

Best wishes.

Hi Fatih, that's great to hear! Glad this code helped. Not sure why it didn't work at first, but glad you made it work! All the best for your experiment!

…

On Mon, 28 Feb 2022 at 20:28, Fatih Tiras ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ Hi Johannes, I wanna let you know that your code has worked on my last run and ı have been able to separate fast5 for each barcode. I dont know why but ıt has not been working my previous run (still generate - folder and put all fast5 in there like here nanoporetech/ont_fast5_api#68 <nanoporetech/ont_fast5_api#68>) so ı decided to basecall all my previous raw data again and see how it come up. I wanna personally say many thanks for your effort. have a great days.:) Best wishes. — Reply to this email directly, view it on GitHub <https://gist.github.com/4af1e37ac96af3d9ed2104658993d944#gistcomment-4080360>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBHB2S2GFQPYBIFO6RVJ33U5NS75ANCNFSM5OOPDVPA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: <JWDebler/fast5 demultiplex by ***@***.***>

hi @BeatrizFaustino

I used demux_fast5 --input fast5_skip/ --save_path ./demutiplexedfast5/ --summary_file sequencing_summary.txt command for demultiplex my fast_skip folder for barcoded. You should use Minknown sequencing summary file for input.

Hi @BeatrizFaustino ,

the sequence summary file that ı used is provided after sample runs (sequenced) from MiKnown generally sequence_summarry_FAPxxxxx.txt. it's not generated a file after base calling like a guppy.
Yes fast5_skip includes all fast5 that is generated after sequencing by MinKnown and they don't separate by MinKnown (or you stopped the basecalling after sequencing finished but basecalling was still continuing). I hope this help you. thanks

Hi
is demux present in guppy or other software?

hi @BeatrizFaustino ,

I used this page as a demultiplex ont_fast5_api

Hi @Fatihlrcfs
Thank you very much, using the commands you gave me the analyzes ended now and I was successful

Hi, yes, the latest version of minknow can also demultiplex fast5s, but it has to do it by basecalling. If you have a minknow setup with a GPU I would just recall the raw data with the super accurate model.

…

On Sat, 17 Dec 2022, 06:19 B13F, ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ Hi @Fatihlrcfs <https://github.com/Fatihlrcfs> Thank you very much, using the commands you gave me the analyzes ended now and I was successful — Reply to this email directly, view it on GitHub <https://gist.github.com/4af1e37ac96af3d9ed2104658993d944#gistcomment-4405609> or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBHB2TQ2K2BXD7IXPBWQS3WNTTIHBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFQKSXMYLMOVS2I5DSOVS2I3TBNVS3W5DIOJSWCZC7OBQXE5DJMNUXAYLOORPWCY3UNF3GS5DZVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVEYTCMZZG42TSNRTU52HE2LHM5SXFJTDOJSWC5DF> . You are receiving this email because you authored the thread. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .