read data from cloud object store with biopython

read_from_s3.py

"""
Reads a fastq file directly from the 1000genomes AWS public dataset
into a Bio.SeqRecord set

Requires an AWS Account
"""

from smart_open import open
from Bio import SeqIO

# file handle-like reference to ~60MB object in S3
fh = open('s3://1000genomes/phase3/data/NA12878/sequence_read/SRR622461.filt.fastq.gz')

for record in SeqIO.parse(fh, 'fastq'):
    print(record.id)

read_from_s3_s3fs.py

"""
Reads a fastq file directly from the 1000genomes AWS public dataset
into a Bio.SeqRecord set

Does not require an AWS Account
"""
import io
from gzip import GzipFile

import s3fs
from Bio import SeqIO

fs = s3fs.S3FileSystem(anon=True)

with fs.open('1000genomes/phase3/data/NA12878/sequence_read/SRR622461.filt.fastq.gz','rb') as f:
     for record in SeqIO.parse(io.TextIOWrapper(GzipFile(fileobj=f)), 'fastq'):
         print(record.id)

An alternative to smart_open would be to use s3fs. The later is based on the newer boto3 package.

s3fs allows anonymous reads from public s3 buckets, which allows testing against available AWS Public Datasets like 1000genomes.

Anonymous access sounds great for an example in the main Biopython tutorial 👍

I wonder if smart_open will be updated to use boto3 and then support anonymous access?

Looks like smart_open does use boto3, but does not provide anonymous access to S3. That feature seems unique to s3fs.