How to stream a CSV file from a large QuerySet using Django's StreamingHttpResponse

django_stream_queryset_to_csv.md

Stream a CSV file from a QuerySet using StreamingHttpResponse

This is a sample on how to stream the results of a large QuerySet into a CSV file using Django StreamingHttpResponse class.

1. Add the CSVStream class in your project, for example a writers.py file:

import csv
from django.http import StreamingHttpResponse

class CSVBuffer:
    """An object that implements just the write method of the file-like
    interface.
    """
    def write(self, value):
        """Return the string to write."""
        return value

class CSVStream:
    """Class to stream (download) an iterator to a 
    CSV file."""
    def export(self, filename, iterator, serializer):
        # 1. Create our writer object with the pseudo buffer
        writer = csv.writer(CSVBuffer())

        # 2. Create the StreamingHttpResponse using our iterator as streaming content
        response = StreamingHttpResponse((writer.writerow(serializer(data)) for data in iterator),
                                         content_type="text/csv")

        # 3. Add additional headers to the response
        response['Content-Disposition'] = f"attachment; filename={filename}.csv"
        # 4. Return the response
        return response

2. On your views.py file, implement the CSVStream class as follows:

from rest_framework import viewsets
from .models import MyModel
from .writers import CSVStream

def csv_serializer(data):
    # Format the row to append to the CSV file
    return [
        data.id,
        data.some_field,
        ...
    ]

class MyViewSet(viewsets.ViewSet):
    def list(self, request):
        # 1. Get the iterator of the QuerySet
        iterator = MyModel.objects.iterator()

        # 2. Create the instance of our CSVStream class
        csv_stream = CSVStream()

        # 3. Stream (download) the file
        return csv_stream.export("myfile", iterator, csv_serializer)

Hello, how could I insert the names of the model columns in the header?

Hello, how could I insert the names of the model columns in the header?

class CSVBuffer:
    """An object that implements just the write method of the file-like
    interface.
    """
    def write(self, value):
        """Return the string to write."""
        return value

class CSVStream:
    """Class to stream (download) an iterator to a
    CSV file."""
    def getHttpResponse(self, filename, iterator, columns,headers = None):
        writer = csv.writer(CSVBuffer())

        def stream():
            if headers:
                yield writer.writerow(headers)
            for data in iterator:
                yield writer.writerow(columns(data))

        response = StreamingHttpResponse(stream(), content_type="text/csv")

        response['Content-Disposition'] = f"attachment; filename={filename}.csv"
        return response

Hi, thanks so much for this! Your method is much neater than the toy example in the Django doc's, and not that much more code to write - so win win!