Desktop operating systems provide powerful abstractions for moving data between software components. In POSIX systems, this is commonly achieved using file descriptors together with read() and write() operations.
For example:
- a generator writes data to a file descriptor,
- while a consumer reads data from it.
These operations typically execute concurrently on different threads, while the kernel buffers and coordinates the transfer.
Embedded systems are often very different:
- there may be no operating system,
- no file descriptor abstraction,
- limited RAM,
- and insufficient resources for multithreaded buffering.
As a result, stream plumbing in embedded systems becomes a more explicit architectural problem.
This document explores a practical approach to stream interfaces for embedded platforms.
POSIX-style interfaces typically use a file descriptor as context:
ssize_t read(int fd, void *buf, size_t count);
ssize_t write(int fd, const void *buf, size_t count);In embedded systems, file descriptors may not exist. A common alternative is to replace the file descriptor with an opaque context pointer:
ssize_t read(void *ctx, void *buf, size_t count);
ssize_t write(void *ctx, const void *buf, size_t count);This approach preserves much of the simplicity and familiarity of POSIX APIs.
It also allows easy desktop shimming during testing. For example, a POSIX file descriptor can simply be stored in an integer and passed via ctx.
One drawback is that both ctx and buf are pointer types. Accidentally swapping them may not generate a compiler warning:
read(buf, ctx, count);This requires careful adherence to argument ordering conventions.
Alternative approaches include opaque typed handles or wrapping the operations in a stream object structure.
Without threads or kernel-managed buffering, embedded transfers generally execute from one side or the other.
Two models emerge:
The generator executes the transfer by writing data into the consumer.
ssize_t write(void *ctx, const void *buf, size_t count);The consumer accepts some amount of data up to count.
This model suits unsolicited or asynchronous data sources, such as:
- UART receive streams,
- ADC sampling,
- interrupt-driven events.
The consumer executes the transfer by requesting data from the generator.
ssize_t read(void *ctx, void *buf, size_t count);The generator provides some amount of data up to count.
This model suits solicited data sources, such as:
- sensor polling,
- flash memory reads,
- protocol responses,
When plumbing modules together, it is common to encounter incompatible transfer styles.
For example:
- a generator may expose a
read()interface, - while a consumer expects a
write()interface.
The application is then forced to act as an intermediary:
- read data into a temporary buffer,
- write some of that data to the consumer,
- track remaining buffered data,
- repeat until complete.
This approach has several disadvantages:
- additional RAM consumption,
- double-copying of data,
- fragmented transfers,
On constrained embedded systems, these costs are significant.
One solution is to allow stream transfers to directly invoke the opposite endpoint.
For example, in addition to:
ssize_t read(void *ctx, void *buf, size_t count);a generator may also expose:
ssize_t read_to(
void *ctx,
ssize_t (*write_fptr)(void*, const void*, size_t),
void *write_ctx
);The application can then directly connect a generator to a consumer:
generator.read_to(generator_ctx, consumer.write, consumer_ctx);The generator pushes data directly into the consumer without requiring an intermediate buffer.
Likewise, a consumer may expose:
ssize_t write_from(
void *ctx,
ssize_t (*read_fptr)(void*, void*, size_t),
void *read_ctx
);allowing direct pull-based transfers from generators.
This approach provides several benefits:
- reduced RAM usage,
- fewer intermediate copies,
- larger uninterrupted transfers,
- reduced application complexity,
- improved composability between modules.
It also allows transfer ownership to remain with the component most naturally suited to driving the transfer.
The interfaces described here are intentionally agnostic to blocking behaviour.
A read() or write() operation may be:
- blocking,
- non-blocking,
- interrupt-driven,
- DMA-backed,
The interfaces merely define which side executes the transfer.
Likewise, callback-based adaptation inherits the execution behaviour of the supplied callback. For example, a read_to() operation using a blocking write() callback will itself behave as a blocking transfer.
This separation allows stream plumbing semantics to remain independent from scheduling.
The distinction between push and pull streams is often hidden on desktop operating systems because:
- threads execute concurrently,
- kernels buffer data,
- and file descriptors abstract transfer ownership.
Embedded systems expose these details more directly.
As a result, stream plumbing becomes an architectural concern rather than merely an implementation detail.
The callback-based adaptation approach described here can be viewed as a lightweight mechanism for composing stream-oriented modules without requiring heavyweight buffering or operating system services.