Reading and Writing to Fifo (named pipes) in Erlang

read_write_fifo_erlang.md

edit

@akash-akya pointed out in the comments that the file module supports named pipes since OTP 20. So none of this post is really needed anymore if you are on >= OTP 21.

---

Erlang and Named Pipes

The intention of this post is to provide a solution (with examples) to a somewhat uncommon issue in Erlang. I hate searching for answers to the same problems over and over, and I had a hard time finding answers to this particular problem, so I wrote it all down once I figured it out. If one day you decide to read and write data through fifo's (named pipes) and then decide you want to read or write the other end of the pipe from Erlang, this post is for you.

The Problem

I wanted to read and write to a fifo from a C/C++ app and have an Erlang app communicate over the other end of that fifo. Put simply, Erlang doesn't really support what I was trying to do. You cannot just file:open/2 a fifo, or any special device for that matter, in Erlang and expect it to work. This is documented in Erlang's FAQ.

The Solution! ... ???

In that FAQ it offers two links that you'd imagine have some reasonable solution to the answer, but both posts do not really give you a good solution, they just further state the problem.

So, next you think, "hey, I'll Google this and surely someone has figured it out". Indeed, maybe people have figured it out and suggest using open_port() to solve all your woes. But alas even this post and this post take more of the "use open_port and that should work for you" stance. While this points you in the right direction, it isn't really helpful if you aren't familiar with ports.

So yes, open_port() is the solution, but getting it to work is non-obvious (at least to me) initially.

The Solution!

First lets setup what we want to do. In bash, we might do the following to read/write to a fifo.

Using just a shell

Create fifo

➜  cd /tmp
➜  mkfifo test.pipe
➜  test -p test.pipe && echo "It worked"
It worked

Reading from a fifo blocks, so you need a writer and reader to test that it is working

Shell 1

➜  cat test.pipe
... # blocking

Shell 2

➜  echo "Hey now" > test.pipe
# returns immediately

Meanwhile back in Shell 1

➜  cat test.pipe
Hey now
➜ # It now returns and closes its end of the pipe

In Erlang

Okay, so we know what is supposed to happen in the shell, now lets try it in Erlang. If you are looking for solid examples online, there really isn't any that deal directly with named pipes, and the one other port example I could find in Erlang's documentation didn't work with fifo's. All we know is "use open_port", cool story.

Instead of walking through all the terrible ways I've failed trying different things, I'll get right to the correct answer. TL;DR, none of the errors I got back were super helpful, just a lot of badsig or bad argument errors that are hinted at in the port_command/2 documentation.

To Read

First, make sure you created that pipe

➜  cd /tmp
➜  mkfifo test.pipe

Then open your Erlang shell

➜  cd /tmp
➜  erl
Erlang/OTP 17 [erts-6.3] [source] [64-bit] [smp:8:8] [async-threads:10] [kernel-poll:false] [dtrace]

Eshell V6.3  (abort with ^G)
1>

Okay, lets attempt to open test.pipe

1> Fifo = open_port("test.pipe", [eof]).
#Port<0.470>

Next, we cannot read from this like a file (see the FAQ linked above), so instead we need to use receive

1> Fifo = open_port("test.pipe", [eof]).
#Port<0.470>
2> receive
2>   {Fifo, {data, Data}} ->
2>     io:format("Got some data: ~p~n", [Data])
2>   end.
%% This is blocking, waiting for data

Back to your shell

➜  echo "Hello Erlang" > test.pipe
➜ 

Erlang now returns

1> Fifo = open_port("test.pipe", [eof]).
#Port<0.470>
2> receive
2>   {Fifo, {data, Data}} ->
2>     io:format("Got some data: ~p~n", [Data])
2>   end.
Got some data: "Hello Erlang\n"
ok
3>

Okay, sweet, that wasn't so hard. That is of course once you know you need to use receive and to know what to match against. Next up, lets write some stuff from Erlang.

To Write

Much like the previous example, make sure you actually have a fifo first.

To speed through, here's the bash side waiting to read

➜ cat test.pipe
➜ # ... blocking

In Erlang

Eshell V6.3  (abort with ^G)
1> Fifo = open_port("test.pipe", [eof]).
#Port<0.470>
2> port_command(Fifo, <<"Hello bash\n">>).
true
3> port_command(Fifo, <<"Howzit?\n">>).
true
4>

Back to Shell

➜ cat test.pipe
Hello bash
Howzit?

So that also works.

Some advice from my mistakes, what you pass into port_command is very important. The newline in <<"Hello bash\n">> is very important. I was leaving it off when sending data and nothing was working. My esteemed boss Andrew suggested using flush in the Erlang shell to flush out the mailbox, but that didn't work. He realized a simple \n was needed. It is also worth noting that passing in the [eof] flag to open_port keeps the port open, whereas if you just used the shell and echo'd, it'd close the port after it sent its line.

I know this wasn't a mind bending problem or solution, but one of those frustrating things you don't want to go searching for late at night when you just want to read and write from a damn pseudo-file.

@jaredmorrow thanks for writing this, it was a huge help!

Some notes on using this method for opening a fifo:

• use of a file name as the port name

The type spec for the port name in open_port/2 is:

PortName = {spawn, Command :: string() | binary()}
                          | {spawn_driver, Command :: string() | binary()}
                          | {spawn_executable, FileName :: file:name()}
                          | {fd,
                             In :: integer() >= 0,
                             Out :: integer() >= 0}

So using a filename for the port is undocumented and might break in the future. But it allows reading and writing to other types of files not supported by the erlang VM like character devices and probably unix sockets. For example:

1> open_port("/dev/random", []).
#Port<0.724>
2> flush().
Shell got {#Port<0.724>,
           {data,[30,0,243,29,10,203,113,95,135,69,13,152,24,195,73,213,77,94,
                  219,184,115,251,71,98,216,3,242,22,188,156,16,249,115,87,54,
                  117,218,254,84,177,228,187,101,189,23,76]}}
ok

• Erlang port as reader/writer

The write tests only work sporadically for me.

Running the example:

1> Fifo = open_port("test.pipe", [eof]).
#Port<0.724>
2> os:getpid().
"15320"

beam opens the fifo read/write:

$ lsof -p 15320 # 10u means fd 10 with read/write
beam.smp 15320 msantos   10u  FIFO   0,12      0t0 8672485 /tmp/test.pipe

On any write by beam, there is a race between beam and the process on the other end of the pipe to read the data from the fifo. The data written by beam can end up also being read by beam and put in the port's mailbox.

The in and out options can be used to specify write-only and read-only modes respectively. open_port/2 will block until a reader or a writer has attached to the fifo:

% erlang is the reader
1> open_port("test.pipe", [eof,in]).
#Port<0.724>

In the shell:

$ lsof -p 15852
beam.smp 15852 msantos   10r  FIFO   0,12      0t0 8672485 /tmp/test.pipe

$ echo 123 > test.pipe

The port exits with eof once the writer exits:

2> flush().
Shell got {#Port<0.724>,{data,"123\n"}}
Shell got {#Port<0.724>,eof}
ok

% erlang is the writer
1>  Fifo = open_port("test.pipe", [eof,out]). % blocks until the reader attaches
2> port_command(Fifo, <<"Hello bash\n">>).

In the shell:

$ lsof -p 16018
beam.smp 16018 msantos   10w  FIFO   0,12      0t0 8672485 /tmp/test.pipe

$ cat /tmp/test.pipe

• detecting EOF
  • erlang as fifo reader: when all the fifo writers have closed, the port will exit (or receive an 'EXIT' message if process_flags(trap_exit,true) is set) or receive an eof message if the eof option was specified.
  • erlang as writer: if the reader exits and the erlang side attempts to write to the fifo, the port with crash with reason epipe.

@msantos Thanks for the addition. I sure wish gists had email/subscribe options as I would've loved to thank you in a more timely manner.

Note that file module natively support named pipe since OTP 21.

fifo = File.open!("test.pipe", [:raw, :binary, :read])
{:ok, data} = :file.read(fifo, 100)

@akash-akya Excellent to hear they added that at some point. Thanks.

@jaredmorrow, its OTP-21 not 20. my bad. updated my reply