ComPort over Network

ComPort over Network.md

Connecting to serial port (com port) over network

(Serial port or com port? - Serial ports are often refered as COM ports. It is the same to be short. You can read abut it in the Wiki article )

• The problem
• The solution in theory
• WINDOWS
  • Server
  • Client
  • Driver Signature
• LINUX:
  • Server
  • Client
• Connect over the internet

The problem

Suppose we have an application that works with some device using serial port (com port or comport - the same thing). It could be GPS reader, IRDA, whatever. So it looks like this:

+--------+   serial   +--------------+
| DEVICE | ~~~~~~~~~~ | PC  with APP |
+--------+            +--------------+ 

Now what we want, is to have the device connected to one machine (server), and run the application on the remote machine (client) over the network. Real life example: a device is connected to raspberry pi (very small single-board machine) that is connected to a local network, and read the data on a desktop.

Since the application (APP on diagrams) knows only how to communicate with the device by serial port (we suppose), the client machine has to have some virtual serial port that is used by the application. It is called "virtual serial port" or "virtual comport" as this is a software emulated bridge between a client and your application. So the diagram is:

+--------+   comport  +--------+       network        +--------+  virtual comport +---+
| DEVICE | ~~~~~~~~~~ | SERVER |========....==========| CLIENT |~~~~~~~~~~~~~~~~~~|APP|
+--------+            +--------+                      +--------+                  +---+

Thus we need:

• SERVER that communicates with the DEVICE through physical serial port and then serves the data over network
• Client that connects to the server
• Virtual comport that mimics physical serial port and interface with the APP

So now the application just works with serial port on the client machine, and doesn't even know that data is actually transmitted over the network.

The solution in theory

One of the solutions is using telnet with RFC2217 - Telnet Com Port Control Option. Is solves exactly the problem above. There are a lot of software that supports telnet+RFC2217 serial port forwarding. It allows you to run the server and the client on linux or windows machines (and MACs I suppose, but haven't tested it). This allows one to run linux server and windows client. Both would use completely different software, but because of RFC2217 standard they 'know' how to communicate.

More over you can multiplex the com ports and encrypt the data. Whatever you want.

The solution in practice

WINDOWS

There is an absolutely brilliant free opensoure solution that can be used for comport forwarding, client and server for windows. It is called com0com. It actually consists of two parts a HUB (hub4com) and kernel-mode virtual serial port driver (com0com) - explained further.

http://sourceforge.net/projects/com0com

Server

For the server you need only hub4com.

Source forge hub4com download link

Configuration (I just cite the documentation):

You have a server computer with phisical COM1 port and you'd like to share it through the network by the RFC 2217 "Telnet Com Port Control Option" protocol:

Start the com2tcp-rfc2217.bat on COM1 port. For example:

com2tcp-rfc2217 COM1 7000

It will listen TCP/IP port 7000 for incaming connections and redirect them to COM1 port.

Client

To be a windows client you have to install com0com virtual comport driver and hub4com (provided as 2 separate files).

Source forge com0com download link
Source forge hub4com download link

Create a PAIR of virtual comports where one is used for RFC2217 and the other is the port for your application will use.

(documentation citation) for RFC 2217 client :

You have a server computer your.comport.server with physical serial port shared through the network by the RFC 2217 protocol (see above example) and you'd like to use it on the client computer like a virtual serial port.

With the com0com's Setup Command Prompt create COM19<->COM20 virtual COM port pair (see com0com's ReadMe.txt for more info). For example:

>setupc.exe
command> install PortName=COM19,EmuBR=yes PortName=COM20

Example. Start the com2tcp-rfc2217.bat on COM19 port:

com2tcp-rfc2217 \\.\COM19 192.168.123.30 7000

It will redirect virtual serial port COM20 on the second computer to the physical serial port on the first computer.

(!) TL;DR; Your Application should connect to COM20.

Explanation

TL;DR; We have a virtual serial port pair COM19<->COM20, we connect com2tcp to one of the ports (COM19) and your application connects to the other (COM20)

It is bit counter intuitive why a virtual pair is created. To explain what happens, imagine we create a virtual comport pair: COM19<->COM20 as in the example above. The reason one needs a pair is that only one thing can be connected to a COM port. If hub4com binds network data to COM19, then the port is taken and your application can't connect to it. So virtual comport pair mirrors everything from COM19 to COM20, which is free and your application can connect to it.

So, the more detailed client diagram looks like this now:

     network  +---------+   +-------+-------+            +---+
....==========| hub4com |~~~| COM19 = COM20 |~~~~~~~~~~~~|APP|
              +---------+   +-------+-------+            +---+
                              virtual pair

Driver Signature

Deprecated part - com0com 3.0.0 comes with driver signarure. Unfortunately on newer windows (since 2018) this doesn't help.

According to Windows Driver Signing Policy "Starting with Windows 10, version 1607, Windows will not load any new kernel-mode drivers which are not signed by the Dev Portal." (link)[https://sourceforge.net/p/com0com/discussion/440109/thread/c4d52f1b/?limit=25]

There are 3 solutions:

• Disable driver signature verification (bad)
• Use version 2.2 (it works)
• Use DSEO

(older problem) According to this bug report on Windows 8x64 you may get problem with driver installation if you don't have the driver signature verification turned off. To enable driver test mode and sign a driver for windows, one may download DSEO

LINUX:

The linux app I've got working pretty easy is ser2net

http://linux.die.net/man/8/ser2net

It has configuration file located at /etc/ser2net.conf.

Ubuntu installation

sudo apt-get ser2net            #install
sudo vim /etc/ser2net.conf      #configure
ser2net                         #run service

Linux Server

The configuration line (for /etc/ser2net.conf) that corresponds to windows setup above

7000:telnet:0:/dev/ttyUSB0:1000000 8DATABITS NONE 1STOPBIT remctl

Here:

1. 7000 - port
2. /dev/ttyUSB0 - name of serial port
3. 1000000 ... - baud rate etc (actually you can skip it because of remctl)
4. remctl - means using remote port configuration as of RFC2217

That is it. Read ser2net docs for more

Linux Client

socat can be used as a linux client. Socat is a command line based utility that establishes two bidirectional byte streams and transfers data between them.

socat /dev/ttyS2,b115200,raw,echo=0 TCP:192.168.123.30:7000

• 192.168.123.30 - your server IP
• 7000 server port
• your app should connect to /dev/ttyS2
• 115200 - port baud rate

socat man

Connect over the internet

All the above solutions basically describe how to forward data from com-port (serial port) to network port and then how to bind a network IP with port to a virtual com-port. It is streight forward for a local network but how to connect the devices over the internet? If you are an experiecned IT person, you may think of tons of solutions here, starting from fixed IPs and counting up to infinity.

For me one of the versatile easy to configure ways was to use ZeroTier VPN services. It is free for up to 100 devices and have a good interface for easy configuration, good manuals, etc. In the end you have a network interface on each of the machines which acts as a single local network (basically what VPN is).

I'm NOT connected anyhow with Zerotier co. Just share the solution which was optimal for me. "As is".

@popliviustefan

The one thing that doesn't work is upload a new sketch to the Arduino board.

Your com2tcp-rfc2217 printout also showing a healthy connection. I don't know what to blame here, but from my experience of using the tool, it might be beneficial to check if BaudRate (and other parameters) are enforced.

Hm... After googling I found that DTR signal is important. found here

Later versions of Arduino made use of an additional serial protocol wire called DTR to reset the Arduino automatically. Since the DTR signal goes from 5V to 0 at the moment a new connection to the Arduino is started, if you send this signal to the reset line of the Arduino's processor, it resets. If reason the new serial connection is being established is because the Arduino IDE is sending a sketch to the bootloader, the bootloader goes ahead and accepts the sketch and stores it. If reason the new serial connection is being established is because you're opening the serial monitor of the Arduino IDE, then the Arduino resets, the bootloader runs just long enough to realize the IDE is not attempting to send a sketch, and the sketch you previously uploaded starts running.

So you have to ensure that DTR signal is emulated (not by default)

@DraTeots

So you have to ensure that DTR signal is emulated (not by default)

Hi again, and thank you for your research!
I tried to do this, but I don't think I have a deep enough understanding of the system.
I thought that I could add DTR to the command i used to create the port pair
or to the command used to bind the local com port to the remote port.
But I couldn't figure out how to do this.

So, if it's a simple config, can you elaborate about how to configure the ports to send also the DTR signal?

Thanks again!

@popliviustefan, there are 2 steps:

1. Ser2net should correctly transfer those signals
2. Null modem should correctly connect them between virtual ports

For ser2net I found 2 links, IDK if you've seen those:

1. Ser2net configuration for Arduino IDE
2. How to use ser2net directly from avrdude

The later is easy, you just enable it in the GUI (probably that is what you've done)

P.S. Please, update us here if you have a success and how you made it. I think it would be a good addition to the text.

Hi, for the moment no luck, even with the configuration from the first link...
If/when I'll figure this out, I'll sure post a detailed explanation.
I'm sure it will be useful to someone!

About point 2: I have the signals connected as in the default config

@popliviustefan well, I already forgot, what those red indicators mean: do you have your DTR emulation enabled between virtual ports or disabled?

I am having issue connecting one of the three devices, two other ones work.

On remote server, I have
com2tcp-rfc2217.bat \\.\COM16 7001
com2tcp-rfc2217.bat \\.\COM17 7000
com2tcp-rfc2217.bat \\.\COM30 7002 or com2tcp-rfc2217.bat \\.\COM15 7002 <<<<< both setups don't work
COM15, 16, 17 are physical ports.
And 'COM30' is a virtual port directing from COM15.( this is a second setup I tried)

On my PC that remotes in the server, I have
com2tcp-rfc2217.bat \\.\COM16 10.1.10.99 7001
com2tcp-rfc2217.bat \\.\COM17 10.1.10.99 7000
com2tcp-rfc2217.bat \\.\COM30 10.1.10.99 7002 or com2tcp-rfc2217.bat \\.\COM15 10.1.10.99 7002 <<<<< both setups don't work
And I have virtual ports going from COM26 to COM16, COM27 to COM17. I can connect to these with Tera Term without issues.
However when I tried COM15 with one of the setup above, it doesn't work, and I think it's because my PC it has built in Bluetooth that uses COM15. I tried disabling Bluetooth in device manager and COM15 doesn't show up as a comport anymore, but it still doesn't work. It would give me invalid handle:

C:\Users\andre\workspace\remote_serial\hub4com-2.1.0.0-386>com2tcp-rfc2217.bat \.\COM15 10.1.10.99 7002
C:\Users\andre\workspace\remote_serial\hub4com-2.1.0.0-386>"hub4com" --create-filter=escparse,com,parse --create-
filter=pinmap,com,pinmap:"--rts=cts --dtr=dsr" --create-filter=linectl,com,lc:"--br=local --lc=local" --add-filters=0:com --create-filter=telnet,tcp,telnet:" --comport=client" --create-filter=pinmap,tcp,pinmap:"--rts=cts --dtr=dsr --break=break" --create-filter=linectl,tcp,lc:"--br=remote --lc=remote" --add-filters=1:tcp --octs=off "\.\COM15" --use-driver=tcp "*10.1.10.99:7002"
ComIo::OpenPath(): CreateFile("\.\COM15") ERROR 2 - The system cannot find the file specified.
ComPort::Init(): Invalid handle

I then tried to direct COM15 to COM30 on server then use COM30 on my PC and direct it to another comport COM31 but that doesn't work either. The call to com2tcp-rfc2217.bat is successful but there's no message being transmitted in between the server and client except when the Tera Term connects or disconnect from COM31.

Does anyone know why this third one that I am trying isn't work while the other two work perfectly fine?

I would only suggest to separate and localize the problem. Try different physical devices are working for different network port connections. So you could figure out if it is com2tcp-rfc2217.bat problem, or virtual ports or physical ports,

Hi,

I am trying to use com0com to achieve the following:
program an Arduino Mega board connected to a raspberry pi. The pi (and arduino) are in a remote location, but the pi is on the same lan as the windows pc from where I try to achieve this.

on the pi I start the server with the following command:
ser2net -C 192.168.1.60,17000:raw:0:/dev/ttyACM0:9600 -d -P /home/pi/pidCOM

On the windows computer I have a pair of virtual com ports as follows:
CNCBUS0 FriendlyName="com0com - bus for serial port pair emulator 0 (COM11 <-> CNCB0)"
CNCA0 FriendlyName="com0com - serial port emulator CNCA0 (COM11)"
CNCB0 FriendlyName="com0com - serial port emulator CNCB0 (CNCB0)"

I connect to the server with the following command:
com2tcp-rfc2217 .\CNCB0 192.168.1.60 17000
and I get the following output:

"hub4com" --create-filter=escparse,com,parse --create-filter=pinmap,com,pinmap:"--rts=cts --dtr=dsr" --create-filter=linectl,com,lc:"--br=local --lc=local" --add-filters=0:com --create-filter=telnet,tcp,telnet:" --comport=client" --create-filter=pinmap,tcp,pinmap:"--rts=cts --dtr=dsr --break=break" --create-filter=linectl,tcp,lc:"--br=remote --lc=remote" --add-filters=1:tcp --octs=off ".\CNCB0" --use-driver=tcp "*192.168.1.60:17000"
CNCB0 Open(".\CNCB0", baud=19200, data=8, parity=no, stop=1, octs=off, odsr=off, ox=off, ix=off, idsr=off, ito=0) - OK
Route data CNCB0(0) --> TCP(1)
Route data TCP(1) --> CNCB0(0)
Route flow control CNCB0(0) --> TCP(1)
Route flow control TCP(1) --> CNCB0(0)
Filters:

     \->{parse.IN}----------------->

CNCB0(0) | /
_________/<-----{pinmap.OUT}<-{lc.OUT}<-

   \->{telnet.IN}------------------------------>

TCP(1) | /
_______/<-----{telnet.OUT}<-{pinmap.OUT}<-{lc.OUT}<-

Socket(0.0.0.0:0) = 168
TCP(1): Connect(168, 192.168.1.60:17000) ...
Started CNCB0(0)
Started TCP(1)
TCP(1): Connected
TCP(1) START
TCP(1) SEND: WILL 44
TCP(1) SEND: SB 44
1 0 0 37 128 SE
TCP(1) SEND: SB 44
2 8 SE
TCP(1) SEND: SB 44
3 1 SE
TCP(1) SEND: SB 44
4 1 SE
TCP(1) SEND: SB 44
5 9 SE
TCP(1) SEND: SB 44
5 12 SE
TCP(1) SEND: SB 44
5 6 SE
TCP(1) SEND: SB 44
8 SE

At this point the Arduino Monitor works fine (I can see the data transmitted by the Arduino sketch that runs on the board and I can also send commands to the sketch (that are handled correctly).

The one thing that doesn't work is upload a new sketch to the Arduino board.

I would be glad for any input from someone who confronted with this problem before!

Thanks!

PS: I think I made a little progress in figuring out why the sketch upload doesn't work: usually when opening the Serial Monitor the Arduino sketch is restarted.
But this is not happening when the COM port is forwarded.
So the first thing to figure out: how to replicate this behavior of Arduino IDE: restart the Arduino.

So, I finally solved this problem! (By that I mean I successfully applied a suggested workflow)

On the linux side (the raspberry pi computer) instead of using ser2net as a server I used this python script that creates the server:
python script link

Script is executed with the port that will be forwarded (the port where arduino is connected)
In my case the command was: ./rfc2217_server.py /dev/ttyACM0
This creates the server listening on the default port 2217

Now, on the windows side, we can follow the exact steps for the client as shown here but we connect to port 2217 instead of 17000.
That's it!
For me, at least, I can now connect to the serial monitor and upload new sketches to the arduino as well!

I found the solution for the linux server part in this blog post:
original solution

I hope it will help anyone who needs this, as I really needed it and it took me quite a long time until I found this working solution!

Thank you for the very helpful article.
However, I simplified things so even a "dum-dum" like me can understand it, like this:

Server side:
com0com : COM1<>CNCB0
hub4com \.\CNCB0 --use-driver=tcp 7000

Client side:
com0com : COM1<>CNCB0
hub4com \.\CNCB0 --use-driver=tcp 192.168.20.3:7000

On both computers you can now use COM1 to send data to the other over your network.
Note: Change the IP address to that of your Server computer.
I recommend using RealTerm to test this.

If anyone is using ser2net version 4.X on Linux (like a Raspberry Pi in my case), it's configured with ser2net.yaml instead of ser2net.conf. I added the following in /etc/ser2net.yaml and it worked with the Windows client following the author's directions:

connection: &con2096
accepter: telnet(rfc2217),tcp,7000
connector: serialdev,/dev/ttyUSB0,local
enable: on

At least on Raspberry pi it also installed it as a service, so you don't run it with ser2net on the command line. Instead you use systemctl start ser2net or systemctl restart ser2net if it's already running. Make sure to do systemctl status ser2net to see that status and make sure there's no errors in your configuration before trying to connect a client to it. netstat -an | grep 7000 is another useful test to make sure it's listening.

Maybe I can help somebody:
Server:
Linux with real device on /dev/ttyS0
Client:
Windows with virtual device on COM19

how to change default baud rate and stop bits for com2tcp-rfc2217?

See under the green line 9600n81

this is for the client side which is in windows, ser2net i think is for server

Perfect!

server (where the device is plugged in, can be mac os x, windows, android, etc.)

use rfc2217_rs::Server; // https://github.com/esp-rs/rfc2217-rs

fn main() {
    // init logging
    env_logger::Builder::from_env(env_logger::Env::default().default_filter_or("debug")).init();
    loop {
        // spawn servers in a loop because they stop after first connection exits
        let serial_port_name = "/dev/tty.usbmodem103"; // TODO: find dynamically
        let tcp_addr = "0.0.0.0:48001"; // Wireguard IP
        let mut server = Server::new(serial_port_name, tcp_addr).unwrap();
        loop {
            match server.run() {
                Ok(()) => (),
                Err(err) => {
                    match err {
                        rfc2217_rs::server::Error::Tcp(tcp_error) => {
                            match tcp_error.kind() {
                                std::io::ErrorKind::ConnectionReset => {
                                    break;
                                }
                                _ => panic!("{tcp_error:?}")
                            }
                        }
                        _ => panic!("{err:?}")
                    }
                },
            }
        }
    }
}

client (where you want the device to show up)

com0com setup GUI
  virtual port pair 1
  com3 + com6
  check "emulate baud rate" for both, press apply

from hub4spot examples:
  .\com2tcp-rfc2217.bat \\.\COM6 10.5.5.3 48001

  where 10.5.5.3 is your server IP

in your application, use COM3 (which is paired to COM6 but not taken, COM6 gets "access is denied" because only one application can bind/connect to it at a time?)

@brandonros Excellent! Thank you! I actually also now use small and fast code snippets to pass SerialPort messages over internet/network but I use C#/.net for that. Probably it would be good Idea to add your comment in the main article and maybe collect a couple of more examples in different languages. What do you think?