IOCTLs and _IOC outputs in strace

ioctls.md

I'm trying to figure out exactly what code is being called on the other side of this bluetooth/bluez btattach call using strace. While I'm pretty sure I have guessed the code that is handling it from context, I'd like to still understand the parts that go into this.

Command am running:

sudo strace btattach -P bcm -B hci_pty2

The system calls and signals output by strace are somewhat familar to me. It's fun to see under the hood sometimes. I've not played with the 'tampering' support in strace but I think it would be fun too. Tracing IOCTLs back to their probably relevant handlers is easy enough:

ioctl(4, TCFLSH, TCIOFLUSH)             = 0
ioctl(4, TIOCGETD, [15])                = 0

Strace does a nice job of filling in a lot of the parameters. With -X raw the above comes out as:

ioctl(4, 0x540b, 0x2)                   = 0
ioctl(4, 0x5424, [15])                  = 0

And I see that 0x540b and 0x5424 match up neatly with the first outputs in kernel header file ./include/uapi/asm-generic/ioctls.h:

// ./include/uapi/asm-generic/ioctls.h

#define TCFLSH		0x540B
...
#define TIOCGETD	0x5424

// ./include/uapi/asm-generic/termbits-common.h
/* tcflush() QUEUE_SELECTOR argument and TCFLSH use these */
#define TCIFLUSH	0		/* Discard data received but not yet read */
#define TCOFLUSH	1		/* Discard data written but not yet sent */
#define TCIOFLUSH	2		/* Discard all pending data */

So I can read those strace outouts as "flush the tty buffer associated with file descriptor 4 and discard anything not yet written to it" then "get the line discipline of the tty associated with file descriptor 4 ". Both calls return a success result (= 0) and the second call populates the out parameter with the value 15.

The manpage for ioctls was quite helpful here.

There were a couple of IOCTLs that strace could not fully decipher:

ioctl(4, _IOC(_IOC_WRITE, 0x55, 0xcb, 0x4), 0x2) = 0
ioctl(4, _IOC(_IOC_WRITE, 0x55, 0xc8, 0x4), 0x7) = -1 EOPNOTSUPP (Operation not supported)

In raw form:

ioctl(4, 0x400455cb, 0x2)               = 0
ioctl(4, 0x400455c8, 0x7)               = -1 EOPNOTSUPP (Operation not supported)

Which was annoying because these were the ones that were breaking for me so it would be great to take a look at what was happening here.

The documentation on ioctl based interfaces was particuarly useful here.

It seems the 'modern' way to define an IOCTL is not just to define it as an arbitrary number, but to use one of the macros _IO/_IOR/_IOW/_IOWR which are defined in include/uapi/asm-generic/ioctl.h as follows

#define _IO(type,nr)		_IOC(_IOC_NONE,(type),(nr),0)
#define _IOR(type,nr,size)	_IOC(_IOC_READ,(type),(nr),(_IOC_TYPECHECK(size)))
#define _IOW(type,nr,size)	_IOC(_IOC_WRITE,(type),(nr),(_IOC_TYPECHECK(size)))
#define _IOWR(type,nr,size)	_IOC(_IOC_READ|_IOC_WRITE,(type),(nr),(_IOC_TYPECHECK(size)))

So I can rewrite those strace outputs :

ioctl(4, _IOW(0x55, 0xcb, 0x4), 0x2) = 0
ioctl(4, _IOW(0x55, 0xc8, 0x4), 0x7) = -1 EOPNOTSUPP (Operation not supported)

So "Call the IOCTL type 0x55, command 0xcb, with a param of size/type 0x4 set to a value of 0x2" and "Call the IOCTL type 0x55, command 0c8, with a param of size/type 0x4 set to a value of 0x7".

According to the above linked docs, commands are to be unique within types. Types are documented in the user-space Ioctl Numbers documentation (Ioctl vs IOCTL vs ioctl - I've seen all variations now I believe).

Our given type of 0x55, ascii 'U':

'U'   all    sound/asound.h                                          conflict!
'U'   00-CF  linux/uinput.h                                          conflict!
'U'   00-EF  linux/usbdevice_fs.h
'U'   C0-CF  drivers/bluetooth/hci_uart.h

I do not know how it disambiguates between these conflicting ranges here. I am sure it all works out :/. But it looks like unsuprisingly for our context, both those codes are defined in drivers/bluetooth/hci_uart.h.

#define HCIUARTSETPROTO		_IOW('U', 200, int)
#define HCIUARTGETPROTO		_IOR('U', 201, int)
#define HCIUARTGETDEVICE	_IOR('U', 202, int)
#define HCIUARTSETFLAGS		_IOW('U', 203, int)
#define HCIUARTGETFLAGS		_IOR('U', 204, int)

So 0xcb and 0xc8 (203 and 200) are HCIUARTSETFLAGS and HCIUARTSETPROTO. Meaning we can finally translate to :

1. "Set HCI UART flags to 0x2 on file descriptor 4"
2. "Set HCI UART protocol to 0x7 on file descriptor 4"

From drivers/bluetooth/hci_ldisc.c

case HCIUARTSETPROTO:
		if (!test_and_set_bit(HCI_UART_PROTO_SET, &hu->flags)) {
			err = hci_uart_set_proto(hu, arg);
			if (err)
				clear_bit(HCI_UART_PROTO_SET, &hu->flags);
		} else
			err = -EBUSY;
		break;
        
...


This does not give me the answer yet but it's a good start and I can drill down from here.