ECS task roles are a great security feature that are hard to set up.
The Amazon ECS documentation on setting up task roles tells you to do some questionable things. Among other things, it tells you to run the ECS agent with host networking (a security risk), use an iptables rule to cut off traffic from bridged containers to the host metadata (brittle), and set up additional iptables rules and sysctl settings to route 169.254.170.2:80 to the ECS agent on 127.0.0.1:51679 (brittle again).
Setting up things like sysctl settings and iptables rules varies a lot by host platform. Inevitably, Amazon's instructions are written for Amazon Linux and for any other distribution you're on your own. That's not terrible, but bootscripts are a solution of last resort for provisioning and on my distributions of choice, there are better options.
The whole point of the latter iptables rules and sysctl setting is effectively to bind a container port to a host port. Docker has this functionality built-in. The oddball IP address is a complication, but it's a minor one.
First of all, your container host needs to understand that it can respond to the 169.254.170.2 destination IP. This is easy. Just add the IP address to your loopback device. Here's what that looks like in a CoreOS cloud-config user data file:
coreos:
units:
- name: 00-loopback.network
content: |
[Match]
Name=lo
[Network]
Address=127.0.0.1/8
Address=169.254.170.2/32
And in a Debian or Ubuntu /etc/network/interfaces file (provision however you like, I use puppet):
auto lo
iface lo inet loopback
iface lo inet static
address 169.254.70.2
netmask 255.255.255.255
And in a bootscript:
ip addr add dev lo 169.254.170.2/32
Second, when you run the ECS agent replace --net host with -p 169.254.170.2:80:51679 to get docker to forward the port.
Just like that, you've got your ECS agent listening on 169.254.170.2:80 without host networking.
You'll still need to block container access to the host metadata, and it's somewhat more complicated now because the ECS agent itself will still need access to it but is now huddled on the bridge with the containers you want to block. Sorry, no good solution for this yet. I think you could run the agent on its own bridge and block anything else from fetching the host metadata, but this puts us back into iptables.
With docker publisheing the ECS agent task metadata port at 169.254.170.2:80, the ECS agent itself sees port 80 as "occupied" on the host. If you've got some other container that you wanted to bind to 0.0.0.0:80 you're not completely out of luck, but this definitely seems like a bug in the ECS agent. I solved this with an Application Load Balancer, which is an evolution of an ELB that includes backend port awareness and allows a docker container managed by an ECS service to have a dynamic port binding. I think Amazon could fix this by recognizing 169.254.170.2:80 and 0.0.0.0:80 as separate, non-colliding ports.
Why is iptables brittle? I generally avoid messing with filtering on container hosts because docker inserts rules and pretends that it owns the tables. On a bad day in 2017, I lost a bunch of rules because the docker daemon restarted. Naturally, the right solution was to shoot the host and let the cluster recover, but the whole situation could have been avoided by letting docker own the netfilter tables and finding other solutions to my other problems (such solutions absolutely exist).