Funny(ish) issue with docker and host network
For the past weeks I have been using my precious spare time for modernizing my old hobby project cavez (pardon me if the page is not up any longer) and deploying it to this domain. Due to unfamiliar tech stack (WebRTC, STUN server, reverse proxy), the deployment was not so smooth in the end. I had a lot of issues, and this post is about one particular issue: using docker container in host mode in a cloud server. If you don’t care about the rambling and just want to check what my issue was, jump directly to the last paragraph.
This project is a game draft which tries to utilize UDP in a browser game instead of TCP sockets. Connections are made using geckos.io and there is a nodejs server both hosting the client page and maintaining the game room. Node is hosted on a docker container reverse proxied to a subdomain.
Geckos.io needs a open a single TCP port for both serving the html for the client and handling the initial handshake of client and server. Geckos.io and WebRTC in general needs UDP ports for the communication. My approach was dead simple: open the needed ports in a docker-compose and we are good to go.
ports:
- '1234:1234/tcp'
- '19900:20000/udp'
However, this did not work out in the end. The initial connection was made successfully, but STUN session was never established properly. I was stuck here for a long time just blindly trying out things, until I found out that firefox has quite nice built-in debugging features for WebRTC. Just type about:webrtc to your browser and you can see session specific logs. When I was blindly trying out things, I also tried host network mode, but that way I was not even able to server the html. I just shrugged and thought that maybe it’s nodejs doing something stupid, but when STUN establishing kept failing I started to study the issue a bit. Quickly I realized, that if I just setup a simple python webserver to my cloud host, I am not able to access it from the outside. Something is definitely wrong.
Many times during the troubleshooting I checked that my cloud provider does not use any firewalls and all (well, almost all) ports are open by default. I recall I have not installed firewall to my ubuntu server either and firewall does not make sense anyway, because when using docker network and port forwarding, everything works from the outside. But this is where I got tricked badly: docker network does not respect software firewall, or at least uncomplicated firewall, aka ufw. I read that it is because docker modifies iptables directly and bypasses ufw completely. This confused me enough to not double check that do I have a firewall enabled, and led me to blindly assume fault must be something very shady.
So in the end, I had completely forgotten that my partially copypaste cloud-init contains minimal ufw configuration - the culprit (me) had indeed enabled a software firewall with no holes punched besides for the ssh connection. After opening the desired tcp and udp ports to ufw and switching my docker network mode to host, my project started to breathe and connections are now succesful. I know this is not the optimal solution and there is still why (it works only on host mode) to be answered, but I am happy to be able to share the game draft for now. I suspect it is the reverse proxy (traefik) and udp, but it might be that I will never get back to this topic.
Moral of the story: double check firewall configuration, even if you think you do not have one enabled.