Running tcpdump inside Docker Swarm

Photo by Taylor Vick on Unsplash

Have you ever deployed a stack in Docker Swarm and wanted to run tcpdump to monitor packets that are passing between different network components? I have.

If you run the tcpdump command directly on the host you won’t monitor most of the packets that are moving between different services can containers. You need to somehow go inside the networks and runt it.

When you run a docker stack, you will have one ingress network and one overlay network. They each have their role. You can read about them here.

So how to run tcpdump in Docker Swarm and monitor the connection between different services and containers? You may need to run multiple tcpdumps. One for the ingress network and one for overlay network. For each of them, you need to use nsenter command and sh into a container. It might be easier to use another Docker container like nicolaka/netshoot. But first, find the ID for ingress and overlay network by listing networks in the host:

docker network ls
ujgie39kofp2          ingress-network          swarm
nmo87n20s1s4          overlay-network          swarm

Then for running nicolaka/netshoot container, use this command:

docker run -it --rm -v /var/run/docker/netns:/var/run/docker/netns --privileged=true nicolaka/netshoot

You need to map /var/run/docker/netns in the container and use the privileged mode. Then you can see a list of networks in the /var/run/docker/netns directory (run a ls to see them). You must see many IDs but you will find these twos:

1-ujgie39kof
1-nmo87n20s1

Which their IDs are showing they are the overlay and ingress network. Let’s say we want to run tcpdump on the first one, so we use:

nsenter --net=/var/run/docker/netns/1-ujgie39kof sh

And now we are in the shell of “1-ujgie39kof”. Now you can run tcpdump here and monitor the packets which are passing between containers (running tcpdump on the host directly doesn’t show them).

Just remember, if you want to see all of the packets you need to run tcpdump -i any, to monitor every network interface. (there are so many of them, just run ifconfig too see them). Also, you need to do this for both ingress and overlay network to see communications inside Docker Swarm.

Everything else about tcpdump is normal here, you can also use -w option to save the dump to a pcap file and later retrieve it using docker cp command.

I hope this helps you, like it helped me. I also need to mention that I used this post on Docker forums to come up with this post. But I wrote this post to both have it documented somewhere and also I’m doing more interesting stuff using this which I’ll post about them soon.