I was benchmarking nginx on a Vultr server and the test ran great, over 17K requests per second, but after a little over 20K requests, nginx would just halt. Memory and CPU usage were nowhere near the limit, and eventually Apache Bench got a timeout.
This error occurred over and over again in /var/log/kern.log as soon as the slowdown hit:
nf_conntrack: table full, dropping packet
Thanks to this security StackExchange post I ran
sysctl net.netfilter.nf_conntrack_max and found that nf_conntrack_max was set to 23788. I checked on my Linode server and it was 65536. To get the Vultr server to use a higher limit, I used this command:
sysctl -w net.netfilter.nf_conntrack_max=65536 and now the benchmarks run as smooth as butter.
I hope this saves you some time if you happen to run into the same situation.
[…] around 700, so something was clearly wrong, but I’m not sure what. I had already fixed the nf_conntrack_max issue, but it could be something else along the same […]