I was running some performance benchmarking using 100.000 requests sent with the h2load tool. They all succeeded and performance was great. But after that Payara did not accept TCP connections anymore. It seems there is some DDoS protection active. Is this a part of Payara 5 or should I rather search in my OS?
Even after googling for several hours, I could not finde anything on the web regarding this issue. It would be great if Team Payara could shade some light on this. Thanks.
Hello,
I confirm Payara do not include an L7 DDoS filter.
I developed a specific filter on my own that drops the http threads generated continuously during the attack… Payara was in good standing after.
roberto(at)javest.com
Thank you for your kind reply. Do you have an idea why the reported behavior is happening?
As I said the http threads are created continuously during all attack period, so the server go in protection but then it is unusable. Thread pool and queue are full.
If you need a ready-solution for your company let me know.
Sorry to say that, but your answers are confusing me. You said you developed a DDoS filter because Payara has none. But what I am asking for in my original question is the exact opposite. So maybe I have misunderstood your answer, and you developed a solution to my actual problem (but not a DDoS filter)?
Sorry, I probably answered not very well to your question… You have practically simulated a DDoS attack with your tests but the status of Payara is that the thread pool is full and queue too, so you need a better tuning or high availability to avoid this situation. Anyway a too repentine set of requests do not give the time to the server to deal with its pending requests and this is a DDoS. The fact that Payara and Glassfish are blocked after this situation is probably because after the queue is full you need to wait the queue is consumed before accepting new requests.
You should receive a RejectedExecutionException.
The TCP is the layer where DDoS happens. It could be you have also used all dynamic port range… are you using Windows ? Under this case you have to wait some timeout or restart your server.
And yes I developed a protection against DDoS but at layer 7 just to protect the app server against thread starvation. It worked good also with Slowloris DDoS attacks.
Thank you for this explanation, but I think you slightly missed the case. In fact I cannot see how it explains the actual behavior of Payara: It serves 100.000 request perfectly well! The connection failed only happens after the very last of these requests! If there would be a tuning problem, then Payara should have missed to correctly process at least some of these 100.000 requests, shouldn
t it? There are zero exceptions found in server.log! I am using Payara’s default settings of domain1, so no, there is no port range, but only 8080. Yes, this is Windows 10 Pro.
Sorry but I didn’t noted your last reply… 100.000 continuous requests are served differently depending of your tuning (you asserted the default one).
I do not know the h2load tool, but 250 requests are served by a single connection (keep-alive).
100000 / 250 = 400 connection ??
But default is 250 connections…
The Payara is using non-blocking IO so the http-threads are not blocking connections and you can have rapidly new requests processed on the way… until the limit…
My suggestion is you try 1 by 1 to change the tuning to see what’s happen.
Change the max number of http thread pool (e.g. 500) and retry.
You can also increase connections from 250 to 500 in the protocol setting for http-listener-1.