hi guys
We've got a service being hammered, and Fabian's had a look - we have no clear solution on howto best mitigate and manage this. We need help.
buildlogs.centos.org hosts, amongst other things, isos for our monthly builds - which are being downloaded repeatedly. Fabian, can you add specifics on what the problem space is ?
The main problem we have is that its causing outage for other services hosted on the same service, including cbs testing repos
regards
On 07/08/17 12:08, Karanbir Singh wrote:
hi guys
We've got a service being hammered, and Fabian's had a look - we have no clear solution on howto best mitigate and manage this. We need help.
buildlogs.centos.org hosts, amongst other things, isos for our monthly builds - which are being downloaded repeatedly. Fabian, can you add specifics on what the problem space is ?
The main problem we have is that its causing outage for other services hosted on the same service, including cbs testing repos
regards
I had recommended and Fabian looked at mod-evasive, but has reservations around that. how do people these days typically handle flood situations ?
On 07/08/17 12:13, Karanbir Singh wrote:
On 07/08/17 12:08, Karanbir Singh wrote:
hi guys
We've got a service being hammered, and Fabian's had a look - we have no clear solution on howto best mitigate and manage this. We need help.
buildlogs.centos.org hosts, amongst other things, isos for our monthly builds - which are being downloaded repeatedly. Fabian, can you add specifics on what the problem space is ?
The main problem we have is that its causing outage for other services hosted on the same service, including cbs testing repos
regards
I had recommended and Fabian looked at mod-evasive, but has reservations around that. how do people these days typically handle flood situations ?
Fabian's taken down the specific iso that was being hit, while we workout a proper solution. if anyone has download issues, reply here
On Mon, Aug 7, 2017 at 5:57 AM, Karanbir Singh mail-lists@karan.org wrote:
I had recommended and Fabian looked at mod-evasive, but has reservations around that. how do people these days typically handle flood situations
?
What are the concerns with mod_evasive? I'm not sure if it makes sense to add Varnish to the mix, but I've been testing the Varnish vsthrottle module for DoS mitigation, and it seems to work well. The nice part with doing this in Varnish is it is very customizable within the VCL -- here's an old post with a small code snippet, but this could be customized to whitelist based on any header, source IP, etc. which seems to be a lot more flexible than mod_evasive -- and you may get some caching benefits from Varnish as well, though not for the larger downloads. https://old.varnish-cache.org/vmod/vsthrottle-rate-limitingthrottling-v4-and...
-Jeff
On 07/08/17 15:22, Jeff Sheltren wrote:
On Mon, Aug 7, 2017 at 5:57 AM, Karanbir Singh <mail-lists@karan.org mailto:mail-lists@karan.org> wrote:
> I had recommended and Fabian looked at mod-evasive, but has reservations > around that. how do people these days typically handle flood situations ? >What are the concerns with mod_evasive? I'm not sure if it makes sense to add Varnish to the mix, but I've been testing the Varnish vsthrottle module for DoS mitigation, and it seems to work well. The nice part with doing this in Varnish is it is very customizable within the VCL -- here's an old post with a small code snippet, but this could be customized to whitelist based on any header, source IP, etc. which seems to be a lot more flexible than mod_evasive -- and you may get some caching benefits from Varnish as well, though not for the larger downloads. https://old.varnish-cache.org/vmod/vsthrottle-rate-limitingthrottling-v4-and...
one of our challenges is that the infra itself is fairly well distributed around the world. so we dont have single egress points.
I believe mod_qos ( based on Patrick Liambocks recommendation ) was finally part of the solution, I will let Fabian comment in depth around the work he did and why
On 07/08/17 13:13, Karanbir Singh wrote:
I had recommended and Fabian looked at mod-evasive, but has reservations around that. how do people these days typically handle flood situations ?
That depends on the size and type of the attack - I think one can only fend off small attacks. People facing DDoS probably host on a service like OVH, which employ expensive hardware (e.g. from Arbor Networks) to handle layer 3 attacks, or, for layer 7 attacks, hide behind an HTTP reverse proxy like CloudFlare. There are also companies specialized in preventing large-scale attacks (like Akamai or Level3), but they tend to be quite expensive. AWS or Google Cloud won't go down either, but the bandwidth bill would probably drive a small company into bankruptcy really fast.
If it's the same set of IPs, perhaps you could ask the upstream provider to filter them. Is the attack still going on?
Regards, Laurențiu
On 07/08/17 22:55, Laurentiu Pancescu wrote:
On 07/08/17 13:13, Karanbir Singh wrote:
I had recommended and Fabian looked at mod-evasive, but has reservations around that. how do people these days typically handle flood situations ?
That depends on the size and type of the attack - I think one can only fend off small attacks. People facing DDoS probably host on a service like OVH, which employ expensive hardware (e.g. from Arbor Networks) to handle layer 3 attacks, or, for layer 7 attacks, hide behind an HTTP reverse proxy like CloudFlare. There are also companies specialized in preventing large-scale attacks (like Akamai or Level3), but they tend to be quite expensive. AWS or Google Cloud won't go down either, but the bandwidth bill would probably drive a small company into bankruptcy really fast.
If it's the same set of IPs, perhaps you could ask the upstream provider to filter them. Is the attack still going on?
all of our infra for edge stuff is hosted in similar facilities. So we have some umbrella, but even then the traffic making it through was huge.
Fabian's done some great work today, and thanks to folks making recommendations we were able to close this off without any major impact.
Not true about AWS or GCP. You don't get charged on ingress and it's in their best interest to mitigate this at the perimeter. L3 and Prolexic(Akamai) have all your traffic go through their scrubbing centers - really expensive. mod_evasive won't work with any half decent reflection attack.
On Mon, Aug 7, 2017 at 5:55 PM, Laurentiu Pancescu < lpancescu@centosproject.org> wrote:
On 07/08/17 13:13, Karanbir Singh wrote:
I had recommended and Fabian looked at mod-evasive, but has reservations around that. how do people these days typically handle flood situations ?
That depends on the size and type of the attack - I think one can only fend off small attacks. People facing DDoS probably host on a service like OVH, which employ expensive hardware (e.g. from Arbor Networks) to handle layer 3 attacks, or, for layer 7 attacks, hide behind an HTTP reverse proxy like CloudFlare. There are also companies specialized in preventing large-scale attacks (like Akamai or Level3), but they tend to be quite expensive. AWS or Google Cloud won't go down either, but the bandwidth bill would probably drive a small company into bankruptcy really fast.
If it's the same set of IPs, perhaps you could ask the upstream provider to filter them. Is the attack still going on?
Regards, Laurențiu
CentOS-devel mailing list CentOS-devel@centos.org https://lists.centos.org/mailman/listinfo/centos-devel
On 08/08/17 01:57, Akshay Kumar wrote:
Not true about AWS or GCP. You don't get charged on ingress and it's in their best interest to mitigate this at the perimeter.
Indeed, they don't charge for ingress, but your server has to answer to HTTP requests. Even small responses can add up quickly, moreso if you are serving ISOs. Another problem is with autoscaling setups - if you automatically spawn several hundreds of EC2 instances to handle the increased number of HTTP requests, you'll end up with a pretty big bill. I've heard of several cases of Amazon choosing to "forgive" the bill resulting from an attack and you can set usage limits, so it's probably not that bad.
L3 and Prolexic(Akamai) have all your traffic go through their scrubbing centers - really expensive. mod_evasive won't work with any half decent reflection attack.
Yes, I think scrubbing centers are technically the best solution (reverse proxy companies are in the position to perform MitM on SSL traffic and can only handle HTTP, but they are the most affordable solution).
Anyway, glad that it's solved now! :)
-
Akshay Kumar -
Jeff Sheltren -
Karanbir Singh -
Laurentiu Pancescu