Rate limiting website payments with nginx and haproxy

A couple of years ago, our website became the target for some credit card thieves to make batch payment attempts in order to verify which of the stolen card numbers from their stash were active, valid cards. Naturally, cards that are verified to be good are worth more to the buyer.

We put a number of measures in place to deter these attempts. One measure that has been especially effective at deterring the carding attempts was rate limiting, and this didn’t require any changes to our code. When the web server detects a specified number of requests hitting a particular URL within a short period of time, it blocks any further attempts. In addition to the rate limiting, we were able to use this as an input to another script to block any future requests from that IP. However, since our web servers are load balanced behind an haproxy host, we have the added challenge of blocking the IP on the haproxy server rather than at the web server.

Let’s jump into the nginx configuration:

# Rate limit carding attack attempts
limit_req_zone $binary_remote_addr zone=paymentlimit:10m rate=5r/m;

# nginx config
# Rate limit carding attack attempts
location ~ ^/payments$ {
  limit_req zone=paymentlimit;
  limit_req_status 444;
  ...
}

First we’re creating a limit request zone called paymentlimit with a maximum rate of 5 requests per minute, or 1 request every 12 seconds. If the average rate exceeds this limit, the zone will be triggered. The zone is only in effect for our payments page. Users making a payment POST to this page from a different page, so refreshing the payment screen will not trigger the rate limiter. Only multiple rapid POSTs will do this. In this case, a custom 444 HTTP status code will be returned, which is easy to find in the nginx logs.

Next comes the script we use to scan the nginx error logs and block IPs that are multiple offenders of the rate limiter. I decided that exceeding the limit 5 times warrants a banned IP.

#!/bin/bash

BLOCK_LIMIT=5

declare -A attackerIPs

tail -Fn0 /var/log/nginx/error.log | \
while read line ; do
  ip=$(echo "$line" | grep "paymentlimit" -a | awk -v RS='([0-9]+\\.){3}[0-9]+' 'RT{print RT}')
  if [ ! -z "$ip" ]
  then
    ((attackerIPs[$ip]++))
    echo "Potential carding attack detected. Adding IP: $ip (${attackerIPs[$ip]})"

    if [[ "${attackerIPs[$ip]}" -ge "$BLOCK_LIMIT" ]]; then
      echo "Threshold reached. Blocking IP $ip."
      bash ~/haproxy_block_ip $ip >> /var/log/monitor-carding-attackers.log
    fi
  fi
done

If you only have a single web server, then line 17, which executes haproxy_block_ip can be replaced with a command to block the IP directly. In our case, I have a separate script that blocks the IP on the haproxy server.

#!/bin/bash

ssh haproxy_server "sudo iptables -A INPUT -s $1 -j DROP"

now=$(date +'%Y/%m/%d %H:%M:%S')
echo "$now Blocked IP Address $1 on Load Balancer"

And the final part of the equation is the init.d script that runs as a background daemon, monitoring the nginx log file at all times.

#! /bin/sh
# chkconfig: 345 99 10
case "$1" in
  start)
    # Executes our script
    bash /home/user/monitor-carding-attackers &
    ;;
  *)
    ;;
esac
exit 0

And that’s it! This handful of small scripts will make life a little more difficult for any credit card criminals who want to take advantage of your payment gateway.