Re: [CentOS] Measure network bandwidth per process

-- Charles Polisher #!/bin/bash # Report on network usage by user, using iptables # C. Polisher 2013-06-03 Based on a post by H. LaDerrick on NANOG-L # Which network interface to use IFACE=eth0 # Sampling interval for recording traffic (integer seconds) INTERVAL=10 # Number of intervals to collect stats for. COUNT=10 # For verbose execution set to value to 1, 2 is very verbose DEBUG=0 # --- no changes should be required below this line if [ $UID -ne 0 ] ; then echo "Must run as root" exit 1 fi if [ $COUNT -eq 0 ] ; then echo "Count must be a positive integer from 1 to $((2**32-1))" exit 1 fi # FIXME Magic constant 1000 should be based on some fact from the OS; # OS users could possibly of interest too (all users). userlist=`awk -F':' '{if ($3 >= 1000)print $1;}' /etc/passwd | sort | tr '\n' ' '` OFILE=`mktemp -t "nettrack.XXXXXX"` || exit 1 approvedelete=0 # # Delete pre-existing rules from pre-existing chains # ruleindex=0 for i in $userlist; do for j in tcp udp icmp ; do ruleindex=$(( ruleindex + 1 )) RULENAME="${i}_${j}" ## preflight=`iptables --list-rules --line-numbers | fgrep "$RULENAME"` preflight=`iptables -L --line-numbers | fgrep "$RULENAME"` [ $DEBUG -gt 0 ] && echo "Username: $i Proto: $j Rulename: $RULENAME Ruleindex:$ruleindex" [ $DEBUG -gt 1 ] && echo '-------->>>>>>>>>>>----------' [ $DEBUG -gt 1 ] && echo $preflight [ $DEBUG -gt 1 ] && echo '--------<<<<<<<<<<<----------' if [[ ( "X$preflight" != X ) && ( $approvedelete -eq 0 ) ]] ; then read -p "Found pre-existing ruleset(s). Delete (Y/N)." m if [ "X$m" != XY ] ; then echo "User abort." exit 1 else approvedelete=1 fi fi if [[ ( "X$preflight" != X ) && ( $approvedelete -eq 1 ) ]] ; then [ $DEBUG -gt 0 ] && echo "iptables --delete OUTPUT 1 ($RULENAME) ($ruleindex)" iptables --delete OUTPUT 1 fi done done # # Delete pre-existing chains # for i in $userlist; do for j in tcp udp ; do RULENAME="${i}_${j}" preflight=`iptables -S | fgrep "$RULENAME"` if [[ ( "X$preflight" != X ) || ( $approvedelete -eq 1 ) ]] ; then [ $DEBUG -gt 0 ] && echo "iptables --delete-chain $RULENAME" iptables --delete-chain "$RULENAME" fi done done # # Instantiate a rule for each user/protocol combination # chaincreated=0 for j in tcp udp ; do for i in $userlist; do RULENAME="${i}_${j}" [ $DEBUG -gt 0 ] && echo "iptables -N $RULENAME" iptables -N "$RULENAME" [ $DEBUG -gt 0 ] && echo iptables -I OUTPUT -m owner -o ${IFACE} -p $j --uid-owner ${i} -j "$RULENAME" iptables -I OUTPUT -m owner -o ${IFACE} -p $j --uid-owner ${i} -j "$RULENAME" done done # TODO: prompt for display on stdout; prompt for save results in $OFILE echo "You may wish to tail the report file $OFILE" echo "Date Time Packets Bytes User Proto Iface" > $OFILE for i in `seq 1 $COUNT` ; do echo "Sleeping for $INTERVAL seconds on cycle $i of $COUNT" > /dev/stderr sleep $INTERVAL # Parameter --zero zeroes the rules counters after outputting them iptables -n -v --exact --zero -L 2>&1 \ | egrep -v '^$|^Chain|^ pkts|^Zeroing' \ | egrep '[_]tcp|[_]udp' \ | awk ' {if ( $1 > 0 ) printf("%s %d %d %s %s %s\n",strftime("%F %H:%M:%S%z"),$1,$2,$3,$4,$7);} ' \ | sort -nr >> $OFILE done exit 0 # Re: Security Guideance # # From: LaDerrick H. # Date: Tue Feb 23 15:49:32 2010 # List-archive: http://mailman.nanog.org/mailman/nanog # List-id: North American Network Operators Group <nanog.nanog.org> # # Paul Stewart wrote: # > We have a strange series of events going on in the past while.... Brief # > history here, looking for input from the community - especially some of # > the security folks on here. # > # > We provide web hosting services - one of our hosting boxes was found a # > while back with root kits installed, un patched software and lots of # > other "goodies". With some staff changes in place (don't think I need # > to elaborate on that) we are trying to clean up several issues including # > this particular server. A new server was provisioned, patched, and # > deployed. User data was moved over and now the same issue is coming # > back.... # > The problem is that a user on this box appears to be launching high # > traffic DOS attacks from it towards other sites. These are UDP based # > floods that move around from time to time - most of these attacks only # > last a few minutes. # # Counting outbound udp bytes and packets can help spot anomalies. # Something like this would help but may be unwieldy if you have thousands # of users on a single box: # # WANIF=eth0 # userlist="userA userB user..." # for i in ${userlist} # do # iptables -N ${i}_UDP # iptables -I OUTPUT -m owner -o ${WANIF} -p udp --uid-owner ${i} -j ${i}_UDP # done # # Then look at counters with: # iptables -nvL OUTPUT | grep _UDP | sort....... # # # I wouldn't leave this in place full-time for thousands of accounts # though without attempting to measure the impact on network performance. # # > I've done tcpdumps within seconds of the attack starting and to date # > been unable to find the source of this attack (we know the server, # > just not sure which customer it is on the server that's been # > compromised). Several hours of scanning for php, cgi, pl type files # > have been wasted and come up nowhere... # > # > It's been suggested to dump IDS in front of this box and I know I'll # > get some feedback positive and negative in that aspect. # > # > What tools/practices do others use to resolve this issue? It's a # > Centos 5.4 box running latest Plesk control panel. # > # > Typically we have found it easy to track down the offending script or # > program - this time hasn't been easy at all...