[CentOS] Measure network bandwidth per process

On 2019-04-18 23:05, Kaushal Shriyan wrote:
> On Tue, Apr 16, 2019 at 10:30 PM Kaushal Shriyan <kaushalshriyan at gmail.com>
> wrote:
> 
> > Hi,
> >
> > Is there a way to measure network bandwidth per process in CentOS Linux
> > release 7.6.1810 (Core) using any utility? I was reading about nethogs but
> > it does not have the option to run it in daemon mode so that we can take a
> > look at historical data to figure out the process which was consuming high
> > network bandwidth instead it is a good tool for Live monitoring.
> >
> > Please suggest. Thanks in Advance.
> >
> > Best Regards,
> >
> > Kaushal

Hi Kaushal,

You might take this as a starting point. Please read carefully and
test on a non-critical system before implementing, in other words,
USE AT YOUR OWN RISK. TL;DR: use iptables UID/GID matching to count
packets.

Best regards,
-- 
Charles Polisher

#!/bin/bash
# Report on network usage by user, using iptables
# C. Polisher 2013-06-03 Based on a post by H. LaDerrick on NANOG-L

# Which network interface to use
IFACE=eth0

# Sampling interval for recording traffic (integer seconds)
INTERVAL=10

# Number of intervals to collect stats for. 
COUNT=10

# For verbose execution set to value to 1, 2 is very verbose
DEBUG=0

# --- no changes should be required below this line

if [ $UID -ne 0 ] ; then
    echo "Must run as root"
    exit 1
fi

if [ $COUNT -eq 0 ] ; then
    echo "Count must be a positive integer from 1 to $((2**32-1))"
    exit 1
fi

# FIXME Magic constant 1000 should be based on some fact from the OS;
#       OS users could possibly of interest too (all users).
userlist=`awk -F':' '{if ($3 >= 1000)print $1;}' /etc/passwd | sort | tr '\n' ' '`


OFILE=`mktemp -t "nettrack.XXXXXX"` || exit 1

approvedelete=0

#
# Delete pre-existing rules from pre-existing chains
#
ruleindex=0
for i in $userlist;
do
    for j in tcp udp icmp ; 
    do
        ruleindex=$(( ruleindex + 1 ))
        RULENAME="${i}_${j}"
        ## preflight=`iptables --list-rules --line-numbers | fgrep "$RULENAME"`
        preflight=`iptables -L --line-numbers | fgrep "$RULENAME"`
        [ $DEBUG -gt 0 ] && echo "Username: $i  Proto: $j Rulename: $RULENAME   Ruleindex:$ruleindex"
        [ $DEBUG -gt 1 ] && echo '-------->>>>>>>>>>>----------'
        [ $DEBUG -gt 1 ] && echo $preflight
        [ $DEBUG -gt 1 ] && echo '--------<<<<<<<<<<<----------'

        if [[ ( "X$preflight" != X ) && ( $approvedelete -eq 0 ) ]] ; then
            read -p "Found pre-existing ruleset(s). Delete (Y/N)." m
            if [ "X$m" != XY ] ; then
                echo "User abort."
                exit 1
            else
                approvedelete=1
            fi
        fi
        if [[ ( "X$preflight" != X ) && ( $approvedelete -eq 1 ) ]] ; then
            [ $DEBUG -gt 0 ] && echo "iptables --delete OUTPUT 1      ($RULENAME) ($ruleindex)"
            iptables --delete OUTPUT 1
        fi
    done
done

#
# Delete pre-existing chains
#
for i in $userlist;
do
    for j in tcp udp ; 
    do
        RULENAME="${i}_${j}"
        preflight=`iptables -S | fgrep "$RULENAME"`
        if [[ ( "X$preflight" != X ) || ( $approvedelete -eq 1 ) ]] ; then
            [ $DEBUG -gt 0 ] && echo "iptables --delete-chain $RULENAME"
            iptables --delete-chain "$RULENAME"
        fi
    done
done

#
# Instantiate a rule for each user/protocol combination
#
chaincreated=0
for j in tcp udp ; 
do
    for i in $userlist;
    do
        RULENAME="${i}_${j}"
        [ $DEBUG -gt 0 ] && echo "iptables -N $RULENAME"
        iptables -N "$RULENAME"
        [ $DEBUG -gt 0 ] && echo iptables -I OUTPUT -m owner -o ${IFACE} -p $j --uid-owner ${i} -j "$RULENAME"
        iptables -I OUTPUT -m owner -o ${IFACE} -p $j --uid-owner ${i} -j "$RULENAME"
    done
done


# TODO: prompt for display on stdout; prompt for save results in $OFILE
echo "You may wish to tail the report file $OFILE"

    echo "Date Time Packets Bytes User Proto Iface" > $OFILE
    for i in `seq 1 $COUNT` ;
    do
        echo "Sleeping for $INTERVAL seconds on cycle $i of $COUNT" > /dev/stderr
        sleep $INTERVAL
        # Parameter --zero zeroes the rules counters after outputting them
        iptables -n -v --exact --zero -L 2>&1 \
        | egrep -v '^$|^Chain|^ pkts|^Zeroing' \
        | egrep '[_]tcp|[_]udp' \
        | awk '
                {if ( $1 > 0 ) printf("%s %d %d %s %s %s\n",strftime("%F %H:%M:%S%z"),$1,$2,$3,$4,$7);}
              ' \
        | sort -nr >> $OFILE
    done

exit 0


#  Re: Security Guideance
#  
#  From: LaDerrick H.
#  Date: Tue Feb 23 15:49:32 2010
#  List-archive: http://mailman.nanog.org/mailman/nanog
#  List-id: North American Network Operators Group <nanog.nanog.org>
#  
#  Paul Stewart wrote:
# > We have a strange series of events going on in the past while.... Brief
# > history here, looking for input from the community - especially some of
# > the security folks on here.
#  >
# > We provide web hosting services - one of our hosting boxes was found a
# > while back with root kits installed, un patched software and lots of
# > other "goodies".    With some staff changes in place (don't think I need
# > to elaborate on that) we are trying to clean up several issues including
# > this particular server.  A new server was provisioned, patched, and
# > deployed.  User data was moved over and now the same issue is coming
# > back....
# > The problem is that a user on this box appears to be launching high
# > traffic DOS attacks from it towards other sites.  These are UDP based
# > floods that move around from time to time - most of these attacks only
# > last a few minutes.
#  
#  Counting outbound udp bytes and packets can help spot anomalies.
#  Something like this would help but may be unwieldy if you have thousands
#  of users on a single box:
# 
#  WANIF=eth0
#  userlist="userA userB user..."
#  for i in ${userlist}
#  do
#     iptables -N ${i}_UDP
#     iptables -I OUTPUT -m owner -o ${WANIF} -p udp --uid-owner ${i} -j ${i}_UDP
#  done
#  
#  Then look at counters with:
#  iptables -nvL OUTPUT | grep _UDP | sort.......
#   
# 
#  I wouldn't leave this in place full-time for thousands of accounts
#  though without attempting to measure the impact on network performance.
#  
# > I've done tcpdumps within seconds of the attack starting and to date
# > been unable to find the source of this attack (we know the server,
# > just not sure which customer it is on the server that's been
# > compromised).  Several hours of scanning for php, cgi, pl type files
# > have been wasted and come up nowhere...
# > 
# > It's been suggested to dump IDS in front of this box and I know I'll
# > get some feedback positive and negative in that aspect.
# > 
# > What tools/practices do others use to resolve this issue?  It's  a
# > Centos 5.4 box running latest Plesk control panel.
# > 
# > Typically we have found it easy to track down the offending script or
# > program - this time hasn't been easy at all...