[CentOS] CPU Scheduling allocates more time for few threads

Hi      I have a Virtual Machine with CentOS 7 64-bit having 6 CPUs (intel). I have a process with 10 threads. two threads of the process pinned to two CPU's using affinity. These threads processes lot of network messages via Sockets. Other threads wait for other events using pthread condition wait. When two threads CPU reaches 75 to 80% then the schedular, schedules most of the time for those two threads, even though remaining 4 CPU's are ideal. Due to this other threads event processing is delayed a lot.  Is it mostly do to more signal's from kernel space to user space tor those threads ? Is there a way to avoid this issue?  RegardsRadha