[CentOS] Performance issues/difference of two servers running same task (one is quicker)

Thu Jul 4 06:43:22 UTC 2019
Jobst Schmalenbach <jobst at barrett.com.au>


I need some advice what to do next, even if someone tells me to
check out (an)other mailing list(s), tuning site or point me in a better
direction how to solve my annoying problem: one server is much faster
for certain tasks although on "shitty" hardware.

I have tried many things to solve my issue
 - changed buffer/pool/cache/etc mysqld
 - changed server settings apache/php
 - changed various OS settings (sysctl) e.g. turned off IPV6
but havent figured it out.

I have a development server (local) and life servers (data center)
Used mainly for many different websites and one online training site.

the development and life server in question run the same software setup:
  - CentOS Linux release 7.6.1810
  - bind 32:9.9.4-74.el7_6.1
  - Apache/2.4.6 (CentOS)
  - PHP 7.1.29
  - mysqld  Ver 5.7.26
  - wordpress, woocommerce, wishlistmember, Sensei etc
  - software are all in the same stages of updates.
  - even many of the linux conf files are the same (/etc/host, bind, etc)
  - the databases are copies/identical

Life server is a Poweredge M710,48GB,2xXeon L5630,LSI Raid1 SSD
Dev  server is a DIY, GIGABYTE MX31-BS0, 32GB, 1xXeon E3-1245,MDADM RAID0 1TB Seagate Spinners

Clearly the development server is hardware wise way below the specs of the Dell but 
software wise they are identical (they get upgraded at the same time).

During normal operations (i.e. display websites, online training courses etc) the DELL 
displays the websites faster although it sits 1000KM up north in a datacenter on
a different network than the local server on the same network as my machine.

Yet the DEV server outshines the DELL when creating a few large custom tables, ie
the local server takes 5s while the DELL takes 15s (small tables), more for bigger tables.

The task is based on:
 - level, member, course, group are all ID's
 - members can belong to a group, a level and can access many courses
 - the ID restricts what they can access and what they belong to.
 - a course for each member can have various stages of completion
 - using an API (wishlist member) that performs LOCAL calls when accessed locally
   I can get who belongs to what and make up my info I need, then use PHP
   to make up the table.
 - DB calls ARE LOCAL!

Now when I try to create a table of members belonging to the same group level
doing the same course with different stages of completion the DELL takes on average
3 times longer to complete the table (normally about 20 to 30 rows).

I have put microtime() calls before and after certain calls, and it's visibly different:
    Jul 04 04:57:26 UTC _members took 0.0005459785461425 ms
    Jul 04 04:57:26 UTC _members took 0.0005321502685546 ms
    Jul 04 05:00:36 UTC _members took 0.0014369487762451 ms
    Jul 04 05:00:36 UTC _members took 0.0013291835784912 ms
If I do this 300+ times, the outcome is very different.

So my questions:

 - How can it be that the DELL takes so much longer alltough on the far better hardware?
 - How can it be allthough everything (software/os/plugins) is the same?
 - This even happens if the DELL is on low load (i.e. middle of the night) and
   only serves a few requests.

Same software, same config, same database, same amount of data in the database
yet on better hardware it's slower?

Any ideas anyone?

Jobst Schmalenbach