[CentOS] Performance issues/difference of two servers running same task (one is quicker)

[Simon Matter]

> Hi
>
> I need some advice what to do next, even if someone tells me to
> check out (an)other mailing list(s), tuning site or point me in a better
> direction how to solve my annoying problem: one server is much faster
> for certain tasks although on "shitty" hardware.
>
> I have tried many things to solve my issue
>  - changed buffer/pool/cache/etc mysqld
>  - changed server settings apache/php
>  - changed various OS settings (sysctl) e.g. turned off IPV6
> but havent figured it out.
>
> I have a development server (local) and life servers (data center)
> Used mainly for many different websites and one online training site.
>
> the development and life server in question run the same software setup:
>   - CentOS Linux release 7.6.1810
>   - bind 32:9.9.4-74.el7_6.1
>   - Apache/2.4.6 (CentOS)
>   - PHP 7.1.29
>   - mysqld  Ver 5.7.26
>   - wordpress, woocommerce, wishlistmember, Sensei etc
>   - software are all in the same stages of updates.
>   - even many of the linux conf files are the same (/etc/host, bind, etc)
>   - the databases are copies/identical
>
> Life server is a Poweredge M710,48GB,2xXeon L5630,LSI Raid1 SSD
> Dev  server is a DIY, GIGABYTE MX31-BS0, 32GB, 1xXeon E3-1245,MDADM RAID0
> 1TB Seagate Spinners
>
> Clearly the development server is hardware wise way below the specs of the
> Dell but
> software wise they are identical (they get upgraded at the same time).
>
> During normal operations (i.e. display websites, online training courses
> etc) the DELL
> displays the websites faster although it sits 1000KM up north in a
> datacenter on
> a different network than the local server on the same network as my
> machine.
>
> Yet the DEV server outshines the DELL when creating a few large custom
> tables, ie
> the local server takes 5s while the DELL takes 15s (small tables), more
> for bigger tables.
>
> The task is based on:
>  - level, member, course, group are all ID's
>  - members can belong to a group, a level and can access many courses
>  - the ID restricts what they can access and what they belong to.
>  - a course for each member can have various stages of completion
>  - using an API (wishlist member) that performs LOCAL calls when accessed
> locally
>    I can get who belongs to what and make up my info I need, then use PHP
>    to make up the table.
>  - DB calls ARE LOCAL!
>
> Now when I try to create a table of members belonging to the same group
> level
> doing the same course with different stages of completion the DELL takes
> on average
> 3 times longer to complete the table (normally about 20 to 30 rows).
>
> I have put microtime() calls before and after certain calls, and it's
> visibly different:
>   DEV
>     Jul 04 04:57:26 UTC _members took 0.0005459785461425 ms
>     Jul 04 04:57:26 UTC _members took 0.0005321502685546 ms
>   LIFE
>     Jul 04 05:00:36 UTC _members took 0.0014369487762451 ms
>     Jul 04 05:00:36 UTC _members took 0.0013291835784912 ms
> If I do this 300+ times, the outcome is very different.
>
>
> So my questions:
>
>  - How can it be that the DELL takes so much longer alltough on the far
> better hardware?
>  - How can it be allthough everything (software/os/plugins) is the same?
>  - This even happens if the DELL is on low load (i.e. middle of the night)
> and
>    only serves a few requests.
>
> Same software, same config, same database, same amount of data in the
> database
> yet on better hardware it's slower?

Two ideas:

a) the DELL maybe faster over all but if I'm right single core speed is
slower than on DEV machine.

b) how do the LSI/SSD perform compared to the MDADM/RAID0 on the DEV
server? I'm not sure the DELL is a clear winner here.

Regards,
Simon