Greetings, absurd subject, but I don't know how else I could describe what is happening. I have a small static website hosted on a Centos 7 VPS (that I'll update as soon as possible, but what follows is much more urgent, I must absolutely fix it asap). The website is generated by Hugo. I run hugo on my computer, then rsync everything on the server, in the /var/www/html/hugo/MYWEBSITE folder. Everything worked great for years. This week I rebuilt the whole website almost from scratch, using a different hugo theme and updating many pages. Then I ran rsync, checked the new website in my browser, jumping from page to page, and everything worked as expected: new theme, new content, new menus, everything. 2/3 minutes later, the OLD website reappeared, all of it, as if I had not ran rsync at all. For example, in the new website there is a new subfolder called "bio", that did not exist in the old version.
If I ssh to the server, cd to /var/www/html/hugo/MYWEBSITE and run "ls -l" RIGHT AFTER running rsync, I see the "bio" subfolder. If I do "ls -l" 3 minutes later, the "bio" subfolder is no more.
I have repeated the rsync process 5/6 times before writing this email, and it's always the same. Every time I rsync, what I uploaded lasts 2/3 minutes, then the WHOLE folder on the server is erased and refilled with the previous full version of the website.
It's as if there were some hidden cron job somewhere that runs hugo on the OLD source files, but I can't find it. All the standard methods one can find by googling "how to list all cron jobs" don't show anything that may be the reason. THANKS in advance for any help, Marco
On 9/7/2023 11:19 AM, Marco Fioretti wrote:
It's as if there were some hidden cron job somewhere that runs hugo on the OLD source files, but I can't find it. All the standard methods one can find by googling "how to list all cron jobs" don't show anything that may be the reason.
Also check for systemd timer units in all the systemd unit directories.
Check the system log files when this happens ("ls -lt /var/log | head") to see if something's running at that moment.
Run top as it happens to see what jumps to the top of the process list.
Use chattr to mark the files immutable and see what complains when it fails to change the content.
From what your describing, a few things to check, some may be overly obvious so I apologize if you have checked these:
1. Did you clear your browser cache? 2. Is there a proxy server that has an old version of the site cached on it, and it's serving up the old site content instead of a refresh? 3. Are there multiple web servers hosting the content in a cluster and you only rsync'd to one of them, and requests for connections is doing a round robin approach? 4. Is there a backup mechanism or security appliance that is checking for the presence of pages on the site, doing a checksum or a diff or something, and if things have changed, then it's putting everything back to it's secured copy, perhaps an antimalware security tool or auto backup tool or some other tool reaching into the server and doing this check.
Check the host names, IP addresses, mac addresses when you see the inconsistent data, and check for your bio folder w/ it present and w/o it present see if anything changes.
Set up a new CentOS VPS w/ a LAMP stack with the same base packages installed as the existing VM (don't clone the existing one), then put the new site there and see if it's happening there. If it isn't then some kind of security mechanism is checking the data and restoring a backup.
Just my 2 cents.
Chris
On 9/7/2023 1:19 PM, Marco Fioretti wrote:
Greetings, absurd subject, but I don't know how else I could describe what is happening. I have a small static website hosted on a Centos 7 VPS (that I'll update as soon as possible, but what follows is much more urgent, I must absolutely fix it asap). The website is generated by Hugo. I run hugo on my computer, then rsync everything on the server, in the /var/www/html/hugo/MYWEBSITE folder. Everything worked great for years. This week I rebuilt the whole website almost from scratch, using a different hugo theme and updating many pages. Then I ran rsync, checked the new website in my browser, jumping from page to page, and everything worked as expected: new theme, new content, new menus, everything. 2/3 minutes later, the OLD website reappeared, all of it, as if I had not ran rsync at all. For example, in the new website there is a new subfolder called "bio", that did not exist in the old version.
If I ssh to the server, cd to /var/www/html/hugo/MYWEBSITE and run "ls -l" RIGHT AFTER running rsync, I see the "bio" subfolder. If I do "ls -l" 3 minutes later, the "bio" subfolder is no more.
I have repeated the rsync process 5/6 times before writing this email, and it's always the same. Every time I rsync, what I uploaded lasts 2/3 minutes, then the WHOLE folder on the server is erased and refilled with the previous full version of the website.
It's as if there were some hidden cron job somewhere that runs hugo on the OLD source files, but I can't find it. All the standard methods one can find by googling "how to list all cron jobs" don't show anything that may be the reason. THANKS in advance for any help, Marco _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
At Thu, 07 Sep 2023 18:19:33 +0000 CentOS mailing list centos@centos.org wrote:
Greetings, absurd subject, but I don't know how else I could describe what is happening. I have a small static website hosted on a Centos 7 VPS (that I'll update as soon as possible, but what follows is much more urgent, I must absolutely fix it asap). The website is generated by Hugo. I run hugo on my computer, then rsync everything on the server, in the /var/www/html/hugo/MYWEBSITE folder. Everything worked great for years. This week I rebuilt the whole website almost from scratch, using a different hugo theme and updating many pages. Then I ran rsync, checked the new website in my browser, jumping from page to page, and everything worked as expected: new theme, new content, new menus, everything. 2/3 minutes later, the OLD website reappeared, all of it, as if I had not ran rsync at all. For example, in the new website there is a new subfolder called "bio", that did not exist in the old version.
If I ssh to the server, cd to /var/www/html/hugo/MYWEBSITE and run "ls -l" RIGHT AFTER running rsync, I see the "bio" subfolder. If I do "ls -l" 3 minutes later, the "bio" subfolder is no more.
I have repeated the rsync process 5/6 times before writing this email, and it's always the same. Every time I rsync, what I uploaded lasts 2/3 minutes, then the WHOLE folder on the server is erased and refilled with the previous full version of the website.
Are you running cPanel?
It's as if there were some hidden cron job somewhere that runs hugo on the OLD source files, but I can't find it. All the standard methods one can find by googling "how to list all cron jobs" don't show anything that may be the reason. THANKS in advance for any help, Marco _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Hello all, and my apologies for the late answer, but as I "fixed" this I had to patch other problems until this week...
TL;DR : I did fix the problem by "brute force", but don't really know what happened. See bottom of email for more details.
On Thu, Sep 07, 2023 18:19:33 PM +0000, Marco Fioretti wrote:
Greetings, absurd subject, but I don't know how else I could describe what is happening. I have a small static website hosted on a Centos 7 VPS (that I'll update as soon as possible, but what follows is much more urgent, I must absolutely fix it asap). The website is generated by Hugo. I run hugo on my computer, then rsync everything on the server, in the /var/www/html/hugo/MYWEBSITE folder. Everything worked great for years. This week I rebuilt the whole website almost from scratch, using a different hugo theme and updating many pages. Then I ran rsync, checked the new website in my browser, jumping from page to page, and everything worked as expected: new theme, new content, new menus, everything. 2/3 minutes later, the OLD website reappeared, all of it, as if I had not ran rsync at all. For example, in the new website there is a new subfolder called "bio", that did not exist in the old version.
If I ssh to the server, cd to /var/www/html/hugo/MYWEBSITE and run "ls -l" RIGHT AFTER running rsync, I see the "bio" subfolder. If I do "ls -l" 3 minutes later, the "bio" subfolder is no more.
I have repeated the rsync process 5/6 times before writing this email, and it's always the same. Every time I rsync, what I uploaded lasts 2/3 minutes, then the WHOLE folder on the server is erased and refilled with the previous full version of the website.
It's as if there were some hidden cron job somewhere that runs hugo on the OLD source files, but I can't find it. All the standard methods one can find by googling "how to list all cron jobs" don't show anything that may be the reason. THANKS in advance for any help, Marco _______________________________________________
In synthesis:
- whatever the problem was, it was only on the server, no browser stuff. I had said since the beginning that I could see the HTML folders reappear by doing "ls -l" on the server.
- the server has no cpanel, no proxies, no security appliance etc, nothing of the sort. It is a VPS with the bare minimum, administered exclusively via SSH and cron jobs that never dido anything to **generate** web pages
- I did look inside /var/log, top, systemd timer units in the systemd unit directories, but could not recognize anything connected to this problem
How I "solved" the problem:
I do have a shell script on the server that I used to run hugo with the right parameters to regenerate the website, but I NEVER included it in any cron job. I only executed it manually when needed, because the site was seldom updated. Well, when I changed name to that script and made it NOT executable, the problem disappeared.
So, I probably I did miss something in the checks above, but no idea what...
Marco
Try adding some or all of these commands to your script, rename it back to its previous name and make it exectable to see who calls it.
Of course let yourlogfile.txt be an absolute path to your desired destination. logfile=/path/to/desired/logfile.txt
date >> ${logfile} echo $PPID >> ${logfile}
#if you need the command from this parent pid: cat /proc/$PPID/comm >> ${logfile}
#if you need the full command line (with all options): cat /proc/$PPID/cmdline >> ${logfile} unset logfile
§:-) Christer
On Tuesday 26 September 2023 20:45:48 Marco Fioretti wrote:
Hello all, and my apologies for the late answer, but as I "fixed" this I had to patch other problems until this week...
TL;DR : I did fix the problem by "brute force", but don't really know what happened. See bottom of email for more details.
On Thu, Sep 07, 2023 18:19:33 PM +0000, Marco Fioretti wrote:
Greetings, absurd subject, but I don't know how else I could describe what is happening. I have a small static website hosted on a Centos 7 VPS (that I'll update as soon as possible, but what follows is much more urgent, I must absolutely fix it asap). The website is generated by Hugo. I run hugo on my computer, then rsync everything on the server, in the /var/www/html/hugo/MYWEBSITE folder. Everything worked great for years. This week I rebuilt the whole website almost from scratch, using a different hugo theme and updating many pages. Then I ran rsync, checked the new website in my browser, jumping from page to page, and everything worked as expected: new theme, new content, new menus, everything. 2/3 minutes later, the OLD website reappeared, all of it, as if I had not ran rsync at all. For example, in the new website there is a new subfolder called "bio", that did not exist in the old version.
If I ssh to the server, cd to /var/www/html/hugo/MYWEBSITE and run "ls -l" RIGHT AFTER running rsync, I see the "bio" subfolder. If I do "ls -l" 3 minutes later, the "bio" subfolder is no more.
I have repeated the rsync process 5/6 times before writing this email, and it's always the same. Every time I rsync, what I uploaded lasts 2/3 minutes, then the WHOLE folder on the server is erased and refilled with the previous full version of the website.
It's as if there were some hidden cron job somewhere that runs hugo on the OLD source files, but I can't find it. All the standard methods one can find by googling "how to list all cron jobs" don't show anything that may be the reason. THANKS in advance for any help, Marco _______________________________________________
In synthesis:
- whatever the problem was, it was only on the server, no browser
stuff. I had said since the beginning that I could see the HTML folders reappear by doing "ls -l" on the server.
the server has no cpanel, no proxies, no security appliance etc, nothing of the sort. It is a VPS with the bare minimum, administered exclusively via SSH and cron jobs that never dido anything to **generate** web pages
I did look inside /var/log, top, systemd timer units in the systemd unit directories, but could not recognize anything connected to this problem
How I "solved" the problem:
I do have a shell script on the server that I used to run hugo with the right parameters to regenerate the website, but I NEVER included it in any cron job. I only executed it manually when needed, because the site was seldom updated. Well, when I changed name to that script and made it NOT executable, the problem disappeared.
So, I probably I did miss something in the checks above, but no idea what...
Marco
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
-
Christer M. Fekjan -
Christopher Wensink -
Kenneth Porter -
Marco Fioretti -
Robert Heller