On 03/03/2024 19:48, Fabian Arrotin wrote: > Today evening (Sunday), I got zabbix notification that some services > hosted on same hypervisor were down. > A quick investigation showed me that despite running on a hardware raid > controller, said server firware confirm data loss and corruption. > > As I'm myself normally on PTO, I still wanted to restore services to > quickly working on trying to redeploy from scratch services, and restore > data from last backup and hope to have news soon ... > Status update : cbs.centos.org kojihub was fully reinstalled from scratch on a different hypervisor, reconfigured by Ansible and DB restored from backup that happened earlier today. Quickly checked and it seems all operations are working fine. The only issue you should eventually see is if you submitted a build today, *after* postgresql backup operation took place, so if that's the case, reconsider rebuilding your rpm (but it's usually quite during the weekend, especially on Sunday) Next item to reinstall/restore : git.centos.org -- Fabian Arrotin The CentOS Project | https://www.centos.org gpg key: 17F3B7A1 | @arrfab[@fosstodon.org] -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature.asc Type: application/pgp-signature Size: 840 bytes Desc: OpenPGP digital signature URL: <http://lists.centos.org/pipermail/centos-devel/attachments/20240303/6e957637/attachment.sig>