hi,
there are a string of updates backed up behind git.centos.org and the reimzul services. Things that we had in the pipeline from before when Seven came through, and then all the dev work and inside instance work that was done to make Seven go through :: lots of bandaid's in place.
I'd like to schedule some downtime for services, so we can rebase out of some of this technical debt.
towards that, I'd like to propose downtime from 09:00 UTC 20th Aug to 23:00 UTC on the 21st Aug; During this time, we'll need to more or less shutdown everything for some period of time.
The actual outage will be much shorter, and I'll know once i've been able to run through the trial-dry-run, but folks should assume and plan for no services during the two days.
Regards,
On Fri, Aug 8, 2014 at 7:36 PM, Karanbir Singh mail-lists@karan.org wrote:
hi,
there are a string of updates backed up behind git.centos.org and the reimzul services. Things that we had in the pipeline from before when Seven came through, and then all the dev work and inside instance work that was done to make Seven go through :: lots of bandaid's in place.
I'd like to schedule some downtime for services, so we can rebase out of some of this technical debt.
Thank you for the heads up. It's always helpful to be able to *schedule* downtime. The improvement in transparancy of CentOS building that came with git.centos.org has been very nice.
I'm very curious about your disaster recovery setup setup, if any. The numerous public mirrors work well for the RPM/SRPM repositories. But if you can say, is there anything in place for git.centos.org if its hosting data center goes toes up? I've been in the field long enough that I've seen any number of "5 9's guaranteed!' hosting sites get knocked completely off the air for a day or more.
I'd also like to point out that this is one of the occasions when using GPG signed git tags would make it safer for me, as a developer, to clone someone else's git repository while you're off line, and be sure what is actually from the git.centos.org repo and what is not. Wouldn't want that to interfere with cleaning up the band-aid work, just a thought.
On 08/09/2014 01:49 PM, Nico Kadel-Garcia wrote:
On Fri, Aug 8, 2014 at 7:36 PM, Karanbir Singh mail-lists@karan.org wrote:
hi,
there are a string of updates backed up behind git.centos.org and the reimzul services. Things that we had in the pipeline from before when Seven came through, and then all the dev work and inside instance work that was done to make Seven go through :: lots of bandaid's in place.
I'd like to schedule some downtime for services, so we can rebase out of some of this technical debt.
Thank you for the heads up. It's always helpful to be able to *schedule* downtime. The improvement in transparancy of CentOS building that came with git.centos.org has been very nice.
I'm very curious about your disaster recovery setup setup, if any. The numerous public mirrors work well for the RPM/SRPM repositories. But if you can say, is there anything in place for git.centos.org if its hosting data center goes toes up? I've been in the field long enough that I've seen any number of "5 9's guaranteed!' hosting sites get knocked completely off the air for a day or more.
stuff is backed up, geo-locally and distributed.
I'd also like to point out that this is one of the occasions when using GPG signed git tags would make it safer for me, as a developer, to clone someone else's git repository while you're off line, and be sure what is actually from the git.centos.org repo and what is not. Wouldn't want that to interfere with cleaning up the band-aid work, just a thought.
the idea that since git is distributed someone else will have a copy - atleast the last person to send the last commit will have a good copy is best ignored.
just going by history, when large git infra has gone offline - so has most code that was contained inside it.
On Sat, Aug 9, 2014 at 6:10 PM, Karanbir Singh mail-lists@karan.org wrote:
the idea that since git is distributed someone else will have a copy - atleast the last person to send the last commit will have a good copy is best ignored.
just going by history, when large git infra has gone offline - so has most code that was contained inside it.
Not at all. People pull from each other's repositories, especially from their branches, all the time in collaborative work. As things stand, the only way to verify the content is to pull from and compare to the upstream, secured repository, and it's going to be offline for a while.
The window of vulnerability for this particular instance is, thankfully, short But "most code has gone offline" is irrelevant to my concern. It's the potential for confusion, or abuse, and the lack of provenance for offsite clones that concerns me. While git.centos.org is offline, the much-relied-on security of that site itself is quite useless to any developers working rom their own or on each other's repositories.
-
Karanbir Singh -
Nico Kadel-Garcia