Today, we are experiencing some network failures in our jobs running on ci.centos.org infra and now our jobs even fail to obtain duffy node (examples: [1], [2]). Is there any expected outage, or ci.centos.org is just so unstable? Radim
[1] - https://ci.centos.org/job/devtools-che-functional-tests-prcheck-openshift.io... [2] - https://ci.centos.org/job/devtools-che-functional-tests-prcheck-openshift.io...
Hi,
We are experiencing slave failure [1]
Really not sure if it is related to it or not.
[1] https://ci.centos.org/computer/minishift-ci-slave02/
On Mon, May 21, 2018 at 3:59 PM, Radim Hopp rhopp@redhat.com wrote:
Today, we are experiencing some network failures in our jobs running on ci.centos.org infra and now our jobs even fail to obtain duffy node (examples: [1], [2]). Is there any expected outage, or ci.centos.org is just so unstable? Radim
[1] - https://ci.centos.org/job/devtools-che-functional-tests- prcheck-openshift.io/448/console [2] - https://ci.centos.org/job/devtools-che-functional-tests- prcheck-openshift.io/448/console --
RADIM HOPP
Red Hat Czech s.r.o. https://www.redhat.com/
rhopp@redhat.com IM: rhopp https://red.ht/sig TRIED. TESTED. TRUSTED. https://redhat.com/trusted @redhatway https://twitter.com/redhatway @redhatinc https://instagram.com/redhatinc @redhatsnaps https://snapchat.com/add/redhatsnaps
Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
Regards, Budh Ram Gurung
On Mon, May 21, 2018 at 4:18 PM Budh Ram Gurung bgurung@redhat.com wrote:
Hi,
We are experiencing slave failure [1]
Really not sure if it is related to it or not.
[1] https://ci.centos.org/computer/minishift-ci-slave02/
On Mon, May 21, 2018 at 3:59 PM, Radim Hopp rhopp@redhat.com wrote:
Today, we are experiencing some network failures in our jobs running on ci.centos.org infra and now our jobs even fail to obtain duffy node (examples: [1], [2]). Is there any expected outage, or ci.centos.org is just so unstable? Radim
[1] - https://ci.centos.org/job/devtools-che-functional-tests-prcheck-openshift.io... [2] - https://ci.centos.org/job/devtools-che-functional-tests-prcheck-openshift.io...
Facing the same issue, unable to obtain duffy nodes with the gluster jobs [1]. Maybe it's the same issue that we hit earlier [2].
~kaushal
[1]: https://ci.centos.org/job/gluster_glusterd2/1749/console [2]: https://lists.centos.org/pipermail/ci-users/2018-April/000790.html
I just opened a bug report with more information: https://bugs.centos.org/view.php?id=14847
From what I can see, not a single duffy get node is working.
On Mon, May 21, 2018 at 12:47 PM, Budh Ram Gurung bgurung@redhat.com wrote:
Hi,
We are experiencing slave failure [1]
Really not sure if it is related to it or not.
[1] https://ci.centos.org/computer/minishift-ci-slave02/
On Mon, May 21, 2018 at 3:59 PM, Radim Hopp rhopp@redhat.com wrote:
Today, we are experiencing some network failures in our jobs running on ci.centos.org infra and now our jobs even fail to obtain duffy node (examples: [1], [2]). Is there any expected outage, or ci.centos.org is just so unstable? Radim
[1] - https://ci.centos.org/job/devtools-che-functional-tests-pr check-openshift.io/448/console [2] - https://ci.centos.org/job/devtools-che-functional-tests-pr check-openshift.io/448/console --
RADIM HOPP
Red Hat Czech s.r.o. https://www.redhat.com/
rhopp@redhat.com IM: rhopp https://red.ht/sig TRIED. TESTED. TRUSTED. https://redhat.com/trusted @redhatway https://twitter.com/redhatway @redhatinc https://instagram.com/redhatinc @redhatsnaps https://snapchat.com/add/redhatsnaps
Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
Regards, Budh Ram Gurung
Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
It looks like it's working now.
Thanks!
On Mon, May 21, 2018 at 3:24 PM, Jaime Melis jmelis@redhat.com wrote:
I just opened a bug report with more information: https://bugs.centos.org/view.php?id=14847
From what I can see, not a single duffy get node is working.
On Mon, May 21, 2018 at 12:47 PM, Budh Ram Gurung bgurung@redhat.com wrote:
Hi,
We are experiencing slave failure [1]
Really not sure if it is related to it or not.
[1] https://ci.centos.org/computer/minishift-ci-slave02/
On Mon, May 21, 2018 at 3:59 PM, Radim Hopp rhopp@redhat.com wrote:
Today, we are experiencing some network failures in our jobs running on ci.centos.org infra and now our jobs even fail to obtain duffy node (examples: [1], [2]). Is there any expected outage, or ci.centos.org is just so unstable? Radim
[1] - https://ci.centos.org/job/devtools-che-functional-tests-pr check-openshift.io/448/console [2] - https://ci.centos.org/job/devtools-che-functional-tests-pr check-openshift.io/448/console --
RADIM HOPP
Red Hat Czech s.r.o. https://www.redhat.com/
rhopp@redhat.com IM: rhopp https://red.ht/sig TRIED. TESTED. TRUSTED. https://redhat.com/trusted @redhatway https://twitter.com/redhatway @redhatinc https://instagram.com/redhatinc @redhatsnaps https://snapchat.com/add/redhatsnaps
Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
Regards, Budh Ram Gurung
Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
-- Jaime Melis Senior Software Engineer, OpenShift.io Red Hat jmelis@redhat.com
On May 21 12:29, Radim Hopp wrote:
Today, we are experiencing some network failures in our jobs running on ci.centos.org infra and now our jobs even fail to obtain duffy node (examples: [1], [2]). Is there any expected outage, or ci.centos.org is just so unstable? Radim
[1] - https://ci.centos.org/job/devtools-che-functional-tests-prcheck-openshift.io... [2] - https://ci.centos.org/job/devtools-che-functional-tests-prcheck-openshift.io... --
RADIM HOPP
Red Hat Czech s.r.o. https://www.redhat.com/
rhopp@redhat.com IM: rhopp https://red.ht/sig TRIED. TESTED. TRUSTED. https://redhat.com/trusted @redhatway https://twitter.com/redhatway @redhatinc https://instagram.com/redhatinc @redhatsnaps https://snapchat.com/add/redhatsnaps
Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
Overnight something caused all of our bare metal machines to end up in the "Failed" state. This caused the errors you saw once the ready pool was exhausted.
We took a short time earlier today to fill the ready pool again, and things returned to normal.
Thanks for the report!
--Brian
On 05/21/2018 02:52 PM, Brian Stinson wrote:
Overnight something caused all of our bare metal machines to end up in the "Failed" state. This caused the errors you saw once the ready pool was exhausted.
Has this problem returned? I've been unable to get Bodhi's tests to run this week:
On May 24 20:06, Randy Barlow wrote:
On 05/21/2018 02:52 PM, Brian Stinson wrote:
Overnight something caused all of our bare metal machines to end up in the "Failed" state. This caused the errors you saw once the ready pool was exhausted.
Has this problem returned? I've been unable to get Bodhi's tests to run this week:
https://bugs.centos.org/view.php?id=14852 _______________________________________________ Ci-users mailing list Ci-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
The trouble with the bodhi workspace was a separate issue. In this case, the bodi-ci-slave03 Jenkins agent was in a semi disconnected state. I went ahead and bounced it, so you should be able to run jobs now.
Apologies for the inconvenience.
--Brian
-
Brian Stinson -
Budh Ram Gurung -
Jaime Melis -
Kaushal M -
Radim Hopp -
Randy Barlow