Hello!
Sorry to bother you during holidays, but is looks like Duffy has a trouble provisioning CentOS 7 nodes:
``` 2019-12-26 07:36:18,171 [agent-control/allocate_node] INFO: Attempting to allocate a node (7 x86_64) ... 2019-12-26 07:52:19,680 [agent-control/allocate_node] INFO: Waiting 1800 seconds before another retry 2019-12-26 08:22:19,757 [agent-control/_execute_api_command] INFO: Duffy request URL: xxx 2019-12-26 08:22:19,759 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): xxx 2019-12-26 08:22:20,029 [agent-control/allocate_node] ERROR: Received unexpected response from the server: Insufficient Nodes in READY State 2019-12-26 08:22:20,029 [agent-control/allocate_node] INFO: Waiting 3600 seconds before another retry 2019-12-26 09:22:20,126 [agent-control/allocate_node] ERROR: Failed to allocated a node 2019-12-26 09:22:20,126 [agent-control/<module>] CRITICAL: Can't continue without a valid node ```
On Thu, Dec 26, 2019 at 3:10 PM František Šumšal frantisek@sumsal.cz wrote:
Hello!
Sorry to bother you during holidays, but is looks like Duffy has a trouble provisioning CentOS 7 nodes:
That's totally alright Thankyou for letting me know :) I have fixed it and in a couple of minutes, things should be back to normal
2019-12-26 07:36:18,171 [agent-control/allocate_node] INFO: Attempting to allocate a node (7 x86_64) ... 2019-12-26 07:52:19,680 [agent-control/allocate_node] INFO: Waiting 1800 seconds before another retry 2019-12-26 08:22:19,757 [agent-control/_execute_api_command] INFO: Duffy request URL: xxx 2019-12-26 08:22:19,759 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): xxx 2019-12-26 08:22:20,029 [agent-control/allocate_node] ERROR: Received unexpected response from the server: Insufficient Nodes in READY State 2019-12-26 08:22:20,029 [agent-control/allocate_node] INFO: Waiting 3600 seconds before another retry 2019-12-26 09:22:20,126 [agent-control/allocate_node] ERROR: Failed to allocated a node 2019-12-26 09:22:20,126 [agent-control/<module>] CRITICAL: Can't continue without a valid node-- Frantisek Sumsal GPG key ID: 0xFB738CE27B634E4B
On 12/26/19 10:44 AM, Vipul Siddharth wrote:
On Thu, Dec 26, 2019 at 3:10 PM František Šumšal frantisek@sumsal.cz wrote:
Hello!
Sorry to bother you during holidays, but is looks like Duffy has a trouble provisioning CentOS 7 nodes:
That's totally alright Thankyou for letting me know :) I have fixed it and in a couple of minutes, things should be back to normal
They indeed are, thank you very much!
2019-12-26 07:36:18,171 [agent-control/allocate_node] INFO: Attempting to allocate a node (7 x86_64) ... 2019-12-26 07:52:19,680 [agent-control/allocate_node] INFO: Waiting 1800 seconds before another retry 2019-12-26 08:22:19,757 [agent-control/_execute_api_command] INFO: Duffy request URL: xxx 2019-12-26 08:22:19,759 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): xxx 2019-12-26 08:22:20,029 [agent-control/allocate_node] ERROR: Received unexpected response from the server: Insufficient Nodes in READY State 2019-12-26 08:22:20,029 [agent-control/allocate_node] INFO: Waiting 3600 seconds before another retry 2019-12-26 09:22:20,126 [agent-control/allocate_node] ERROR: Failed to allocated a node 2019-12-26 09:22:20,126 [agent-control/<module>] CRITICAL: Can't continue without a valid node-- Frantisek Sumsal GPG key ID: 0xFB738CE27B634E4B
Hello again,
Unfortunately, it seems like Duffy can't provision CentOS 7 nodes once again since yesterday's night:
``` 2019-12-28 22:17:42,883 [agent-control/allocate_node] INFO: Attempting to allocate a node (7 x86_64) 2019-12-28 22:17:42,883 [agent-control/_execute_api_command] INFO: Duffy request URL: xxx 2019-12-28 22:17:42,890 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): xxx 2019-12-28 22:17:43,176 [agent-control/allocate_node] ERROR: Received unexpected response from the server: Insufficient Nodes in READY State 2019-12-28 22:17:43,176 [agent-control/allocate_node] INFO: Waiting 60 seconds before another retry ```
could you, please, look into that?
Thank you!
On 12/26/19 10:40 AM, František Šumšal wrote:
Hello!
Sorry to bother you during holidays, but is looks like Duffy has a trouble provisioning CentOS 7 nodes:
2019-12-26 07:36:18,171 [agent-control/allocate_node] INFO: Attempting to allocate a node (7 x86_64) ... 2019-12-26 07:52:19,680 [agent-control/allocate_node] INFO: Waiting 1800 seconds before another retry 2019-12-26 08:22:19,757 [agent-control/_execute_api_command] INFO: Duffy request URL: xxx 2019-12-26 08:22:19,759 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): xxx 2019-12-26 08:22:20,029 [agent-control/allocate_node] ERROR: Received unexpected response from the server: Insufficient Nodes in READY State 2019-12-26 08:22:20,029 [agent-control/allocate_node] INFO: Waiting 3600 seconds before another retry 2019-12-26 09:22:20,126 [agent-control/allocate_node] ERROR: Failed to allocated a node 2019-12-26 09:22:20,126 [agent-control/<module>] CRITICAL: Can't continue without a valid node
CI-users mailing list CI-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
On Sun, Dec 29, 2019 at 3:55 PM František Šumšal frantisek@sumsal.cz wrote:
Hello again,
Unfortunately, it seems like Duffy can't provision CentOS 7 nodes once again since yesterday's night:
2019-12-28 22:17:42,883 [agent-control/allocate_node] INFO: Attempting to allocate a node (7 x86_64) 2019-12-28 22:17:42,883 [agent-control/_execute_api_command] INFO: Duffy request URL: xxx 2019-12-28 22:17:42,890 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): xxx 2019-12-28 22:17:43,176 [agent-control/allocate_node] ERROR: Received unexpected response from the server: Insufficient Nodes in READY State 2019-12-28 22:17:43,176 [agent-control/allocate_node] INFO: Waiting 60 seconds before another retrycould you, please, look into that?
Sorry for these inconsistencies I am looking into it
Thank you!
On 12/26/19 10:40 AM, František Šumšal wrote:
Hello!
Sorry to bother you during holidays, but is looks like Duffy has a trouble provisioning CentOS 7 nodes:
2019-12-26 07:36:18,171 [agent-control/allocate_node] INFO: Attempting to allocate a node (7 x86_64) ... 2019-12-26 07:52:19,680 [agent-control/allocate_node] INFO: Waiting 1800 seconds before another retry 2019-12-26 08:22:19,757 [agent-control/_execute_api_command] INFO: Duffy request URL: xxx 2019-12-26 08:22:19,759 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): xxx 2019-12-26 08:22:20,029 [agent-control/allocate_node] ERROR: Received unexpected response from the server: Insufficient Nodes in READY State 2019-12-26 08:22:20,029 [agent-control/allocate_node] INFO: Waiting 3600 seconds before another retry 2019-12-26 09:22:20,126 [agent-control/allocate_node] ERROR: Failed to allocated a node 2019-12-26 09:22:20,126 [agent-control/<module>] CRITICAL: Can't continue without a valid node
CI-users mailing list CI-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
-- Frantisek Sumsal GPG key ID: 0xFB738CE27B634E4B
On Sun, Dec 29, 2019 at 3:55 PM František Šumšal frantisek@sumsal.cz wrote:
Hello again,
Unfortunately, it seems like Duffy can't provision CentOS 7 nodes once again since yesterday's night:
The problem seems to be with 8-stream nodes. I have disabled 8-stream nodes for now (duffy won't try to provision these) till I have identified the problem. Will keep you updated with the progress.
2019-12-28 22:17:42,883 [agent-control/allocate_node] INFO: Attempting to allocate a node (7 x86_64) 2019-12-28 22:17:42,883 [agent-control/_execute_api_command] INFO: Duffy request URL: xxx 2019-12-28 22:17:42,890 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): xxx 2019-12-28 22:17:43,176 [agent-control/allocate_node] ERROR: Received unexpected response from the server: Insufficient Nodes in READY State 2019-12-28 22:17:43,176 [agent-control/allocate_node] INFO: Waiting 60 seconds before another retrycould you, please, look into that?
Thank you!
On 12/26/19 10:40 AM, František Šumšal wrote:
Hello!
Sorry to bother you during holidays, but is looks like Duffy has a trouble provisioning CentOS 7 nodes:
2019-12-26 07:36:18,171 [agent-control/allocate_node] INFO: Attempting to allocate a node (7 x86_64) ... 2019-12-26 07:52:19,680 [agent-control/allocate_node] INFO: Waiting 1800 seconds before another retry 2019-12-26 08:22:19,757 [agent-control/_execute_api_command] INFO: Duffy request URL: xxx 2019-12-26 08:22:19,759 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): xxx 2019-12-26 08:22:20,029 [agent-control/allocate_node] ERROR: Received unexpected response from the server: Insufficient Nodes in READY State 2019-12-26 08:22:20,029 [agent-control/allocate_node] INFO: Waiting 3600 seconds before another retry 2019-12-26 09:22:20,126 [agent-control/allocate_node] ERROR: Failed to allocated a node 2019-12-26 09:22:20,126 [agent-control/<module>] CRITICAL: Can't continue without a valid node
CI-users mailing list CI-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
-- Frantisek Sumsal GPG key ID: 0xFB738CE27B634E4B
On Sun, Dec 29, 2019 at 4:07 PM Vipul Siddharth vipul@redhat.com wrote:
On Sun, Dec 29, 2019 at 3:55 PM František Šumšal frantisek@sumsal.cz wrote:
Hello again,
Unfortunately, it seems like Duffy can't provision CentOS 7 nodes once again since yesterday's night:
The problem seems to be with 8-stream nodes. I have disabled 8-stream nodes for now (duffy won't try to provision these) till I have identified the problem.
8-Stream is back in the pool since yesterday and since then I have been monitoring 8-stream nodes closely to see if they cause similar problems. So far, all looks good. In case you notice anything, please let me know asap and I will attend it immediately. Thank you
Will keep you updated with the progress.
2019-12-28 22:17:42,883 [agent-control/allocate_node] INFO: Attempting to allocate a node (7 x86_64) 2019-12-28 22:17:42,883 [agent-control/_execute_api_command] INFO: Duffy request URL: xxx 2019-12-28 22:17:42,890 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): xxx 2019-12-28 22:17:43,176 [agent-control/allocate_node] ERROR: Received unexpected response from the server: Insufficient Nodes in READY State 2019-12-28 22:17:43,176 [agent-control/allocate_node] INFO: Waiting 60 seconds before another retrycould you, please, look into that?
Thank you!
On 12/26/19 10:40 AM, František Šumšal wrote:
Hello!
Sorry to bother you during holidays, but is looks like Duffy has a trouble provisioning CentOS 7 nodes:
2019-12-26 07:36:18,171 [agent-control/allocate_node] INFO: Attempting to allocate a node (7 x86_64) ... 2019-12-26 07:52:19,680 [agent-control/allocate_node] INFO: Waiting 1800 seconds before another retry 2019-12-26 08:22:19,757 [agent-control/_execute_api_command] INFO: Duffy request URL: xxx 2019-12-26 08:22:19,759 [connectionpool/_new_conn] INFO: Starting new HTTP connection (1): xxx 2019-12-26 08:22:20,029 [agent-control/allocate_node] ERROR: Received unexpected response from the server: Insufficient Nodes in READY State 2019-12-26 08:22:20,029 [agent-control/allocate_node] INFO: Waiting 3600 seconds before another retry 2019-12-26 09:22:20,126 [agent-control/allocate_node] ERROR: Failed to allocated a node 2019-12-26 09:22:20,126 [agent-control/<module>] CRITICAL: Can't continue without a valid node
CI-users mailing list CI-users@centos.org https://lists.centos.org/mailman/listinfo/ci-users
-- Frantisek Sumsal GPG key ID: 0xFB738CE27B634E4B
-- Vipul Siddharth He/His/Him Fedora | CentOS CI Infrastructure Team Red Hat w: vipul.dev
On 1/3/20 11:09 AM, Vipul Siddharth wrote:
On Sun, Dec 29, 2019 at 4:07 PM Vipul Siddharth vipul@redhat.com wrote:
On Sun, Dec 29, 2019 at 3:55 PM František Šumšal frantisek@sumsal.cz wrote:
Hello again,
Unfortunately, it seems like Duffy can't provision CentOS 7 nodes once again since yesterday's night:
The problem seems to be with 8-stream nodes. I have disabled 8-stream nodes for now (duffy won't try to provision these) till I have identified the problem.
8-Stream is back in the pool since yesterday and since then I have been monitoring 8-stream nodes closely to see if they cause similar problems. So far, all looks good. In case you notice anything, please let me know asap and I will attend it immediately. Thank you
Everything seems to be in order, thank you very much!
-
František Šumšal -
Vipul Siddharth