Okay, I know about 'cron' and 'at', but how do you schedule jobs (scripts) based on events.
For instance: Run script xyz when file abcd.efg arrives via ftp?
TIA,
Frank M. Ramaekers Jr. Systems Programmer; MCP, MCP+I, MCSE & RHCE American Income Life Insurance Company Phone: (254) 761-6649 Fax: (254) 741-5777
You could schedule (cron) a script that checks for abcd.efg and if it is there, do something.
Andrew Cotter Manager of Information Systems Somerset Capital Group, Ltd. P: 203.336.7827 Andrew.Cotter@SomersetCapital.com www.SomersetCapital.com
-----Original Message----- From: "Frank M. Ramaekers" FRamaekers@ailife.com Date: Fri, 16 Feb 2007 09:03:04 To:"CentOS mailing list" centos@centos.org Subject: [CentOS] Event scheduling
Okay, I know about 'cron' and 'at', but how do you schedule jobs (scripts) based on events.
For instance: Run script xyz when file abcd.efg arrives via ftp?
TIA,
Frank M. Ramaekers Jr. Systems Programmer; MCP, MCP+I, MCSE & RHCE American Income Life Insurance Company Phone: (254) 761-6649 Fax: (254) 741-5777
_______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Sure I understand that, but If I wanted it to be within 15 seconds response, I'd have to check 4times x 60 seconds x 60 minutes x 24 hours per day. Kind of a waste for a file that may or may not be transmitted each day.
I was thinking that perhaps there was something that would monitor the SYSLOG and could trigger jobs (scripts) based on a syslog entry.
Frank M. Ramaekers Jr. Systems Programmer; MCP, MCP+I, MCSE & RHCE American Income Life Insurance Company Phone: (254) 761-6649 Fax: (254) 741-5777 -----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Andrew Cotter Sent: Friday, February 16, 2007 9:42 AM To: CentOS mailing list Subject: Re: [CentOS] Event scheduling
You could schedule (cron) a script that checks for abcd.efg and if it is there, do something.
Andrew Cotter Manager of Information Systems Somerset Capital Group, Ltd. P: 203.336.7827 Andrew.Cotter@SomersetCapital.com www.SomersetCapital.com
-----Original Message----- From: "Frank M. Ramaekers" FRamaekers@ailife.com Date: Fri, 16 Feb 2007 09:03:04 To:"CentOS mailing list" centos@centos.org Subject: [CentOS] Event scheduling
Okay, I know about 'cron' and 'at', but how do you schedule jobs (scripts) based on events.
For instance: Run script xyz when file abcd.efg arrives via ftp?
TIA,
Frank M. Ramaekers Jr. Systems Programmer; MCP, MCP+I, MCSE & RHCE American Income Life Insurance Company Phone: (254) 761-6649 Fax: (254) 741-5777
_______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
_______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Is it definately coming in via ftp?
Andrew Cotter Manager of Information Systems Somerset Capital Group, Ltd. P: 203.336.7827 Andrew.Cotter@SomersetCapital.com www.SomersetCapital.com
-----Original Message----- From: "Frank M. Ramaekers" FRamaekers@ailife.com Date: Fri, 16 Feb 2007 09:48:45 To:"CentOS mailing list" centos@centos.org Subject: RE: [CentOS] Event scheduling
Sure I understand that, but If I wanted it to be within 15 seconds response, I'd have to check 4times x 60 seconds x 60 minutes x 24 hours per day. Kind of a waste for a file that may or may not be transmitted each day.
I was thinking that perhaps there was something that would monitor the SYSLOG and could trigger jobs (scripts) based on a syslog entry.
Frank M. Ramaekers Jr. Systems Programmer; MCP, MCP+I, MCSE & RHCE American Income Life Insurance Company Phone: (254) 761-6649 Fax: (254) 741-5777 -----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Andrew Cotter Sent: Friday, February 16, 2007 9:42 AM To: CentOS mailing list Subject: Re: [CentOS] Event scheduling
You could schedule (cron) a script that checks for abcd.efg and if it is there, do something.
Andrew Cotter Manager of Information Systems Somerset Capital Group, Ltd. P: 203.336.7827 Andrew.Cotter@SomersetCapital.com www.SomersetCapital.com
-----Original Message----- From: "Frank M. Ramaekers" FRamaekers@ailife.com Date: Fri, 16 Feb 2007 09:03:04 To:"CentOS mailing list" centos@centos.org Subject: [CentOS] Event scheduling
Okay, I know about 'cron' and 'at', but how do you schedule jobs (scripts) based on events.
For instance: Run script xyz when file abcd.efg arrives via ftp?
TIA,
Frank M. Ramaekers Jr. Systems Programmer; MCP, MCP+I, MCSE & RHCE American Income Life Insurance Company Phone: (254) 761-6649 Fax: (254) 741-5777
_______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
_______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Yes, via vsFTPd
Frank M. Ramaekers Jr. Systems Programmer; MCP, MCP+I, MCSE & RHCE American Income Life Insurance Company Phone: (254) 761-6649 Fax: (254) 741-5777 -----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Andrew Cotter Sent: Friday, February 16, 2007 9:53 AM To: CentOS mailing list Subject: Re: [CentOS] Event scheduling
Is it definately coming in via ftp?
Andrew Cotter Manager of Information Systems Somerset Capital Group, Ltd. P: 203.336.7827 Andrew.Cotter@SomersetCapital.com www.SomersetCapital.com
-----Original Message----- From: "Frank M. Ramaekers" FRamaekers@ailife.com Date: Fri, 16 Feb 2007 09:48:45 To:"CentOS mailing list" centos@centos.org Subject: RE: [CentOS] Event scheduling
Sure I understand that, but If I wanted it to be within 15 seconds response, I'd have to check 4times x 60 seconds x 60 minutes x 24 hours per day. Kind of a waste for a file that may or may not be transmitted each day.
I was thinking that perhaps there was something that would monitor the SYSLOG and could trigger jobs (scripts) based on a syslog entry.
Frank M. Ramaekers Jr. Systems Programmer; MCP, MCP+I, MCSE & RHCE American Income Life Insurance Company Phone: (254) 761-6649 Fax: (254) 741-5777 -----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Andrew Cotter Sent: Friday, February 16, 2007 9:42 AM To: CentOS mailing list Subject: Re: [CentOS] Event scheduling
You could schedule (cron) a script that checks for abcd.efg and if it is there, do something.
Andrew Cotter Manager of Information Systems Somerset Capital Group, Ltd. P: 203.336.7827 Andrew.Cotter@SomersetCapital.com www.SomersetCapital.com
-----Original Message----- From: "Frank M. Ramaekers" FRamaekers@ailife.com Date: Fri, 16 Feb 2007 09:03:04 To:"CentOS mailing list" centos@centos.org Subject: [CentOS] Event scheduling
Okay, I know about 'cron' and 'at', but how do you schedule jobs (scripts) based on events.
For instance: Run script xyz when file abcd.efg arrives via ftp?
TIA,
Frank M. Ramaekers Jr. Systems Programmer; MCP, MCP+I, MCSE & RHCE American Income Life Insurance Company Phone: (254) 761-6649 Fax: (254) 741-5777
_______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
_______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Fri, 16 Feb 2007 10:34:55 -0600 "Frank M. Ramaekers" FRamaekers@ailife.com wrote:
Yes, via vsFTPd
Then use pure-ftpd, iirc it has the capability to run a script after uploaded file.
Sure I understand that, but If I wanted it to be within 15 seconds response, I'd have to check 4times x 60 seconds x 60 minutes x 24 hours per day. Kind of a waste for a file that may or may not be transmitted each day.
I was thinking that perhaps there was something that would monitor the SYSLOG and could trigger jobs (scripts) based on a syslog entry.
You could use syslog-ng, and some verbose logging on your FTP daemon. Setup a filter to either trigger based on a file transfer, or based on a specific file name. based on that filter, syslog-ng can launch a program, or send data to a pipe that could be picked up by a simple daemon...
The syslog-ng FAQ has some good examples.
Mike
Frank M. Ramaekers wrote:
Sure I understand that, but If I wanted it to be within 15 seconds response, I'd have to check 4times x 60 seconds x 60 minutes x 24 hours per day. Kind of a waste for a file that may or may not be transmitted each day.
Usually any machine that can handle actually doing a job will have no problem with the work checking and not doing it the rest of the time... If you are concerned about it, a stat() is pretty cheap compared to starting a program to do it, so you might write a long-running program to monitor for a new file or a timestamp change instead of starting one every few seconds. If you really want a file-event driven interface you might look at dazuko. http://www.dazuko.de/faq.shtml
Les Mikesell wrote:
Frank M. Ramaekers wrote:
Sure I understand that, but If I wanted it to be within 15 seconds response, I'd have to check 4times x 60 seconds x 60 minutes x 24 hours per day. Kind of a waste for a file that may or may not be transmitted each day.
Usually any machine that can handle actually doing a job will have no problem with the work checking and not doing it the rest of the time... If you are concerned about it, a stat() is pretty cheap compared to starting a program to do it, so you might write a long-running program to monitor for a new file or a timestamp change instead of starting one every few seconds. If you really want a file-event driven interface you might look at dazuko. http://www.dazuko.de/faq.shtml
That's not a very good thing to do in a virtual machine (by way of example). Contemplate doing this in a zSeries box where there are likely 100 or more virtual machines.
John Summerfield wrote:
Les Mikesell wrote:
Frank M. Ramaekers wrote:
Sure I understand that, but If I wanted it to be within 15 seconds response, I'd have to check 4times x 60 seconds x 60 minutes x 24 hours per day. Kind of a waste for a file that may or may not be transmitted each day.
Usually any machine that can handle actually doing a job will have no problem with the work checking and not doing it the rest of the time... If you are concerned about it, a stat() is pretty cheap compared to starting a program to do it, so you might write a long-running program to monitor for a new file or a timestamp change instead of starting one every few seconds. If you really want a file-event driven interface you might look at dazuko. http://www.dazuko.de/faq.shtml
That's not a very good thing to do in a virtual machine (by way of example). Contemplate doing this in a zSeries box where there are likely 100 or more virtual machines.
If a stat() every 15 seconds causes a problem, even with hundreds of them running at once you need to look for a different machine. Since you are doing it frequently the inode in question will be in cache and this will be a very lightweight operation.
On Fri, Feb 16, 2007 at 09:48:45AM -0600, Frank M. Ramaekers wrote:
Sure I understand that, but If I wanted it to be within 15 seconds response, I'd have to check 4times x 60 seconds x 60 minutes x 24 hours per day. Kind of a waste for a file that may or may not be transmitted each day.
I was thinking that perhaps there was something that would monitor the SYSLOG and could trigger jobs (scripts) based on a syslog entry.
You mean...
tail -f messages | grep "line.i.want" | while read line do something $line done
Be careful with log rotation.
Stephen Harris wrote:
On Fri, Feb 16, 2007 at 09:48:45AM -0600, Frank M. Ramaekers wrote:
Sure I understand that, but If I wanted it to be within 15 seconds response, I'd have to check 4times x 60 seconds x 60 minutes x 24 hours per day. Kind of a waste for a file that may or may not be transmitted each day.
I was thinking that perhaps there was something that would monitor the SYSLOG and could trigger jobs (scripts) based on a syslog entry.
You mean...
tail -f messages | grep "line.i.want" | while read line do something $line done
Be careful with log rotation.
Look for the pop-before-smtp package (It's in Debian, I don't recall where it came from). It's Perl, it monitors the log and copes with log rotation, and could be adapted to your task.
However, that ftp daemon looks good. This wouldn't be good to run in virtual machines either.
On Fri, 16 Feb 2007, Frank M. Ramaekers wrote:
Sure I understand that, but If I wanted it to be within 15 seconds response, I'd have to check 4times x 60 seconds x 60 minutes x 24 hours per day. Kind of a waste for a file that may or may not be transmitted each day.
push v pull, poll v select -- SGI formerly had 'fam' and the fam daemon to make this more economical, and we shipped it in centos-2
http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=bks&...
and
http://oss.sgi.com/projects/fam/
--------------------
If one wanted to enqueue events without polling, use a tail -f on the xferlog, and | grep the desired events, feeding those to an enqueuer, or just fire off a child as each new item is completed -- Such a script can live in the /etc/inittab quite happily, and Linux will handle the scheduling for you.
-- Russ Herrold
R P Herrold wrote:
On Fri, 16 Feb 2007, Frank M. Ramaekers wrote:
Sure I understand that, but If I wanted it to be within 15 seconds response, I'd have to check 4times x 60 seconds x 60 minutes x 24 hours per day. Kind of a waste for a file that may or may not be transmitted each day.
push v pull, poll v select -- SGI formerly had 'fam' and the fam daemon to make this more economical, and we shipped it in centos-2
http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=bks&...
and
CentOS 4 ships with gamin, which is API compatible to fam. CentOS 5 will have inotify.
Ralph
you could try swatch.
LB.
Andrew Cotter wrote:
You could schedule (cron) a script that checks for abcd.efg and if it is there, do something.
Andrew Cotter Manager of Information Systems Somerset Capital Group, Ltd. P: 203.336.7827 Andrew.Cotter@SomersetCapital.com www.SomersetCapital.com
-----Original Message----- From: "Frank M. Ramaekers" FRamaekers@ailife.com Date: Fri, 16 Feb 2007 09:03:04 To:"CentOS mailing list" centos@centos.org Subject: [CentOS] Event scheduling
Okay, I know about 'cron' and 'at', but how do you schedule jobs (scripts) based on events.
For instance: Run script xyz when file abcd.efg arrives via ftp?
TIA,
Frank M. Ramaekers Jr. Systems Programmer; MCP, MCP+I, MCSE & RHCE American Income Life Insurance Company Phone: (254) 761-6649 Fax: (254) 741-5777
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Fri, 16 Feb 2007, Frank M. Ramaekers wrote:
Okay, I know about 'cron' and 'at', but how do you schedule jobs (scripts) based on events.
For instance: Run script xyz when file abcd.efg arrives via ftp?
Emit an event into a queue, and periodically wake up and harvest the queue --
ls and diff can detect newly present files
so can find
procmail can trivially detect and fire off email related enqueuing events
-- Russ Herrold
Frank M. Ramaekers wrote:
Okay, I know about 'cron' and 'at', but how do you schedule jobs (scripts) based on events.
For instance: Run script xyz when file abcd.efg arrives via ftp?
TIA,
Frank M. Ramaekers Jr. Systems Programmer; MCP, MCP+I, MCSE & RHCE American Income Life Insurance Company Phone: (254) 761-6649 Fax: (254) 741-5777
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Hi Frank,
We handle this scenario by having a cron job running a script periodically (every few minutes). This script first checks whether the file is present before trying to process it.
Where multiple files are transferred, we send an empty "marker" file at the end of the transmission cycle. The above script then checks for the presence of this marker, processes the transferred files if it is present and deletes it at the end of the processing cycle.
We transfer several thousand files between servers per day using this technique - with very few problems.
Contact me off-list if you require assistance in setting this up.
Chris Geldenhuis
-
Andrew Cotter -
Chris Geldenhuis -
Frank M. Ramaekers -
John Summerfield -
Jure Pečar -
Les Mikesell -
Lou Baccari -
mike.redan＠bell.ca -
R P Herrold -
Ralph Angenendt -
Stephen Harris