Hello
I have a large list of URLs (from a database, generated automatically
during tests) that I want to download using several wget processes at
the same time. With our internal web servers, this will be a lot faster
than downloading the pages one at a time with a single process.
So I create 20 pipes in my script with `mkfifo´ and connect the read end
of each one to a new wget process for that fifo. The write end of each
pipe is then connected to my script, with shell commands like `exec
18>>fifo_file_name´
Then my script outputs, in a loop, one line with an URL to each of the
pipes, in turn, and then starts over again with the first pipe until
there are no more URLs from the database client.
Much to my dismay I find that there is no concurrent / parallel download
with the child `wget´ processes, and that for some strange reason only
one wget process can download pages at a time, and after that process
completes, another one can begin.
My script does feed *all* the pipes with data, one line to each pipe in
turn, and has all the pipes written and closed by the time the first
child process has even finished downloading.
Do you know why my child processes manifest this behavior of waiting in
turn for each other in order to start reading the fifo and download ?
I figure it must be something about the pipes, because if I use regular
files instead (and reverse the order: first write the URLs, then start
wget to read them) than the child processes run in parallel as expected.
The child processes also run in parallel if I open the write end of the
pipes first, and the start the wget processes for the read end.
They even run in parallel with my pipes, but I could see them run like
this only for once in all my attempts. I do not know what was special
about that attempt, it happened at the beginning of the day, and the
computers where not restarted nor logged off over night.
The pipes are created and deleted on ever run, with mkfifo and rm.
Is there something special about fifos to make them run in sequence if I
open the read end first ?
My script is attached here, I believe it is nicely formatted and clear
enough.
Thank you,
Timothy Madden