In bash, given a string assignment as follows, how do I "add slashes" automagically, so that it can be safely passed to another program? Notice that the assignment contains spaces, single-quotes and double-quotes, maybe god-only-knows-what-else. It's untrusted data.
Yet I need to pass it all *safely*.
The appropriate function in PHP is addslashes(); but what is the bash equivalent? EG:
#! /bin/sh A="This isn't a "parameter""; B=`/path/to/somecommand.sh $A`; exit 0;
Thanks,
-Ben -- Only those who reach toward a goal are likely to achieve it.
On Tue, Feb 26, 2008 at 10:11 AM, Benjamin Smith lists@benjamindsmith.com wrote:
In bash, given a string assignment as follows, how do I "add slashes" automagically, so that it can be safely passed to another program? Notice that the assignment contains spaces, single-quotes and double-quotes, maybe god-only-knows-what-else. It's untrusted data.
Yet I need to pass it all *safely*.
The appropriate function in PHP is addslashes(); but what is the bash equivalent? EG:
short answer: single quotes will handle all characters, except single quotes.
long answer: man bash the section called QUOTING may help you figure a solution.
#! /bin/sh A="This isn't a "parameter"";
B=`/path/to/somecommand.sh $A`; exit 0;
Thanks,
-Ben
HTH, -Bob
On Tuesday 26 February 2008, Bob Beers wrote:
short answer: single quotes will handle all characters, except single
quotes.
long answer: man bash the section called QUOTING may help you figure a solution.
I've read the man page. It helps if I already know the input - I don't have a problem with manually putting slashes in front of spaces and single quotes. But in this case, I don't know the input. It's untrusted data.
There is no mechanism for escaping untrusted input?
-Ben
On Tuesday 26 February 2008, Ralph Angenendt wrote:
There is no mechanism for escaping untrusted input?
Correct. At least there's no magic quoting function.
Ok. So I'm going to have to pull up my sleeves and do this with sed/awk pipes. Got it. I'll quit looking for a simply solution to this (I thought) simple problem.
Now for a more philosophical question....
WHY THE @!#! NOT?!?!?
Bash is used, extensively in many cases, to deal with untrusted data. This can include random file names in user home directories, parameters on various scripts, etc. It's highly sensitive to being passed characters that have, over the past NN years, resulted in quite a number of security holes and problems.
Yet there exists NO MECHANISM for simply ensuring that a given argument is an escaped string?
How many "homebrew" ISP or hosting administration scripts could be compromised by simply putting a file in your home directory called ";rm -rf /" ?
This doesn't strike you as fundamentally borkeD? Why would we accept a work environment that is effectively laden with randomly placed, loaded rat traps? Not trying to bash (ahem) bash needlessly, but this is a problem that so smacks of 1977...
I guess I just hadn't noticed how bad this was, since I started using PHP as shell scripts years ago to run everything, despite the mild performance hit. escapeshallarg() and addslashes() combined with a few backticks provides easy access to the power of the shell, and excellent "don't need to worry about it" security.
This just blows my mind....
-Ben -- Only those who reach toward a goal are likely to achieve it.
Benjamin Smith wrote:
On Tuesday 26 February 2008, Ralph Angenendt wrote:
There is no mechanism for escaping untrusted input?
Correct. At least there's no magic quoting function.
WHY THE @!#! NOT?!?!?
Bash is used, extensively in many cases, to deal with untrusted data. This can include random file names in user home directories, parameters on various scripts, etc. It's highly sensitive to being passed characters that have, over the past NN years, resulted in quite a number of security holes and problems.
Perl is probably better for this.
Yet there exists NO MECHANISM for simply ensuring that a given argument is an escaped string?
How many "homebrew" ISP or hosting administration scripts could be compromised by simply putting a file in your home directory called ";rm -rf /" ?
why would you do that... it'd be much more interesting to do something like ";usermod -u 0 mylogin"
On Tue, Feb 26, 2008 at 08:25:54AM -0800, Benjamin Smith alleged:
On Tuesday 26 February 2008, Ralph Angenendt wrote:
There is no mechanism for escaping untrusted input?
Correct. At least there's no magic quoting function.
Ok. So I'm going to have to pull up my sleeves and do this with sed/awk pipes. Got it. I'll quit looking for a simply solution to this (I thought) simple problem.
Now for a more philosophical question....
WHY THE @!#! NOT?!?!?
Bash is used, extensively in many cases, to deal with untrusted data. This can include random file names in user home directories, parameters on various scripts, etc. It's highly sensitive to being passed characters that have, over the past NN years, resulted in quite a number of security holes and problems.
Yet there exists NO MECHANISM for simply ensuring that a given argument is an escaped string?
How many "homebrew" ISP or hosting administration scripts could be compromised by simply putting a file in your home directory called ";rm -rf /" ?
It's not as bad as you think because of the order of operations.
In all cases, these perform exactly as a string should regardless of inner characters.
$ f='echo a; echo b' $ $f a; echo b
$ dq="echo a; echo b; echo `\ '\ "" $ $dq a; echo b; echo `\ '\ " $ echo $dq echo a; echo b; echo `\ '\ " $ `$dq` -bash: a;: command not found $ `echo $dq` a; echo b; echo `\ '\ "
Garrick Staples wrote:
How many "homebrew" ISP or hosting administration scripts could be compromised by simply putting a file in your home directory called ";rm -rf /" ?
It's not as bad as you think because of the order of operations.
In all cases, these perform exactly as a string should regardless of inner characters.
He's probably thinking of a scripted operation that does a find . -print |xargs some_command (without print0) or a backtick or $(..) generated expansion. A lot of the usefulness of the shell happens because you can generate and reparse text programatically and have it become commands - and a side effect is that metacharacters that appear in the text get processed even if they aren't what you expected. I think it is kind of silly that common shell metacharacters are permitted in filenames, but there's not much you can do about it now.
On Tue, Feb 26, 2008 at 12:45:41PM -0600, Les Mikesell alleged:
Garrick Staples wrote:
How many "homebrew" ISP or hosting administration scripts could be compromised by simply putting a file in your home directory called ";rm -rf /" ?
It's not as bad as you think because of the order of operations.
In all cases, these perform exactly as a string should regardless of inner characters.
He's probably thinking of a scripted operation that does a find . -print |xargs some_command (without print0) or a backtick or $(..) generated expansion. A lot of
Yes, so was I. That's why I had some examples of string with quotes being evaluated by the shell.
the usefulness of the shell happens because you can generate and reparse text programatically and have it become commands - and a side effect is that metacharacters that appear in the text get processed even if they aren't what you expected. I think it is kind of silly that common shell metacharacters are permitted in filenames, but there's not much you can do about it now.
My point is that the problem isn't actually all that bad. Just like all languages, you have to know what you are doing.
Benjamin Smith wrote:
On Tuesday 26 February 2008, Ralph Angenendt wrote:
There is no mechanism for escaping untrusted input?
Correct. At least there's no magic quoting function.
Ok. So I'm going to have to pull up my sleeves and do this with sed/awk pipes. Got it. I'll quit looking for a simply solution to this (I thought) simple problem.
Now for a more philosophical question....
WHY THE @!#! NOT?!?!?
The shell is 'supposed' to be run by a user that is allowed to run any command he wants, and permission/trust issues are handled by the login/authentication process that happens before you get to the shell. If you give the shell a bad command under your own account, it's not supposed to second guess what you wanted.
Bash is used, extensively in many cases, to deal with untrusted data.
Why?
This can include random file names in user home directories, parameters on various scripts, etc. It's highly sensitive to being passed characters that have, over the past NN years, resulted in quite a number of security holes and problems.
If it hurts, don't do it. Build your own argument list and exec programs directly if you want to avoid shell command line parsing.
Yet there exists NO MECHANISM for simply ensuring that a given argument is an escaped string?
What does that mean? If you can define it you can make it happen, but who knows what characters at what depth of quoting will have some special meaning?
How many "homebrew" ISP or hosting administration scripts could be compromised by simply putting a file in your home directory called ";rm -rf /" ?
Probably none that are still in business.
This doesn't strike you as fundamentally borkeD?
No, if you stop bad things from happening, you'll also stop good things.
Why would we accept a work environment that is effectively laden with randomly placed, loaded rat traps? Not trying to bash (ahem) bash needlessly, but this is a problem that so smacks of 1977...
The problem is that you aren't using the shell as intended. If you run it under your own user id, it does exactly what you tell it to do and there is no element of trust involved.
I guess I just hadn't noticed how bad this was, since I started using PHP as shell scripts years ago to run everything, despite the mild performance hit. escapeshallarg() and addslashes() combined with a few backticks provides easy access to the power of the shell, and excellent "don't need to worry about it" security.
Errr what??? Php has about the worst security history of any program around.
This just blows my mind....
What would you like your computer to prevent you from doing to yourself?
On Tuesday 26 February 2008, Les Mikesell wrote:
WHY THE @!#! NOT?!?!?
The shell is 'supposed' to be run by a user that is allowed to run any command he wants, and permission/trust issues are handled by the login/authentication process that happens before you get to the shell. If you give the shell a bad command under your own account, it's not supposed to second guess what you wanted.
I'm not asking for this. I'm only asking for the option to be able to trust that a parameter is... a parameter. EG:
file: script1.sh #! /bin/bash script2.sh $1 exit 0;
file: script2.sh #! /bin/bash echo $1;
$ script1.sh "this\ parameter";
I get output of "this"! script2 gets two parameters! I want a way for 1 parameter to STAY 1 parameter upon request, so that script2.sh would output "this parameter", like
file:script1.sh #! /bin/bash PassToShell2=escapethis $1; script2.sh $PassToShell; exit 0;
Bash is used, extensively in many cases, to deal with untrusted data.
Why?
How about an installer script? How about a magical script copied from TLDP to rename all files in pwd?
This can include random file names in user home directories, parameters on various scripts, etc. It's highly sensitive to being passed characters that have, over the past NN years, resulted in quite a number of security holes and problems.
If it hurts, don't do it. Build your own argument list and exec programs directly if you want to avoid shell command line parsing.
So, I'm supposed to know the contents of a user's home directory? And code for these in advance?
Yet there exists NO MECHANISM for simply ensuring that a given argument is
an
escaped string?
What does that mean? If you can define it you can make it happen, but who knows what characters at what depth of quoting will have some special meaning?
Can I define it? Thought I did that already: http://us.php.net/manual/en/function.escapeshellarg.php
Or its perl equivalent: http://search.cpan.org/~gaas/URI-1.35/URI/Escape.pm
See how I'd like to see it in implementation in above example, "passToShell2"
How many "homebrew" ISP or hosting administration scripts could be
compromised
by simply putting a file in your home directory called ";rm -rf /" ?
Probably none that are still in business.
Google "bash howto" for lots of vulnerable and problematic examples. Here's a beaut that fails if you have a file called "-a" in the pwd, see "File re-namer". It's a renamer that doesn't, if the file contains any spaces, dashes, etc.
http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-12.html#ss12.1
Here's what I get:
mv: invalid option -- a Try `mv --help' for more information.
Or with a file with a space: echo "blah" > "d"; echo "blah" > "d foo";
The TLDP's example doesn't move file "d foo". I get: mv: cannot stat `d': No such file or directory mv: cannot stat `foo': No such file or directory
So I ask again: This doesn't strike you as fundamentally borkeD? The emperor wears no clothes!
This doesn't strike you as fundamentally borkeD?
No, if you stop bad things from happening, you'll also stop good things.
Yes. But you don't have to stop the good things. I think the *OPTION* of saying "parameter 1 is STILL parameter 1" is a good thing. If you want to leave things be, so be it. See my above example.
Why would we accept a work environment that is effectively laden with randomly placed, loaded rat
traps?
Not trying to bash (ahem) bash needlessly, but this is a problem that so smacks of 1977...
The problem is that you aren't using the shell as intended. If you run it under your own user id, it does exactly what you tell it to do and there is no element of trust involved.
The problem, as I see it, is that the shell provides access variables without any means of preserving them as variables across calls and incantations.
I guess I just hadn't noticed how bad this was, since I started using PHP
as
shell scripts years ago to run everything, despite the mild performance
hit.
escapeshallarg() and addslashes() combined with a few backticks provides
easy
access to the power of the shell, and excellent "don't need to worry about it" security.
Errr what??? Php has about the worst security history of any program around.
Thanks for confusing the issue with a red herring. Or should I ignore the buggy and probably vulnerable TLDP example above? Maybe a google search for "bash escape vulnerability" might illuminate the issue I speak of?
This just blows my mind....
What would you like your computer to prevent you from doing to yourself?
I hate to belabor it: give me the OPTION to trust that I can keep a single parameter as a single parameter across incantations and calls. If I'm looping thru a listing files, I should be able to trust that my $FILENAME variable contains the name of... a file! If I want to pass parameter 1 of my script to another script, that other script should be ABLE get my parameter 1 as... parameter 1!
A simple call that lets me take a string and get a passable parameter would suffice.
You want to apply scripts that break with a space in a file name? Great. Have at it. I don't want to detract from your ability to make an alphabet soup of your directories with borken examples from TLDP, and all the goodness that implies to you.
But I think admins the world over would like it if a simple, one-liner could be added to ensure that a single parameter remains a single parameter!
Am I being unclear?
On Tue, Feb 26, 2008 at 11:22:55AM -0800, Benjamin Smith alleged:
On Tuesday 26 February 2008, Les Mikesell wrote:
WHY THE @!#! NOT?!?!?
The shell is 'supposed' to be run by a user that is allowed to run any command he wants, and permission/trust issues are handled by the login/authentication process that happens before you get to the shell. If you give the shell a bad command under your own account, it's not supposed to second guess what you wanted.
I'm not asking for this. I'm only asking for the option to be able to trust that a parameter is... a parameter. EG:
file: script1.sh #! /bin/bash script2.sh $1 exit 0;
file: script2.sh #! /bin/bash echo $1;
$ script1.sh "this\ parameter";
I get output of "this"! script2 gets two parameters! I want a way for 1
You need to quote the variable: #!/bin/bash echo "$1"
parameter to STAY 1 parameter upon request, so that script2.sh would output "this parameter", like
file:script1.sh #! /bin/bash PassToShell2=escapethis $1; script2.sh $PassToShell; exit 0;
You are missing two sets of quotes: #!/bin/bash PassToShell2="escapethis $1" script2.sh "$PassToShell"
[...snip blah blah rant...]
http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-12.html#ss12.1
Here's what I get:
mv: invalid option -- a Try `mv --help' for more information.
That's a bug in the script.
It should be: mv -- "$file" "$file$suffix"
Or with a file with a space: echo "blah" > "d"; echo "blah" > "d foo";
The TLDP's example doesn't move file "d foo". I get: mv: cannot stat `d': No such file or directory mv: cannot stat `foo': No such file or directory
So I ask again: This doesn't strike you as fundamentally borkeD? The emperor wears no clothes!
Just another case of missing double quotes.
It's the programmer that is borked, but the fundamentals :)
[...snip more rants...]
On Tuesday 26 February 2008, Garrick Staples wrote:
I'm not asking for this. I'm only asking for the option to be able to
trust
that a parameter is... a parameter. EG:
file: script1.sh #! /bin/bash script2.sh $1 exit 0;
file: script2.sh #! /bin/bash echo $1;
$ script1.sh "this\ parameter";
I get output of "this"! script2 gets two parameters! I want a way for 1
You need to quote the variable: #!/bin/bash echo "$1"
You missed the point.
In script2.sh, $1 only contains the string "this". There is no safe way to pass $1 (containing string "this parameter") from script1 to script2 as a single, trustable parameter.
You can't do it. Bash is incapable of passing a parameter safely.
You can sorta do it with *$ in the case of spaces. But this is all but powerless against file names containing quotes or other "special" characters. See below Disney example.
So $1 in script 1 contains "this parameter". $1 in script 2 contains "this".
Instead, I have to hork it up with awk, sed, or something similar, and try to account for every possible interpreted character.
(I'm feeling that powerful goodness already! =)
parameter to STAY 1 parameter upon request, so that script2.sh would output "this parameter", like
file:script1.sh #! /bin/bash PassToShell2=escapethis $1; script2.sh $PassToShell; exit 0;
You are missing two sets of quotes: #!/bin/bash PassToShell2="escapethis $1" script2.sh "$PassToShell"
You missed the point here too. Maybe, to make it more clear, try this:
#!/bin/bash PassToShell2=`escapethis $1`; script2.sh "$PassToShell"
escapethis is intended to be a call, not a parameter. Please re-read my earlier note with this new understanding?
[...snip blah blah rant...]
http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-12.html#ss12.1
Here's what I get:
mv: invalid option -- a Try `mv --help' for more information.
That's a bug in the script.
It should be: mv -- "$file" "$file$suffix"
Again, you're missing the point. (practice makes perfect?)
While this "--" on the mv line is a good way to work around the fact that "-a" is being interpreted, it doesn't change the fact that $file is unescaped. The "-a" can be part of a file called "Disney Trip -a mother's journey.doc"
No amount of quoting the on the mv line will change the fact that there is no way to pass a parameter SAFELY. "-a" is an example that can be matched by files with quotes, doublequotes, dashes, semicolons and other characters. Never do we actually have a trustable value in $file, only an interpreted one.
Bash is incapable of passing a parameter SAFELY.
Here are the offending lines:
for file in $* do mv ${file} $prefix$file done
But you didn't read that, did you? Try it yourself!
echo blah > "Disney trip -a mother's journey.doc";
I tried the following code with the above example. Note the quotes!
for file in $* do echo "$file"; done;
called like: /bin/bash ../test.sh *
Disney trip -a mother's journey.doc
5 parameters, one file. Wheeeee!
Bash is incapable of passing a parameter Safely. No amount of quoting will make TLDP's "move a bunch of files" script actually work reliably.
Or with a file with a space: echo "blah" > "d"; echo "blah" > "d foo";
The TLDP's example doesn't move file "d foo". I get: mv: cannot stat `d': No such file or directory mv: cannot stat `foo': No such file or directory
Just another case of missing double quotes.
(Sigh) You missed the point... (see above about bash being incapable of passing a parameter safely)
Explain to "me" where 'ANY' amount 'of' quoting will "fix" this ? If only to yourself... You can sorta do it with `find -print0`, a whispered admission to a bloated, blaring, gaping white elephant of a problem.
I'd like to have a informed discussion, which, apparently, you either aren't interested in, or aren't capable of.
Maybe if I alter the case?
bASH IS INCAPABLE OF PASSING A PARAMETER safely. bAsH iS iNcApAbLe Of PaSsInG a PaRaMeTeR sAfElY.
I mean, argue with me if you want on how my scripts are implemented but the previous two (TRUE) sentences sound like a philosophical deficiency to me.
But seriously, why do you consider this OK? Is it an ego thing?
-Ben
On Tue, Feb 26, 2008 at 12:40:06PM -0800, Benjamin Smith alleged:
I'd like to have a informed discussion, which, apparently, you either aren't interested in, or aren't capable of.
*shrug* I thought we were having a discussion. I'll leave you to it and stay out of your way.
For someone who apparently has no idea what he's talking about, you sure say a lot.
On Tue, Feb 26, 2008 at 12:40 PM, Benjamin Smith lists@benjamindsmith.com wrote:
You missed the point.
No, you missed it. You need the quotes *everywhere* that a variable is referenced.
In script2.sh, $1 only contains the string "this". There is no safe way to pass $1 (containing string "this parameter") from script1 to script2 as a single, trustable parameter.
file: script1.sh #! /bin/bash script2.sh "$1" # Doesn't help to quote in script2 if not quoted in script1 exit 0;
file: script2.sh #! /bin/bash echo "$1";
Here are the offending lines:
for file in $* do mv ${file} $prefix$file done
for file in "$@" do mv -- "${file}" "$prefix$file" done
No amount of quoting will make TLDP's "move a bunch of files" script actually work reliably.
That was a bad URL to have pointed you to, because that's a horrible example of shell programming. I hope "felix hudson" has gotten a bit smarter since then. However, just because felix wrote a bad script does not make "bash is incapable ..." true, any more than you chanting it repeatedly does.
On Tuesday 26 February 2008, Bart Schaefer wrote:
For someone who apparently has no idea what he's talking about, you sure say a lot.
Sorry. It's how I think aloud. Sorry if I offended.
No, you missed it. You need the quotes *everywhere* that a variable is referenced.
Yes, I missed this point. I now see the error in my ways.
In script2.sh, $1 only contains the string "this". There is no safe way
to
pass $1 (containing string "this parameter") from script1 to script2 as a single, trustable parameter.
file: script1.sh #! /bin/bash script2.sh "$1" # Doesn't help to quote in script2 if not quoted in script1 exit 0;
file: script2.sh #! /bin/bash echo "$1";
This is the point that I missed. (hat in hand)
Here are the offending lines:
for file in $* do mv ${file} $prefix$file done
for file in "$@" do mv -- "${file}" "$prefix$file" done
No amount of quoting will make TLDP's "move a bunch of files" script actually work reliably.
That was a bad URL to have pointed you to, because that's a horrible example of shell programming. I hope "felix hudson" has gotten a bit smarter since then. However, just because felix wrote a bad script does not make "bash is incapable ..." true, any more than you chanting it repeatedly does.
It's a bad URL that's also very commonly referenced.
Unless I'm terribly mistaken (again?), the only way I've been able to see "loop thru a list of files" work reliably is with "find" using the "-print0" option, in cahoots with xargs.
Is there any other way?
-Ben -- Only those who reach toward a goal are likely to achieve it.
Unless I'm terribly mistaken (again?), the only way I've been able to see "loop thru a list of files" work reliably is with "find" using the "-print0" option, in cahoots with xargs.
Is there any other way?
-Ben
If I understand you correctly, you are referring to the problem caused by spaces in filenames? Steve mentioned the environment variable IFS ("individual field separator" if memory serves me correctly). By default it's space, tab, or newline. You can change that in your script to be newline only in order to process file names with spaces in it, and then change it back afterwards (so save the value of $IFS at the beginning of the script to something like Default_IFS and then just prior to exiting the script reassign that value back to IFS to return it to its original state). If that's what you are looking at doing I'm sure someone here can fill in the blanks on that one. If not when I get to the office tomorrow I can have a look at some of my scripts where I had to do that and post sample code for you.
Jacques B.
On Tue, Feb 26, 2008 at 05:30:12PM -0500, Jacques B. wrote:
If I understand you correctly, you are referring to the problem caused by spaces in filenames? Steve mentioned the environment variable IFS ("individual field separator" if memory serves me correctly). By default it's space, tab, or newline. You can change that in your script to be newline only in order to process file names with spaces in it, and then change it back afterwards (so save the value of $IFS at the beginning of the script to something like Default_IFS and then just prior to exiting the script reassign that value back to IFS to return it to its original state). If that's what you are looking at
You don't need to do any of that in a script, because scripts are run as a sub-process and don't impact the current environment. You only need to save/restore IFS if you're doing this as part of a larger script (or as a function called in the current shell).
However, spaces AREN'T an issue with proper quoting.
$ touch "a file with spaces in" $ touch "another file" $ ls a file with spaces in another file $ for a in *
do echo "File: $a" done
File: a file with spaces in File: another file
Indeed, carriage returns aren't an issue either!
$ a=$(echo "a\nb") $ touch "$a" $ touch c $ ls a?b c [ Note the ? in the ls output; that's "ls" saying there's a funny character! ] $ for a in *
do echo "File: $a" done
File: a b File: c
All works nicely.
You only need to use find if you're doing things deep down in a directory tree.
Benjamin Smith wrote:
Unless I'm terribly mistaken (again?), the only way I've been able to
see "loop thru a list of files" work reliably is with "find" using the "-print0" option, in cahoots with xargs.
Is there any other way?
for $file in wildcard* do ls -l "$file" done
But this is the point where you should be asking what to do about quotes embedded in the filenames which won't hurt here because of the order of operations but would if you tried to collect the strings and use them in some other ways.
Does anyone have a quick reference to the order of operations as the shell parses a command line (variable parsing,i/o redirection, wildcard and variable expansion, splitting on IFS, quote removal, command substitution etc.)? That's really the first thing you need to know about the shell and if there is a simple description it must be buried in the middle of some obscure manual.
On Tue, Feb 26, 2008 at 04:33:30PM -0600, Les Mikesell alleged:
Does anyone have a quick reference to the order of operations as the shell parses a command line (variable parsing,i/o redirection, wildcard and variable expansion, splitting on IFS, quote removal, command substitution etc.)? That's really the first thing you need to know about the shell and if there is a simple description it must be buried in the middle of some obscure manual.
This is from the "EXPANSION" section of the bash manpage:
The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion.
Garrick Staples wrote:
On Tue, Feb 26, 2008 at 04:33:30PM -0600, Les Mikesell alleged:
Does anyone have a quick reference to the order of operations as the shell parses a command line (variable parsing,i/o redirection, wildcard and variable expansion, splitting on IFS, quote removal, command substitution etc.)? That's really the first thing you need to know about the shell and if there is a simple description it must be buried in the middle of some obscure manual.
This is from the "EXPANSION" section of the bash manpage:
The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion.
That's one step in the bigger picture. I want the one that includes variable assignment, i/o redirection, quote removal, and a few other operations. I think I knew that a few decades ago, but now I don't even know where to look it up.
On Tue, Feb 26, 2008 at 05:13:12PM -0600, Les Mikesell alleged:
Garrick Staples wrote:
On Tue, Feb 26, 2008 at 04:33:30PM -0600, Les Mikesell alleged:
Does anyone have a quick reference to the order of operations as the shell parses a command line (variable parsing,i/o redirection, wildcard and variable expansion, splitting on IFS, quote removal, command substitution etc.)? That's really the first thing you need to know about the shell and if there is a simple description it must be buried in the middle of some obscure manual.
This is from the "EXPANSION" section of the bash manpage:
The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion.That's one step in the bigger picture. I want the one that includes variable assignment, i/o redirection, quote removal, and a few other operations. I think I knew that a few decades ago, but now I don't even know where to look it up.
That's pretty much the entire process for your basic expression. Quotes are obeyed the entire time, but are actually _removed_ after the expansion. And finally, file descriptors are opened the command is executed.
I don't think you can write a simple list because the actual process is too complex. It would really be a tree or flowchart.
Garrick Staples wrote:
On Tue, Feb 26, 2008 at 05:13:12PM -0600, Les Mikesell alleged:
Garrick Staples wrote:
On Tue, Feb 26, 2008 at 04:33:30PM -0600, Les Mikesell alleged:
Does anyone have a quick reference to the order of operations as the shell parses a command line (variable parsing,i/o redirection, wildcard and variable expansion, splitting on IFS, quote removal, command substitution etc.)? That's really the first thing you need to know about the shell and if there is a simple description it must be buried in the middle of some obscure manual.
This is from the "EXPANSION" section of the bash manpage:
The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion.That's one step in the bigger picture. I want the one that includes variable assignment, i/o redirection, quote removal, and a few other operations. I think I knew that a few decades ago, but now I don't even know where to look it up.
That's pretty much the entire process for your basic expression.
Yes, but I'm looking for what happens before and after. Why does unset foo foo=bar >$foo do something you might expect, but unset foo foo=bar echo $foo >$foo doesn't?
Or why doesn't unset foo foo=bar echo $foo work like you'd expect while unset foo foo=bar some_other_command will put foo in the environment for the other command?
Quotes are obeyed the entire time, but are actually _removed_ after the expansion. And finally, file descriptors are opened the command is executed.
And how does this relate to ||, && and things on the right hand side of |'s in terms of evaluation order and side effects?
I don't think you can write a simple list because the actual process is too complex. It would really be a tree or flowchart.
I'm sure I saw a simple list of the order of operations for the bourne shell years ago with about 6 steps and which are repeated if you add an eval to the line. Bash handles some more complex expressions, but it must still do the steps in the same order to be compatible. You really need to know this to do anything even slightly complicated and I'm having trouble finding it again.
On Tue, Feb 26, 2008 at 11:16:27PM -0600, Les Mikesell alleged:
Yes, but I'm looking for what happens before and after. Why does unset foo foo=bar >$foo
This is interesting. I would have guessed a syntax error. I was surprised when it worked.
This is a special case because there isn't a command being executed. I can't find this syntax in the manpage but it is behaving like two different expressions because the variable is getting set in the _current_ shell:
$ unset foo $ foo=bar >$foo $ echo $foo bar
So it syntactical equivalent with: foo=bar; >$foo
In the end, I think this may be a parsing bug.
do something you might expect, but unset foo foo=bar echo $foo >$foo doesn't?
Or why doesn't unset foo foo=bar echo $foo work like you'd expect while unset foo foo=bar some_other_command will put foo in the environment for the other command?
These examples do make sense to me. They put foo in the env of the commands (meaning, inside of the child process), but the string has already been interpreted before execution. some_other_command is free to use $foo internally, but its args will be interpreted by the _current_ shell.
Looking at the second example, the first thing we see is variable replacement turning it into 'foo=bar echo'; which then executes as you'd expect.
Also, I think these examples are a bit forced because it's not how you would do it. The point of supplying the env is for the use of child processes. 'echo' is a shell-builtin. The correct way is: (foo=bar; echo $foo)
Or a less likely: foo=bar eval echo $foo
Quotes are obeyed the entire time, but are actually _removed_ after the expansion. And finally, file descriptors are opened the command is executed.
And how does this relate to ||, && and things on the right hand side of |'s in terms of evaluation order and side effects?
The boolean operators are not part of the expressions. They are looked at first to divide the string into expressions.
I don't think you can write a simple list because the actual process is too complex. It would really be a tree or flowchart.
I'm sure I saw a simple list of the order of operations for the bourne shell years ago with about 6 steps and which are repeated if you add an eval to the line. Bash handles some more complex expressions, but it must still do the steps in the same order to be compatible. You really need to know this to do anything even slightly complicated and I'm having trouble finding it again.
Well, let us know if you find it :)
On Tue, Feb 26, 2008 at 11:16:27PM -0600, Les Mikesell wrote:
Yes, but I'm looking for what happens before and after. Why does unset foo foo=bar >$foo do something you might expect, but unset foo foo=bar echo $foo >$foo doesn't?
What would you expect the last to do? "foo=bar echo $foo" only sets "foo" inside the scope of the "echo" statement, so in the scope of ">$foo" the variable is unset. In the first case there's no command so the shell is evaluating left-to-right and it works. I actually wouldn't code written that, myself! Too close to obfuscation; much easier written as two lines; foo=bar
$foo
for ease of understanding.
Or why doesn't unset foo foo=bar echo $foo work like you'd expect while
It does work like I'd expect; $foo is evaulated by the current shell to be null, and then it's set to bar for the execution of the "echo" statement. But echo doesn't use the variable, it merely prints what was passed on the command line (ie nothing)
unset foo foo=bar some_other_command will put foo in the environment for the other command?
It worked the same in both cases. Note foo=bar some_other_command $foo will do the same as the echo case above. eg
bash-3.00$ ls bash-3.00$ touch file1 file2 file3 bash-3.00$ foo=bar ls $foo file1 file2 file3
Remember, "$foo" is evaluated by the calling shell and not by the "echo" sub-process (or builtin, depending on sh variant) nor the "ls" process, and in that context it's still unset.
And how does this relate to ||, && and things on the right hand side of |'s in terms of evaluation order and side effects?
Mostly you can think of those things as causing subprocesses, so each part can be wrapped in ( ) eg foo=bar a_command | env | grep foo is close enough to ( foo=bar a_command ) | ( env ) | ( grep foo ) for evaluation purposes.
The "foo" variable is only set in the scope of "a_command" and so env never sees it.
I'm sure I saw a simple list of the order of operations for the bourne shell years ago with about 6 steps and which are repeated if you add an
I've never seen one. I'm not even sure I could write one :-)
must still do the steps in the same order to be compatible. You really need to know this to do anything even slightly complicated and I'm
Not really. Mostly if you find yourself hitting that sort of problem then you're writing overly complicated code and are probably better off refactoring it into something more readable.
I've been coding in ksh for 18 years now and haven't had to worry too much about precedence that simple test cases can't answer for me. And that included 1100 line scripts than run a messaging system :-)
Stephen Harris wrote:
Yes, but I'm looking for what happens before and after. Why does unset foo foo=bar >$foo do something you might expect, but unset foo foo=bar echo $foo >$foo doesn't?
What would you expect the last to do? "foo=bar echo $foo" only sets "foo" inside the scope of the "echo" statement, so in the scope of ">$foo" the variable is unset. In the first case there's no command so the shell is evaluating left-to-right and it works.
How does the shell know that there's no command without evaluating, and if it evaluates left to right, why isn't the result the same? Actually I found the answer to that, which is that name=val command is processed like (name=val; command) which also explains why the value isn't left lingering for the next operation.
I actually wouldn't code written that, myself! Too close to obfuscation; much easier written as two lines; foo=bar
$foo
for ease of understanding.
I find it much easier to understand code written to keep setting/using values in the smallest scope possible.
And how does this relate to ||, && and things on the right hand side of |'s in terms of evaluation order and side effects?
Mostly you can think of those things as causing subprocesses, so each part can be wrapped in ( )
Which would be more meaningful in a document that explains when and how the () tokens are handled...
I'm sure I saw a simple list of the order of operations for the bourne shell years ago with about 6 steps and which are repeated if you add an
I've never seen one. I'm not even sure I could write one :-)
Section 7.8 of this might be close to what I remember seeing. http://books.google.com/books?id=_mbgnxL-QvoC&pg=PA140&lpg=PA140&...
must still do the steps in the same order to be compatible. You really need to know this to do anything even slightly complicated and I'm
Not really. Mostly if you find yourself hitting that sort of problem then you're writing overly complicated code and are probably better off refactoring it into something more readable.
But I find compact code most readable.
I've been coding in ksh for 18 years now and haven't had to worry too much about precedence that simple test cases can't answer for me.
I don't trust test cases on one implementation to be portable unless they match the documented behavior.
And that included 1100 line scripts than run a messaging system :-)
Maybe it would only have taken a couple hundred lines if you took advantage of order of operations on each line... But my rule of thumb is that if I expect something to fill more than a page, I'd start in some other language.
On Wed, Feb 27, 2008 at 11:46:13AM -0600, Les Mikesell alleged:
Stephen Harris wrote:
Yes, but I'm looking for what happens before and after. Why does unset foo foo=bar >$foo do something you might expect, but unset foo foo=bar echo $foo >$foo doesn't?
What would you expect the last to do? "foo=bar echo $foo" only sets "foo" inside the scope of the "echo" statement, so in the scope of ">$foo" the variable is unset. In the first case there's no command so the shell is evaluating left-to-right and it works.
How does the shell know that there's no command without evaluating, and if it evaluates left to right, why isn't the result the same? Actually I found the answer to that, which is that name=val command is processed like (name=val; command) which also explains why the value isn't left lingering for the next operation.
If was like '(name=val; command)' then 'foo=bar echo $foo' would do what you want.
'name=val' is very different from 'name=val command'. They aren't parsed the same way and follow different rules. The first is a variable assignment. The second is a command execution with a supplied env list.
'name=val command' is not "evaluated from left-to-right". After all variable substitution, expansions, and word splitting is done, then the command is executed with the supplied env var.
'name=val >$name' is a weird case that I can't explain. It is not related to 'name=val command' because there is no command. It acts like two statements when syntacticaly it is one. It think it is a bug. Avoid it.
On Wed, Feb 27, 2008 at 11:13 AM, Garrick Staples garrick@usc.edu wrote:
'name=val >$name' is a weird case that I can't explain. It is not related to 'name=val command' because there is no command. It acts like two statements when syntacticaly it is one. It think it is a bug. Avoid it.
The tidbit that no one has pointed out yet is that redirections are not actually part of the command in the sense that you might think they are.
These are all the same:
$ >foo echo bar $ echo >foo bar $ echo bar >foo
The command is "echo bar" no matter where you put the redirection. Hence this:
$ foo=bar
and this:
$ foo=bar >$foo
are exactly the same in so far as the execution of "foo=bar" is concerned; and "foo=bar" all by itself is the special construct that means "assign foo in the current shell", so that's what happens. On the other hand
$ foo=bar echo
is syntax that means (as has previously been explained) "assign foo in the process environment of echo" so nothing happens in the current shell.
The other tidbit (see the book link Les posted) is that assignments happen very early and redirections happen very late, in the order of evaluation.
If you really want to warp your brain, try this one (not in your only login shell, though):
$ 2>$foo foo=yourOutputIsHere exec
And stop thinking from left to right.
On Tue, Feb 26, 2008 at 04:33:30PM -0600, Les Mikesell wrote:
Benjamin Smith wrote:
Unless I'm terribly mistaken (again?), the only way I've been able to
see "loop thru a list of files" work reliably is with "find" using the "-print0" option, in cahoots with xargs.
Is there any other way?
for $file in wildcard*
No $ needed there!!!
Stephen Harris wrote:
On Tue, Feb 26, 2008 at 04:33:30PM -0600, Les Mikesell wrote:
Benjamin Smith wrote:
Unless I'm terribly mistaken (again?), the only way I've been able to
see "loop thru a list of files" work reliably is with "find" using the "-print0" option, in cahoots with xargs.
Is there any other way?
for $file in wildcard*
No $ needed there!!!
Good catch - I must have been thinking partly in perl.
On Tuesday 26 February 2008, Les Mikesell wrote:
Benjamin Smith wrote:
Unless I'm terribly mistaken (again?), the only way I've been able to
see "loop thru a list of files" work reliably is with "find" using the "-print0" option, in cahoots with xargs.
Is there any other way?
for $file in wildcard* do ls -l "$file" done
But this is the point where you should be asking what to do about quotes embedded in the filenames which won't hurt here because of the order of operations but would if you tried to collect the strings and use them in some other ways.
Exactly. Here's my example:
$ ls -laFd * -rw-r--r-- 1 bens nobody 5 2008-02-26 12:21 Disney\ trip\ -a\ mother\'s\ journey.doc -rwxr--r-- 1 bens nobody 103 2008-02-26 13:35 script1.sh* -rwxr--r-- 1 bens nobody 26 2008-02-26 11:54 script2.sh* -rw-r--r-- 1 bens nobody 52 2008-02-26 15:15 script3.sh -rw-r--r-- 1 bens nobody 55 2008-02-26 13:17 t
Note that, even here, there's a file called "-b" that's been inadvertently hidden!
$ ls -laFd -- * -b Disney trip -a mother's journey.doc script1.sh script2.sh script3.sh t $ cat -- -b blah $
File script3.sh contains the following: $ cat script3.sh #! /bin/sh for file in $* do ls -l "$file"; done [bens@turing tt]$
And when I run script3.sh, I get: $ /bin/bash ./script3.sh * total 48 -rw-r--r-- 1 bens nobody 5 2008-02-26 12:14 -b -rw-r--r-- 1 bens nobody 5 2008-02-26 12:21 Disney\ trip\ -a\ mother\'s\ journey.doc -rwxr--r-- 1 bens nobody 103 2008-02-26 13:35 script1.sh -rwxr--r-- 1 bens nobody 26 2008-02-26 11:54 script2.sh -rw-r--r-- 1 bens nobody 52 2008-02-26 15:18 script3.sh -rw-r--r-- 1 bens nobody 55 2008-02-26 13:17 t ls: cannot access Disney: No such file or directory ls: cannot access trip: No such file or directory total 64 drwxr-xr-x 2 bens nobody 4096 2008-02-26 13:17 . drwxr-xr-x 14 bens bens 4096 2008-02-26 11:54 .. -rw-r--r-- 1 bens nobody 5 2008-02-26 12:14 -b -rw-r--r-- 1 bens nobody 5 2008-02-26 12:21 Disney trip -a mother's journey.doc -rwxr--r-- 1 bens nobody 103 2008-02-26 13:35 script1.sh -rwxr--r-- 1 bens nobody 26 2008-02-26 11:54 script2.sh -rw-r--r-- 1 bens nobody 52 2008-02-26 15:18 script3.sh -rw-r--r-- 1 bens nobody 55 2008-02-26 13:17 t ls: cannot access mother's: No such file or directory ls: cannot access journey.doc: No such file or directory -rwxr--r-- 1 bens nobody 103 2008-02-26 13:35 script1.sh -rwxr--r-- 1 bens nobody 26 2008-02-26 11:54 script2.sh -rw-r--r-- 1 bens nobody 52 2008-02-26 15:18 script3.sh -rw-r--r-- 1 bens nobody 55 2008-02-26 13:17 t $
It's obviously getting slipped on on the "-b". Tried again: $ cat script3.sh #! /bin/bash for file in $* do ls -l -- "$file"; done $ /bin/bash ./script3.sh * -rw-r--r-- 1 bens nobody 5 2008-02-26 12:14 -b ls: cannot access Disney: No such file or directory ls: cannot access trip: No such file or directory ls: cannot access -a: No such file or directory ls: cannot access mother's: No such file or directory ls: cannot access journey.doc: No such file or directory -rwxr--r-- 1 bens nobody 103 2008-02-26 13:35 script1.sh -rwxr--r-- 1 bens nobody 26 2008-02-26 11:54 script2.sh -rw-r--r-- 1 bens nobody 57 2008-02-26 15:21 script3.sh -rw-r--r-- 1 bens nobody 55 2008-02-26 13:17 t
Still has bad errors, properly quoted, otherwise legal file names. Redefine IFS?
Does anyone have a quick reference to the order of operations as the shell parses a command line (variable parsing,i/o redirection, wildcard and variable expansion, splitting on IFS, quote removal, command substitution etc.)? That's really the first thing you need to know about the shell and if there is a simple description it must be buried in the middle of some obscure manual.
-- Les Mikesell lesmikesell@gmail.com
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
-- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
On Tue, Feb 26, 2008 at 03:30:02PM -0800, Benjamin Smith wrote:
Exactly. Here's my example:
$ ls -laFd *
You're doing it wrong: ls -laFD -- *
ls -l "$file";
You're doing it wrong: ls -l -- "$file"
$ /bin/bash ./script3.sh *
You're doing it wrong: bash ./script3.sh "*"
(I already addressed why that is in an earlier message; you're doing two levels of shell parsing by calling the script in that odd way)
Benjamin Smith wrote:
It's obviously getting slipped on on the "-b". Tried again: $ cat script3.sh #! /bin/bash for file in $* do ls -l -- "$file"; done $ /bin/bash ./script3.sh * -rw-r--r-- 1 bens nobody 5 2008-02-26 12:14 -b ls: cannot access Disney: No such file or directory ls: cannot access trip: No such file or directory ls: cannot access -a: No such file or directory ls: cannot access mother's: No such file or directory ls: cannot access journey.doc: No such file or directory -rwxr--r-- 1 bens nobody 103 2008-02-26 13:35 script1.sh -rwxr--r-- 1 bens nobody 26 2008-02-26 11:54 script2.sh -rw-r--r-- 1 bens nobody 57 2008-02-26 15:21 script3.sh -rw-r--r-- 1 bens nobody 55 2008-02-26 13:17 t
Still has bad errors, properly quoted, otherwise legal file names. Redefine IFS?
Still not properly quoted. What you need in the "for ..." line is the syntax that quotes each individual argument (so that embedded white space doesn't get treated as argument delimiters) while still maintaining "$1" "$2" "$3" etc. as separate arguments. That's what "$@" does:
for file in "$@" do ls -l -- "$file" done
Here's a little script that I have to play around with positional parameters. I'm pretty certain I did not author this one but got it either off the web or ina book. I added a line of comment in it but I don't believe I made any other contributions to it.
Jacques B.
#!/bin/bash # arglist.sh # Invoke this script with several arguments, such as # ./scriptname one "two three" four "five;six" seven eight 'nine ten'
E_BADARGS=65
if [ ! -n "$1" ] then echo "Usage: `basename $0` argument1 argument2 etc." exit $E_BADARGS fi
echo
index=1 # Initialize count.
echo "Listing args with "$*":" for arg in "$*" # Doesn't work properly if "$*" isn't quoted. do echo "Arg #$index = $arg" let "index+=1" done # $* sees all arguments as single word. echo "Entire arg list seen as single word."
echo
index=1 # Reset count. # What happens if you forget to do this?
echo "Listing args with "$@":" for arg in "$@" do echo "Arg #$index = $arg" let "index+=1" done # $@ sees arguments as separate words. echo "Arg list seen as separate words."
echo
index=1 # Reset count.
echo "Listing args with $* (unquoted):" for arg in $* do echo "Arg #$index = $arg" let "index+=1" done # Unquoted $* sees arguments as separate words. echo "Arg list seen as separate words."
exit 0
On Tue, Feb 26, 2008 at 01:51:28PM -0800, Benjamin Smith wrote:
Unless I'm terribly mistaken (again?), the only way I've been able to see "loop thru a list of files" work reliably is with "find" using the "-print0" option, in cahoots with xargs.
What is it you're trying to do? You typically only need to use "find" if you want to recursively descend a directory tree. Otherwise simple globs will do the job
for a in * do blah "$a" done
On Tue, Feb 26, 2008 at 12:40:06PM -0800, Benjamin Smith wrote:
In script2.sh, $1 only contains the string "this". There is no safe way to pass $1 (containing string "this parameter") from script1 to script2 as a single, trustable parameter.
The statement is meaningless. Trusted in WHAT CONTEXT? In an eval? Passed to a SQL command? Or merely passed as a parameter to another program? If the last, then USE QUOTES.
So $1 in script 1 contains "this parameter". $1 in script 2 contains "this".
Because you are doing it wrong. You want to pass "$1" which tells the shell to pass the contents of $1 WITHOUT parsing it. You're passing $1 which tells the shell to break the arguments up at whitespace (well, $IFS) when passing to an external program.
Instead, I have to hork it up with awk, sed, or something similar, and try to account for every possible interpreted character.
No you don't. You need to learn how to quote properly.
#!/bin/bash PassToShell2=`escapethis $1`;
You're missing "" around the $1 again
Again, you're missing the point. (practice makes perfect?)
You've missed every single clue thrown at you, so far. You are MISUSING the tool, and blaming it for your mistakes.
is being interpreted, it doesn't change the fact that $file is unescaped.
You're missing "" again.
mv -- "$file" wheee
for file in $* do mv ${file} $prefix$file
mv -- "$file" "$prefix$file"
echo blah > "Disney trip -a mother's journey.doc";
$ ls Disney trip -a mother's journey.doc $ for a in *
do mv -- "$a" "whee$a" done
$ ls wheeDisney trip -a mother's journey.doc
Clever that, innit?
for file in $* do echo "$file"; done;
called like: /bin/bash ../test.sh *
You're doing it wrong. You've now got TWO SHELLS expanding the command line. bash ../test.sh "*"
Again, you don't understand Unix shell quoting.
I mean, argue with me if you want on how my scripts are implemented but the previous two (TRUE) sentences sound like a philosophical deficiency to me.
Only because YOU ARE DOING IT WRONG.
Every single case has been fixed by proper use of "" around the variable expansion.
You are at fault, not the shell.
(Now there _are_ odd corner cases, but you haven't hit one, yet!)
Are you trying to pass all parameters from one script to another or just the first one ($1). If it's the former, have you tried using "$@"? For the latter, "$1" might work.
Alfred
On Tue, Feb 26, 2008 at 11:22:55AM -0800, Benjamin Smith wrote:
file: script1.sh #! /bin/bash script2.sh $1
There's your mistake. It should be script2.sh "$1" Otherwise $1 is evaluated and passed through as potentially multiple parameters to script2.sh
For example: $ cat x #!/bin/sh ./y "$1"
$ cat y #!/bin/sh echo "$1"
$ ./x "hello\ there" hello\ there
The problem isn't the shell doing bad things, it's you not understanding how shell variable expansion is done when calling external commands.
Benjamin Smith wrote:
On Tuesday 26 February 2008, Les Mikesell wrote:
WHY THE @!#! NOT?!?!?
The shell is 'supposed' to be run by a user that is allowed to run any command he wants, and permission/trust issues are handled by the login/authentication process that happens before you get to the shell. If you give the shell a bad command under your own account, it's not supposed to second guess what you wanted.
I'm not asking for this. I'm only asking for the option to be able to trust that a parameter is... a parameter. EG:
You can, but you have to know how many times shells will parse it. One layer of quoting is removed each time. Things inside single quotes are literal, double quotes still do variable expansion.
file: script1.sh #! /bin/bash script2.sh $1 exit 0;
file: script2.sh #! /bin/bash echo $1;
$ script1.sh "this\ parameter";
I get output of "this"! script2 gets two parameters! I want a way for 1 parameter to STAY 1 parameter upon request, so that script2.sh would output "this parameter", like
One layer of quotes for each time it is parsed... The first is on the initial command line, so if you want to hold it together, put double quotes around "$1".
file:script1.sh #! /bin/bash PassToShell2=escapethis $1; script2.sh $PassToShell; exit 0;
Bash is used, extensively in many cases, to deal with untrusted data.
Why?
How about an installer script? How about a magical script copied from TLDP to rename all files in pwd?
These things run with the permissions of the user running them. Why should they be concerned about that person giving them untrusted embedded commands that they could just as easily run directly?
This can include random file names in user home directories, parameters on various scripts, etc. It's highly sensitive to being passed characters that have, over the past NN years, resulted in quite a number of security holes and problems.
If it hurts, don't do it. Build your own argument list and exec programs directly if you want to avoid shell command line parsing.
So, I'm supposed to know the contents of a user's home directory? And code for these in advance?
Code so you don't let the shell parse their names. It doesn't have to if you just want to hand them to some other program.
Yet there exists NO MECHANISM for simply ensuring that a given argument is
an
escaped string?
What does that mean? If you can define it you can make it happen, but who knows what characters at what depth of quoting will have some special meaning?
Can I define it? Thought I did that already: http://us.php.net/manual/en/function.escapeshellarg.php
Can you pass that via ssh to some other system(s) and have i/o redirection or variable expansions happen in the right places?
Or its perl equivalent: http://search.cpan.org/~gaas/URI-1.35/URI/Escape.pm
See how I'd like to see it in implementation in above example, "passToShell2"
What's the point? If you are in the shell already it's too late. If you are in some other program and don't want shell metacharacter processing to happen, don't feed it to the shell in the first place.
How many "homebrew" ISP or hosting administration scripts could be
compromised
by simply putting a file in your home directory called ";rm -rf /" ?
Probably none that are still in business.
Google "bash howto" for lots of vulnerable and problematic examples. Here's a beaut that fails if you have a file called "-a" in the pwd, see "File re-namer". It's a renamer that doesn't, if the file contains any spaces, dashes, etc.
There are any number of ways to do things wrong. I don't need to look up more of them.
http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-12.html#ss12.1
Here's what I get:
mv: invalid option -- a Try `mv --help' for more information.
Most programs accept -- as an option to end option parsing. But if you don't want this effect, don't start your filenames with a -.
Or with a file with a space: echo "blah" > "d"; echo "blah" > "d foo";
The TLDP's example doesn't move file "d foo". I get: mv: cannot stat `d': No such file or directory mv: cannot stat `foo': No such file or directory
If you don't want the shell to split on spaces you can tell it not to. Or you can quote filenames correctly. Or not put spaces in filenames.
So I ask again: This doesn't strike you as fundamentally borkeD? The emperor wears no clothes!
No, it works the way I'd want it to work most of the time. And for the exceptions, you use quotes in the right places the right number of times.
This doesn't strike you as fundamentally borkeD?
No, if you stop bad things from happening, you'll also stop good things.
Yes. But you don't have to stop the good things.
You don't like the fact that the shell does things like splitting on white space?
I think the *OPTION* of saying "parameter 1 is STILL parameter 1" is a good thing. If you want to leave things be, so be it. See my above example.
You can't do that without knowing at what point you want it to stop being an opaque blob.
The problem is that you aren't using the shell as intended. If you run it under your own user id, it does exactly what you tell it to do and there is no element of trust involved.
The problem, as I see it, is that the shell provides access variables without any means of preserving them as variables across calls and incantations.
It does. You just have to know when it should and when it shouldn't
Errr what??? Php has about the worst security history of any program around.
Thanks for confusing the issue with a red herring. Or should I ignore the buggy and probably vulnerable TLDP example above? Maybe a google search for "bash escape vulnerability" might illuminate the issue I speak of?
No, you should just learn to use quotes. Or bypass shell processing.
This just blows my mind....
What would you like your computer to prevent you from doing to yourself?
I hate to belabor it: give me the OPTION to trust that I can keep a single parameter as a single parameter across incantations and calls. If I'm looping thru a listing files, I should be able to trust that my $FILENAME variable contains the name of... a file!
A file can be be named pretty much anything, so what does that mean? All you can do is make sure it is quoted as many times as you let the shell parse it.
If I want to pass parameter 1 of my script to another script, that other script should be ABLE get my parameter 1 as... parameter 1!
OK - quote it.
A simple call that lets me take a string and get a passable parameter would suffice.
But in shell, such a call would parse whatever you give it - and the parsing would follow the same rules as every other parsing operation...
You want to apply scripts that break with a space in a file name? Great. Have at it. I don't want to detract from your ability to make an alphabet soup of your directories with borken examples from TLDP, and all the goodness that implies to you.
But I think admins the world over would like it if a simple, one-liner could be added to ensure that a single parameter remains a single parameter!
Am I being unclear?
You want the shell to not be the shell, I think. Everything the shell does, it does the same way, which is why we like it in the first place. You can't tell it to do something without parsing the command and its arguments.
Benjamin Smith wrote:
On Tuesday 26 February 2008, Bob Beers wrote:
short answer: single quotes will handle all characters,
except single quotes.
long answer: man bash the section called QUOTING may help you figure a solution.
I've read the man page. It helps if I already know the input
- I don't have a
problem with manually putting slashes in front of spaces and single quotes. But in this case, I don't know the input. It's untrusted data.
There is no mechanism for escaping untrusted input?
You could try uuencode/uudecode and handling the uuencoded strings.
-Ross
______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof.
-
Alfred von Campe -
Bart Schaefer -
Benjamin Smith -
Bob Beers -
Garrick Staples -
Jacques B. -
Les Mikesell -
Milton Calnek -
Ralph Angenendt -
Robert Nichols -
Ross S. W. Walker -
Stephen Harris