The shell interpreter is far from being a programming language.
Just look at how weird it is, and it will be clear that it
was _evolution_, not design.

Shell scripting is ugly. My rule of thumb is: you should quit
(and switch to Perl, or Python) as soon as you realize you need a
hash...

Shell scripting is powerful! And it is defitely the common
ground of any respectable UNIX system, from the first boot
after a fresh installation!

Shell scripting is popular! Regardless of how hard it is.
My day job often offers me the opportunity of improving my
skills. At the same time I realized that many people
tend to ignore the details of this complicated art, ending
up to fragile and unmaintainable tools. Too bad!

I started to collect some best practices, things that I've
figured out over time. My collection is not complete, nor properly
organized, but all projects need to start somewhere.

I've published them here for the moment:

gopher://tilde.institute/0/%7edacav/txt/sh.txt

Any feedback and contribution is well accepted!


Cheers
- dacav
--- Synchronet 3.18b-Linux NewsLink 1.113

gopher://tilde.institute/0/~dacav/txt/sh.txt from 2021-01-19:

Alternative method:

ssh_pipe() {
ssh -T -l "${1:?user}" "${2:?host}" sh -xe
}

ssh_pipe bob 192.168.1.1 <<EOF
... # here goes a shell script!
EOF

I think there should also be a warning that anything called in the
remote shell script which reads from standard input would *not* work.

gopher://tilde.institute/0/~dacav/txt/sh.txt from 2021-01-19:

$ cat <<'EOF'
$USER
EOF

Okay, didn't realize that there is a quoted-literal version
of here-document too; thanks for mentioning it.

gopher://tilde.institute/0/~dacav/txt/sh.txt from 2021-01-19:

{ cat <<END_OF_VARS; cat <<'END_OF_SCRIPT'; } | ssh_pipe bob 192.168.1.1
remote_var="$local_var"
END_OF_VARS
echo "I can use $remote_var"
END_OF_SCRIPT

Haha, haven't personally used multiple here-documents in single line yet;
POSIX specs explicitly mentioned it too, will have to use it sometimes.

Also, today I learned that POSIX also specified `<<- SOMETHING` too,
which is a kill-all-leading-TABs kind of here document- might be useful
for people who wish to indent stuff in their scripts without interfering
with the result. (Like, when using here-document inside one or more levels
of loops or conditions)

For example:

if true; then
cat <<- THE_END
This
is
not
supposed
to
be
indented.
THE_END
fi

Would give an equivalent output to:

if true; then
cat << THE_END
This
is
not
supposed
to
be
indented.
THE_END
fi

Cheers,
~xwindows
--- Synchronet 3.18b-Linux NewsLink 1.113

On 2021-01-20, xwindows <xwindows@tilde.club> wrote:

gopher://tilde.institute/0/~dacav/txt/sh.txt from 2021-01-19:

Alternative method:

ssh_pipe() {
ssh -T -l "${1:?user}" "${2:?host}" sh -xe
}

ssh_pipe bob 192.168.1.1 <<EOF
... # here goes a shell script!
EOF

I think there should also be a warning that anything called in the
remote shell script which reads from standard input would *not* work.

Aha! I never had the need so far, but you're right: this remote grap won't work:

ssh_pipe <<-'EOF'
grep f.o
EOF

…clearly because stdin is already busy. Definitely a blind spot to mention, thanks!

gopher://tilde.institute/0/~dacav/txt/sh.txt from 2021-01-19:

$ cat <<'EOF'
$USER
EOF

Okay, didn't realize that there is a quoted-literal version
of here-document too; thanks for mentioning it.

I discovered it accidentally while I was still wearing my Perl hat!
I was obviously very happy with my finding. :D

gopher://tilde.institute/0/~dacav/txt/sh.txt from 2021-01-19:

{ cat <<END_OF_VARS; cat <<'END_OF_SCRIPT'; } | ssh_pipe bob 192.168.1.1
remote_var="$local_var"
END_OF_VARS
echo "I can use $remote_var"
END_OF_SCRIPT

Haha, haven't personally used multiple here-documents in single line yet; POSIX specs explicitly mentioned it too, will have to use it sometimes.

Also, today I learned that POSIX also specified `<<- SOMETHING` too,
which is a kill-all-leading-TABs kind of here document- might be useful
for people who wish to indent stuff in their scripts without interfering
with the result. (Like, when using here-document inside one or more levels
of loops or conditions)

Yes! It is great for readability of scripts. And by the way it is the primary reason why I would advocate for the use of TAB in indentation: the `<<-X` heredoc only works with TAB.

It's hard to draw the line between things that should enter this "best practices" guide and things that should not. Unfortunately the detection of what is well known and what is not is very subjective

For instance, would you add the ${something:+--option "$something"} pattern?


Thanks for your feedback, xwindows!
--- Synchronet 3.18b-Linux NewsLink 1.113

Dacav Doe <dacav@tilde.institute> wrote:

The shell interpreter is far from being a programming language.
Just look at how weird it is, and it will be clear that it
was _evolution_, not design.

Although that's kind-of what I like about it (talking about Bash
specifically), all of the odd details and alternative ways of doing
things make it fun and, strangely, memorable to me. That said, the
only other scripting language that I've tried to learn is PHP
(because extensive web programming doesn't look fun in Bash or C,
though I know there are some occasional people who claim otherwise).

Just so long as all the different ways of doing things do actually
work (as in no bugs if you do things the "wrong" way). Any system
where there's three ways to do everything, but only one that works,
is hell.

My day job often offers me the opportunity of improving my
skills. At the same time I realized that many people
tend to ignore the details of this complicated art, ending
up to fragile and unmaintainable tools. Too bad!

I started to collect some best practices, things that I've
figured out over time. My collection is not complete, nor properly organized, but all projects need to start somewhere.

I've published them here for the moment:

gopher://tilde.institute/0/%7edacav/txt/sh.txt

Any feedback and contribution is well accepted!

Thanks for posting them, some handy tips. While I admire POSIX
scripting, I'm a bit too lazy myself and just stick with Bash.
Practically there's not many situations where Bash isn't an easy
and fairly small install, and it's usually already there anyway.
I try to avoid "young" features for the sake of compatibility
(and likelihood of future change) unless they're just irresistable.

Feedback:

I don't really see the point of using getops to be honest, compared
to positional parameters. I know it's a common practice, but
personally I don't really see any advantage, unless there's a
limitation in POSIX that I don't know about.

"Perl style" booleans seem a little confusing for someone who always
thinks of FALSE as zero.

The ssh_pipe stuff is neat. Probably not useful for anything I do
myself, but neat.

--

- The Free Thinker | gopher://aussies.space/1/%7efreet/
--- Synchronet 3.18b-Linux NewsLink 1.113

On 2021-01-23, The Free Thinker <freet@aussies.space> wrote:

Thanks for posting them, some handy tips. While I admire POSIX
scripting, I'm a bit too lazy myself and just stick with Bash.

There are indeed some features that I do miss in POSIX scripting,
that is pipefail (of course!) and <(subshells as files). And to
add a third one: arrays. Even though given the choice I would
give up arrays my whole life in exchange for a portable pipefail!

Practically there's not many situations where Bash isn't an easy
and fairly small install, and it's usually already there anyway.

I'm now thinking of a pristine FreeBSD system, where you want to
keep it barebones.

And I'll also think of a busybox-based environment, inside a
small embedded system :)

I love this shit :D

Feedback:

I don't really see the point of using getops to be honest, compared
to positional parameters. I know it's a common practice, but
personally I don't really see any advantage, unless there's a
limitation in POSIX that I don't know about.

This is in fact something I use only when exposing the functions
to the user, which boils down to the use-case of a script you
"source".

In such situations, your functions take the place of commands,
and that's where getopts comes in handy.

"Perl style" booleans seem a little confusing for someone who always
thinks of FALSE as zero.

True. But after a moment of disorientation it feels to me very
natural to do

if [ "$mybool" ]; then ...

Instead of

if [ "$mybool" = TRUE ]; then ...

But hey, …opinions :)

Oh, now that I think of it, it's not even 100% Perl-ish!
Perl would for instance evaluate as false a non-empty scalar
whose value is 0.

The ssh_pipe stuff is neat. Probably not useful for anything I do
myself, but neat.

One of those "good to know", sort of like "if you ever meet a
bear, play dead — so it can eat you without wasting energies in
hunting!"

Thanks for your feedback! :)
--- Synchronet 3.18b-Linux NewsLink 1.113

On 2021-01-23, The Free Thinker <freet@aussies.space> wrote:

Thanks for posting them, some handy tips. While I admire POSIX
scripting, I'm a bit too lazy myself and just stick with Bash.
Practically there's not many situations where Bash isn't an easy
and fairly small install, and it's usually already there anyway.
I try to avoid "young" features for the sake of compatibility
(and likelihood of future change) unless they're just irresistable.

For myself it's a bit of a challenge. If I'm tossing together a quick
script to automate something I'll run for myself I may use Bash, but
anything I intend to share deserves that little extra effort to make it
as portable as I can. I do appreciate people that share snippets like
these to address common challenges. Such fun!
--- Synchronet 3.18b-Linux NewsLink 1.113

James Tomasino <tomasino@cosmic.voyage> writes:

On 2021-01-23, The Free Thinker <freet@aussies.space> wrote:
If I'm tossing together a quick script to automate something I'll run
for myself I may use Bash, but anything I intend to share deserves
that little extra effort to make it as portable as I can.

Shouldn't AWK be near to omnipresent? I started to use it a bit more
when I noticed it being in OpenWrt's busybox, so even on a router
without LuCI (the WebGUI which pulls in Lua as dependency) you still
have SH and AWK and the later one sure is much more readable.

I only have some badly commented examples because on these small systems
one typically don't waste space:

https://openwrt.org/user/yeti/ncomm
https://openwrt.org/user/yeti/mirror-opkg-repositories
https://openwrt.org/user/yeti/opkg-list-user-installed-packages

I think AWK is massively underrated these days.
--- Synchronet 3.18b-Linux NewsLink 1.113

yeti <yeti@tilde.institute> wrote:

James Tomasino <tomasino@cosmic.voyage> writes:

If I'm tossing together a quick script to automate something I'll run
for myself I may use Bash, but anything I intend to share deserves
that little extra effort to make it as portable as I can.

Shouldnt AWK be near to omnipresent? I started to use it a bit more
when I noticed it being in OpenWrt's busybox, so even on a router
without LuCI (the WebGUI which pulls in Lua as dependency) you still
have SH and AWK and the later one sure is much more readable.

I only have some badly commented examples because on these small systems
one typically don't waste space:

https://openwrt.org/user/yeti/ncomm
https://openwrt.org/user/yeti/mirror-opkg-repositories
https://openwrt.org/user/yeti/opkg-list-user-installed-packages

Thanks for posting, those might come in handy as a fellow OpenWrt
user (though one who usually installs the Bash package).

I think AWK is massively underrated these days.

May be, I've never looked deep into it myself.

--

- The Free Thinker | gopher://aussies.space/1/%7efreet/
--- Synchronet 3.18b-Linux NewsLink 1.113

If done in a less compact style than my examples, AWK would be even much
more readable. Maybe like a BASIC with functions, subroutines, and long variable names.

Maybe AWK is the language with the fastest payback of invested time.
Probably after the first 2 hours of reading about it, you can already do
far more then hello-world like textbook examples.
--- Synchronet 3.18b-Linux NewsLink 1.113