Gregg's MOTD

Removing Blank Spaces from a Group of File Names

This set of commands iterate over each file in the current directory and will replace any blank spaces in a filename with an underscore (_).

First, we’ll create a set of sample files:

$ for i in {0..9}; do touch "file-${i} (number-${i}).txt"; done
$ ls
file-0\ (number-0).txt file-4\ (number-4).txt file-8\ (number-8).txt
file-1\ (number-1).txt file-5\ (number-5).txt file-9\ (number-9).txt
file-2\ (number-2).txt file-6\ (number-6).txt
file-3\ (number-3).txt file-7\ (number-7).txt

Now, we’ll execute the set of commands:

$ for f in *\ *; do mv "$f" "${f// /_}"; done

And now you can see that the blanks have been replaced.

$ ls
file-0_(number-0).txt file-4_(number-4).txt file-8_(number-8).txt
file-1_(number-1).txt file-5_(number-5).txt file-9_(number-9).txt
file-2_(number-2).txt file-6_(number-6).txt
file-3_(number-3).txt file-7_(number-7).txt

Tags: cli, bash, rename, motd

Remove Parentheses from File Names

This script will rename Windows backup files by removing the ’ (date)’ from the filename. It will remove the parentheses and everything between them from the file name, so that a file with the name file-1 (2023-09-10).txt will be renamed to file-1.txt. Note that since the data between the parentheses will be removed, the resulting file name must be unique for this script to prevent data from being overwritten. Other than that it should work for any fileset that meets the beforementioned constraints.

First, let’s create some sample files:

$ for i in {0..9}; do touch "file-${i} (number-${i}).txt"; done
$ ls
file-0\ (number-0).txt file-4\ (number-4).txt file-8\ (number-8).txt
file-1\ (number-1).txt file-5\ (number-5).txt file-9\ (number-9).txt
file-2\ (number-2).txt file-6\ (number-6).txt
file-3\ (number-3).txt file-7\ (number-7).txt

Now, we’ll execute the script on the sample files:

$ for f in *; do n=$(echo $f | sed "s/[ (][^)]*[)]//g");mv "${f}" "${n}"; done

If we look at the directory listing now we see that the file names are updated:

$ ls
file-0.txt file-2.txt file-4.txt file-6.txt file-8.txt
file-1.txt file-3.txt file-5.txt file-7.txt file-9.txt

Tags: cli, bash, rename, motd

Processing a List of Files from an Input File

Suppose we have a list of files stored in a text file and we want to perform an operation on each of them within a bash script. How would we go about doing that? Well, there are several options, here are a few.

$ cat input-files.txt
file-1.txt
file-2.txt
file-3.txt
file-4.txt
file-5.txt
file-6.txt
file-7.txt
file-8.txt
file-9.txt

Now, if we want to operate on each file in this list we can do something like this:

while IFS= read -r filename
do
echo "Do something on ${filename} here..."
done < "input-files.txt"

or alternatively,

input="input-files.txt"
while IFS= read -r filename
do
printf '%s\n' "${filename}"
done < "${input}"

Tags: cli, bash, shell-scripting, motd

String Processing with Bash

There are various tools built into bash that enable you to manipulate a variable or string which come in handy when writing shell scripts. Here are a few notable ones:

Find the length of a string

${#string}

Get a Substring from a String

${string:pos} or ${string:pos:len}

Removing Substrings from a String

${string#substr}
${string%substr}
${string##substr}
${string%%substr}

Some examples

Here’s a few examples of how you can process a variable that points to an absolute path of a file and shows how to extract certain parts of said file’s path:

var=/home/user/code/blogstuff/index.html.j2
echo ${var} # => /home/user/code/blogstuff/index.html.j2
echo ${var#*.} # => html.j2
echo ${var##*.} # => j2
echo ${var%/*.*} # => /home/user/code/blogstuff

Replace a Substring of a String

${string/pattern/substr} # => Replaces the 1st match found
${string//pattern/substr} # => Replaces all of the matches found

Replace the Beginning or End of a String

${string/#pattern/substr}
${string/%pattern/substr}
file=${var##/*/} # => index.html.j2
echo ${file/#index/fubar} # => fubar.html.j2
echo ${file/%j2/fubar} # => index.html.fubar

Tags: cli, bash, shell-scripting, motd

Test If a Port is Open with Bash

If netcat isn’t available on your machine and you don’t have the priviledge to install it you can use this trick to test if a port is open or not. It will throw a connection refused message if a port is closed.

$ : </dev/tcp/127.0.0.1/80

And you can use it in a script like this:

(: </dev/tcp/127.0.0.1/80) &>/dev/null && echo "OPEN" || echo "CLOSED"

Tags: cli, networking, bash, motd

Command Line Redirection

Redirection is very significant in shell scripting. It provides you a means to save the output of a command to a file or multiple files (one for stdout and one for stderr).

Below is a table of simple redirections that are the most useful in shell scripting. Here we are using the following naming conventions:

• stdout – The output of the script/command
• stderr – The errors generated by the script/command
• outfile – A target filename where you wish to store the output
• errfile – A target filename where you wish to store the errors

Command	Description/Purpose
command 2>errfile	Redirect stderr to errfile
command >outfile 2>errfile	Redirect stderr to file named errfile and stdout to file named outfile
command &> outfile	Redirect stderr and stdout to outfile
command 2>&-	Just suppress error messages. No file created. No error message displayed on screen
command 2>&1	Redirect error messages to standard output. Useful in shell script when you need to forcefully display error messages on screen

Tags: cli, bash, scripting, redirection, motd

Printing Numbers using Thousand Separators

You can use a pipe to awk to output numbers with thousands separators (commas). For Example, here’s how you can total the 5th column of the ls -l command and print it with thousands separators:

$ ls -l | awk '{total = total + $5}END{print total}' | LC_ALL=en_US.UTF-8 awk '{printf("%'"'"'d\n", $0) }'
21,387

This can be adapted to other commands as necessary.

Tags: cli, bash, awk, motd

Bulk Change of File Extensions in Directory

If you have a directory of files where you want to change all of the file extensions you may find this code snippet useful to rename them:

for i in *.tmp
do
mv ${i} ${i%.*}.txt
done

Tags: cli, bash, rename, motd

Bash Environment Variables

Here are some bash environment variables that are useful to know when you're using the command prompt:

$0 - name of shell or shell script
$1, $2, $3, ... - positional parameters to script
$# - count of positional parameters
$? - exit status of most recent foreground task
$- - current options that are set for the shell
$$ - PID of the current shell (not subshell)
$! - the PID of the most recent background command
$DESKTOP_SESSION - path to the current display manager
$EDITOR - preferred text editor
$LANG - current language
$PATH - directory list to search for executables (programs)
$PWD - current working directory
$SHELL - current shell
$USER - current username
$HOME - current user's home directory
$HOSTNAME - current name of the host
$TERM - current terminal

Tags: cli, bash, motd