University:
Boston University
Course:
MET CS 201 | Introduction to Programming
Academic year:

2023
Views:

63

Pages:

22
Author:

Triston Franco

Introduction to Shell Scripting with Bash Topics for Today ● ● ● ● ● ● ● ● ● Introductions Basic Terminology How to get help Command-line vs. Scripting Variables Handling Arguments Standard I/O, Pipes, and Redirection Control Structures (loops and If statements) SCC Job Submission Example Research Computing Services Research Computing Services (RCS) A group within Information Services & Technology at Boston University provides computing, storage, and visualization resources and services to support research that has specialized or highly intensive computation, storage, bandwidth, or graphics requirements. Three Primary Services: ● Research Computation ● Research Visualization ● Research Consulting and Training Breadth of Research on the Shared Computing Cluster (SCC) Basic Terminology The Command-line The line on which commands are typed and passed to the shell. Username Hostname Current Directory [username@scc1 ~]$ Prompt Command Line (input) The Shell ● The interface between the user and the operating system ● Program that interprets and executes input ● Provides: ○ Built-in commands ○ Programming control structures ○ Environment variables Script ● A text file containing a series of commands that an interpreter (like shell) can read and run. Interpreter ● A program that runs commands without compiling (directly from text) Bash The name of the most common shell interpreter, it’s language, and syntax. The default shell on SCC and What we are going to use today Teach a Programmer to Fish How to Get Help Manuals (“man”) and Info (“info”) scc1 $ man bash BASH(1) scc1 $ info bash General Commands Manual BASH(1) NAME File: bash.info, Node: Top, Next: Introduction, Prev: (dir), Up: dir Bash Features ************* bash - GNU Bourne-Again SHell SYNOPSIS bash [options] [file] This text is a brief description of the features that are present in the Bash shell (version 4.2, 28 December 2010). COPYRIGHT Bash is Copyright Foundation, Inc. by the Free Software This is Edition 4.2, last updated 28 December 2010, of 'The GNU Bash Reference Manual', for 'Bash', Version 4.2. DESCRIPTION Bash is an sh-compatible command language interpreter that executes commands read from the standard input or from a file. Bash also incorporates useful features from the Korn and C shells (ksh and csh). Bash contains features that appear in other popular shells, and some features that only appear in Bash. Some of the shells that Bash has borrowed concepts from are the Bourne Shell ('sh'), the Korn Shell ('ksh'), and the C-shell ('csh' and its successor, 'tcsh'). The following menu breaks the features up into categories based upon which one of these other shells inspired the feature. (C) 1989-2011 Bash is intended to be a conformant implementation of the Shell and Utilities portion of the IEEE POSIX specification (IEEE Standard 1003.1). Bash can be configured to be POSIX-conformant by default. This manual is meant as a brief introduction to features found in Bash. The Bash manual page should be used as the definitive reference on shell behavior. * Menu: Bash “help” ● Bash comes with built in help functionality ○ Just type “help” ● Read deeper into help chapters by searching specific keywords ○ “help [keyword]” ● “Help help” ● “Help for” scc1 $ help for help GNU bash, help: for: for help NAME version [-dms] [in WORDS 4.2.46(2)-release [pattern ......] ] ; do COMMANDS; (x86_64-redhat-linux-gnu) done These Display Execute shell commands information areabout for defined each builtin member internally. in commands. a list. Type `help' to see this list. Type `help name' to find out more about the function `name'. Use The `info Displays `for' bash' brief loop to executes find out a summaries more of sequence builtin about of the commands. commands shell infor If general. PATTERN each member is in a Use list `man of specified, -k'items. orgives `info' If detailed to find `in WORDS help out ...;' more on all is about commands not commands present, matching not in then PATTERN, `in this "$@"' list. is otherwise For assumed. the each list element of help in topics WORDS, is NAME printed. is set to that element, and A star the(*) COMMANDS next to are a executed. name means that the command is disabled. Options: job_spec Exit -d Status: [&] output short description historyfor [-c] each [-dtopic offset] [n] > (( Returns expression -m the)) displayof status usage the last in pseudo-manpage if command COMMANDS; executed. then format COMMANDS; [> for . filename ((: -sfor (( [arguments] exp1; output exp2; onlyexp3 a short )); jobs usage do [-lnprs] COMMANDS; synopsis [jobspec done for each ...] topic o> matching : Arithmetic PATTERNfor loop. kill [-s sigspec | -n signum > [ arg... ] let arg [arg ...] [[ Equivalent expression to Arguments: ]] local [option] name[=value] .> aliasPATTERN [-p] (( EXP1 [name[=value] Pattern specifiying )) ... ]> logout a help [n] topic bg [job_spec while (( ...] EXP2 )); do mapfile [-n count] [-O origin> bind Exit [-lpvsPVS] Status:COMMANDS [-m keymap] [-> popd [-n] [+N | -N] ...Returns success (( EXP3 unless )) PATTERN is not found or an invalid option is given. .. done . EXP1, EXP2, and EXP3 are arithmetic expressions. If any expression is function omitted, name it{behaves COMMANDS as;if } it o> evaluates variablesto - 1. Names and meaning> getopts optstring name [arg] wait [id] hash Exit [-lr] Status: [-p pathname] [-dt]> while COMMANDS; do COMMANDS; > help Returns [-dms]the [pattern status...] of the last{command COMMANDS executed. ; }? Documentation The official documentation is very good! So good, you might even see some examples copied directly into this tutorial. Command-line vs. Scripting Recap of Command Line vs Script Definitions Command-line ● ● ● ● Has a prompt Not saved One line at a time The text based way to interact with a computer Script ● ● ● ● No prompt Is a file Still runs one line at a time Runs all the lines in file without interaction Example CLI Task: Organize some downloaded data [username@scc1 ~]$ cd /projectnb/scv/jpessin/introToBashScripting_sampleScripts/cli_script [username@scc1 cli_script]$ ls data LICENSE sample1.chr1.bam sample1.chr4.bam sample2.chr1.bam sample2.chr4.bam sample3.chr1.bam sample3.chr4.bam README sample1.chr2.bam sample1.chr5.bam sample2.chr2.bam sample2.chr5.bam sample3.chr2.bam sample3.chr5.bam report.html sample1.chr3.bam sample1.log sample2.chr3.bam sample2.log sample3.chr3.bam sample3.log [username@scc1 cli_script]$ cd data [username@scc1 data]$ mkdir sample1 [username@scc1 data]$ mv sample1.chr*.bam > sample1 -bash: sample1: Is a directory [username@scc1 data]$ mv sample1.chr*.bam sample1/ [username@scc1 data]$ cd sample1/ [username@scc1 sample1]$ ls sample1.* > sample1.fileset.txt [username@scc1 sample1]$ less sample1.fileset.txt [username@scc1 sample1]$ mv sample1.fileset.txt ../ [username@scc1 sample1]$ cd .. [username@scc1 data]$ ls LICENSE sample1 sample2.chr1.bam sample2.chr4.bam sample3.chr1.bam sample3.chr4.bam README sample1.fileset.txt sample2.chr2.bam sample2.chr5.bam sample3.chr2.bam sample3.chr5.bam report.html sample1.log sample2.chr3.bam sample2.log sample3.chr3.bam sample3.log Example CLI Task (cont.) [username@scc1 data]$ ls LICENSE sample1 sample2.chr1.bam sample2.chr4.bam sample3.chr1.bam sample3.chr4.bam README sample1.fileset.txt sample2.chr2.bam sample2.chr5.bam sample3.chr2.bam sample3.chr5.bam report.html sample1.log sample2.chr3.bam sample2.log sample3.chr3.bam sample3.log [username@scc1 data]$ mkdir sample2 [username@scc1 data]$ mv sample2.chr*.bam sample2 [username@scc1 data]$ mkdir sample3 [username@scc1 data]$ mv sample3.chr*.bam sample3 [username@scc1 data]$ ls LICENSE report.html sample1.fileset.txt sample2 sample2.log sample3.fileset.txt sample4 README sample1 sample1.log sample2.fileset.txt sample3 sample3.log sample4.fileset.txt [username@scc1 data]$ mkdir logs [username@scc1 data]$ mv sample*.log logs/ [username@scc1 data]$ rm LICENSE rm: remove regular empty file 'LICENSE'? y [username@scc1 data]$ rm README rm: remove regular empty file 'README'? y [username@scc1 data]$ ls logs sample1 sample2 sample3 sample4 report.html sample1.fileset.txt sample2.fileset.txt sample3.fileset.txt sample4.fileset.txt sample4.log Command-line Interface ● Difficult to read ● One-directional / Non-reproducible ○ What did I do last time? ○ What should someone do next time? ● Manual ● Potentially error-prone ● Wasn’t really that fast Write a Script Instead reorgData.sh #!/bin/bash # Take datadir from input datadir=$1 cd $datadir # Detect number of samples numSamples=$(ls sample*.bam | cut -d. -f1 | uniq | wc -l) # Reorg sample files into sample dirs for sampleNum in $(seq 1 $numSamples); do mkdir sample$sampleNum mv sample$sampleNum*.chr*.bam sample$sampleNum/ ls sample$sampleNum > sample$sampleNum.filelist.txt done # Organize Logs mkdir logs mv sample*.log logs/ # Remove extra files rm -f LICENSE rm -f README scc1 $ ls data LICENSE README report.html sample1.chr1.bam sample1.chr2.bam sample1.chr3.bam sample1.chr4.bam sample1.chr5.bam sample1.log sample2.chr1.bam sample2.chr2.bam sample2.chr3.bam sample2.chr4.bam sample2.chr5.bam sample2.log sample3.chr1.bam sample3.chr2.bam sample3.chr3.bam sample3.chr4.bam sample3.chr5.bam sample3.log scc1 $ bash reorgData.sh data/ scc1 $ ls data logs sample1 report.html sample1.files sample2 sample2.files sample3 sample3.files Running Scripts: Interpreter ● Simply call the “bash” interpreter and provide the script. ● It will read line by line as if on the command line scc1 $ ls data LICENSE README report.html sample1.chr1.bam sample1.chr2.bam sample1.chr3.bam sample1.chr4.bam sample1.chr5.bam sample1.log sample2.chr1.bam sample2.chr2.bam sample2.chr3.bam sample2.chr4.bam sample2.chr5.bam sample2.log sample3.chr1.bam sample3.chr2.bam sample3.chr3.bam sample3.chr4.bam sample3.chr5.bam sample3.log scc1 $ bash reorgData.sh data/ This is what we did previously. scc1 $ ls data logs sample1 report.html sample1.files sample2 sample2.files sample3 sample3.files Running Scripts: Executable scc1 $ head -n 1 reorgData.sh #!/bin/bash Files can be made “executable” on their own. To do this, we need to: ● Provide interpreter information in script ● Set executable permission ● Run the script directly ./script scc1 $ ls -l drwxr-sr-x 6 cjahnke scv 32768 Jun 1 2:36 data -rw-r--r-- 1 cjahnke scv 453 Jun 1 2:37 reorgData.sh scc1 $ chmod +x reorgData.sh scc1 $ ls -l drwxr-sr-x 6 cjahnke scv 32768 Jun 1 2:36 data -rwxr-xr-x 1 cjahnke scv 453 Jun 1 2:37 reorgData.sh scc1 $ ./reorgData.sh scc1 $