> For the complete documentation index, see [llms.txt](https://lu-lab.gitbook.io/training/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://lu-lab.gitbook.io/training/part-i-basic-skills/3.bash-and-github.md). # 3.Bash and Github How to set up multiple jobs as a pipeline ## 1) Github ### (1) Desktop Version for Mac or Windows I downloaded the [desktop version of github](https://desktop.github.com/) on mac. ### (2) Terminal Version for Linux or Mac **Setup:** 1\) add a setting file: \~/.gitconfig: ``` [user] email =[NNN@NN.com] name = Shared ``` 2\) Copy your ssh key to github website ([set up ssh key](/training/part-i-basic-skills/3.bash-and-github.md))\ 3\) Add a repository in github website **Clone a repository:** `git clone git@github.com:lulab/RNAfinder_Server.git` **Add:** ``` git add exmaples/ git commit -a git push origin ``` **Change:** ``` git commit -a git push origin ``` **Remove:** ``` git rm *.file git commit -a git push origin git pull? ``` **Pull (update)**: ``` git pull origin master git log -n 2 # look at the last two log entries. ``` [**more**](https://www.evernote.com/l/ABK7Gt9sva1CkLG9QfxqpZoog5uQoPDB_BU) > **Case study**: how to use google to solve question > > * Q1: How to install terminal version of git in mac? > > * A1: google "[install git \_on \_mac](https://www.google.com.hk/search?safe=strict\&q=install+git+on+mac\&spell=1\&sa=X\&ved=0ahUKEwiin6jE1urZAhVFzWMKHfb5BuMQBQgjKAA)" --> I got [the answer](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) > > * Q2: After I installed git, I got such error: > > > xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun" > > * A2: google this whole sentence --> I got [the answer](https://apple.stackexchange.com/questions/254380/macos-sierra-invalid-active-developer-path) ## 2) Bash Examples ### Example I **install software and tools in a brand-new OS** (1) 安装vim ``` yum -y install vim ``` (2) 打开一个空shell文件 ``` vim myinstall.sh ``` (3) 复制如下shell脚本到smyinstall.sh内。此shell文件会自动安装包括：R-2.15.3，perl-5.26.1等 ```bash yum -y install wget yum -y install readline-devel yum -y install libXt-devel yum -y install gcc gcc-c++ yum -y install gcc-gfortran yum -y install libg2c.so.0 yum -y install less yum -y install bzip2-devel yum -y install zlib-devel yum -y install xz-devel.x86_64 yum -y install make yum -y install zip unzip mkdir /download cd /download wget -c http://www.cpan.org/src/5.0/perl-5.26.1.tar.gz tar -xvzf perl-5.26.1.tar.gz cd /download/perl-5.26.1 sh Configure -de make make test make install ``` (4) 执行shell脚本 ``` bash myinstall.sh > myinstall.log # Standard output to a log file bash myinstall.sh >& myinstall.log # Log both Standard output and Standard error bash myinstall.sh > myinstall.log 2> myinstall.err # Log STDOUT and STDERR separately ``` ### Example II **backup files using rsync and crontab** (0) install rsync and crontab ``` yum -y install rsync yum -y install crontabs ``` ``` mkdir /mac/backup # prepare the backup dir ``` (1) Prepare a backup script, for example, "*\~/backup.sh*" ```bash #!/bin/bash #1. Local backup RSYNC="rsync --stats --compress --recursive --times --perms --links --delete --max-size=100M --exclude-from=/home/john/.rsync/exclude" echo "1. Backup of /home/john start at:" date $RSYNC /home/john/data/ /mac/backup/ echo "Backup end at:" date #2. Remote backup RSYNC="rsync --stats --compress --recursive --times --perms --links --delete --max-size=100M" echo "2. Backup 172.22.220.20:/data/ to /mac/backup2/ start at:" date $RSYNC john@172.22.220.20:/home/john/data/ /mac/backup2/ echo "Backup end at:" date ``` (2) Using "crontab" command to execute the backup script routinely, and record in a log file, for example, execute the command "*\~/backup.sh > \~/backup.log*" in 5:10am everyday: ``` chmod +x ~/backup.sh ``` ``` 1) open crontab and edit it by the following command: crontab -e or crontab ~/cronjob 2) type in the following lines or write the following in a file (i.e. ~/crontab): # minute hour day_in_month month day_in_week command 10 5 * * * ~/backup.sh > ~/backup.log 3) exit and save (like in VIM) ``` ## 3) Tips * **awk** ```bash gawk '{print $2 $1}' FILE | head -100 cat FILE | gawk '(NR>2) {print }' gawk -f FILE # take command from file Begin { a=1; ORS = " "} # ORS: output record seperator ... ``` * **input STANDIN in your script** ```bash rnatotal.o <" test.fasta | \ # ^ means start with here grep --color -i "[^ATCG]" # ^ means non- here ``` * **A tee in your pipe** ```bash program1 input.txt | tee intermediate-file.txt | program2 > results.txt ``` * **Exit Status** ```bash program1 input.txt > inter.txt && \ # run program 2 only if program1 is success (i.e. exit status is 0: $? = 0) program2 inter.txt> results.txt program1 input.txt > inter.txt || \ echo "warning: an error occurred ``` * **Name dir with today's date** ```bash mkdir results-$(date +F%) # +F% format is 2018-03-18 alias today="date +F%" # you can put this in ~/.bashrc mkdir results-$(today) ``` ## 4）More Readings and Practices * [Teaching Video](/training/getting-startted.md#learning-materials)：Week II. 3. Bash * **for Beginners** 阅读和练习《鸟哥的Linux私房菜-基础学习篇》如下章节: > 第11章 认识与学习bash\ > 第12章 正则表达式与文件格式化处理\ > 第13章 学习shell script * **for Advanced Readers** * [**More Bash Examples at Github**](https://github.com/urluzhi/scripts/tree/master/workflow) * [**Additional Tutorial**](/training/getting-startted.md#learning-materials)**:** Bash * <\> * 5\) Git for Scientists * 6\) Bioinformatics Data * 12\) Bioinformatics Shell Scripting, Writing Pipelines, and Parallelizing Tasks ## 5）Homework 1. 利用google等工具找到如何让awk读输入文件时只按照tab来做separator？ 2. 学习和完成这里面的作业：[**Additional Tutorial**](/training/getting-startted.md#learning-materials) : Bash ([Teaching Video](/training/getting-startted.md#learning-materials)：Week II. Bash and R -- Shared by Li) ## 6) Video ### 6a) Bash and Github [@Youtube](https://youtu.be/mXAdfWw-OBg) [@Bilibili](https://player.bilibili.com/player.html?aid=30590318\&cid=53392706\&page=1)