CBW AWS/UNIX Primer 2026 | 05: Putting it all together

Okay. Um so for this example we are going to be creating a script. Uh a script is uh essentially a routine that you can tell uh that you can tell the uh compute environment uh to run and it can make uh repetitive tasks easier. Uh it can it uh it's also helpful for these repetitive tasks because uh you're less likely to make typos uh just based on uh pure chance and uh you will be able to reproduce your results which is becoming increasingly uh important in bioinformatics if you can demonstrate uh how your results were obtained. So to do this script uh I've navigated back to my home directory but where we're going where we are going is the data directory and inside the data directory we're going into the genomes directory. Okay. Um so inside of genomes uh we would have seen this a little earlier uh if we list here we have uh several GBF files and uh some of the files that uh we created earlier. Uh just because we don't just because I know we're not going to need them anymore. I'm going to make a point of actually removing um removing uh bad file and lengths.txt. So I can do this by RM bad file. I just typed bad and tab to complete it. And similarly I will do rm len tab to complete. And if I hit or if I use ls again we can see that we only have the gbff files now which is perfect. Could you do both in one line? The reason I did it in two lines is because I forget. We can try it together and see what happens. Uh let's make a couple test files. Uh a command you can do to make an empty file. I don't know how often you'll use this, but you can use touch uh the arrow uh test one. T Whoops. Uh the mistake I made there is I was trying to use touch like I would use cat when the reality is that uh instead touch doesn't need one of those uh arrow uh operators. I can just hit touch test one.txt. Uh I'm going to touch another file. And so if I list the directory now I have test one and test two. So the way the easiest perhaps the most efficient the fewest keystrokes would be rm star.txt because uh our other files are gbff at the end and the ones we want to remove are txt. Um but for the sake of demonstration we can rm test one.txt. I'll add a space test 2.txt txt and hit enter to see what happens. The shell appears to be happy. We can hit list and our GBF files are still there and our txt files are gone. So, great question, Katherine. I learned some I learned something else today. Okay. Now looking at our uh GBA FFF files uh if we want to know what's in them recall that we can use uh the less program uh I will just look at the first one which is Atlanta. So I'll type ATL tab and enter. And so I myself don't work with uh these types of files. So um we can see now visually that there uh appear to be uh two columns. Uh one column appears to have some labels. So I'm seeing locus definition accession on the left side and then uh uh uh next to these uh there are some text fields. So under locus I'm seeing this um some version title etc. And as we scroll through here we can see that there are some uh some aspects of this file that repeat. uh specifically um when we're looking underneath the features and the location/qualifiers portion of this file, we can see that uh there are locus tags for uh genes that are contained within this GBF. So this gene for instance has this locus tag. If we look a little further uh down the file um there is another locus tag. Ah yes we are in Atlanta.gdff. And so I can see that there are some repeating patterns in uh in this file. And so we can start to ask questions about what's contained within. If I were to use something like WC to count all of the lines in the file, uh that might give me something I want to know. But if I wanted to know for instance how many of these uh locust tags uh and by proxy how many genes are contained within the file uh WC would give me a a very large overestimate on its own because it's counting the number of lines when the reality is a lot of the lines of this file are things that we're just not interested in. So, this is where we can uh use some of our pipes. And we can do this inside of a script. So, I'm going to hit Q to get out of less. I'm going to use nano to create our script. Uh just because uh since we're already in the shell environment, it'll be easy to do so. So, nano. And then I'm going to call I'm going to call this script count tags.sh. So two things to note about this name. The first is uh when I first demonstrated nano I did not give it a name and you don't have to but uh it is helpful uh to name your file from the beginning. And the second is the extension. Now again uh I mentioned that uh the shell is generally agnostic to file extensions. Um but it it helps us as humans differentiate what we're looking at. And so what we're creating here is a shell script. And so should indicate to us the humans that um the file count tags is a shell script similar to how a txt extension would indicate to us that it would be a text file. So I'll hit enter. We can see uh we're in GNU Nano7.2 and our working file is count tags.sh sh and there's nothing in it because we just created it. So, I'm going to work through a script that first is going to uh look at that atlant.gbff file and try to find out how many tag how many locust tags are inside it. So, we can start by uh using the echo command I showed you earlier, which just outputs uh some text to the standard out. I'm going to use a uh special option with echo called uh dash n. Uh, this tells Echo that I want uh after I give it some text, I want it to put a uh a new line character in. This is equivalent to pressing enter on a keyboard. Uh, but by default, Echo doesn't do this. It'll just string everything together on one line. So, I'm trying to create a second one. Uh, to tell Echo uh what to write inside these quotations, uh, I'm going to write, how do you spell Atlanta? Atlant GB FFF colon and close the quotes. I'm going to press enter and go to a new line. And this is where I'm going to show you a new command called GP. Uh, Grep is a program that can look through uh, look through text. Uh, whether that text is in the standard out and you are piping it from another command or it that text is inside of an existing file. Uh, GP will look for uh, something called a regular expression or you can think of it as just looking for a match uh, to some text that you give it. So, uh, I will use the GP command. I'm going to tell Grep that I want to find all of the parts of the text that start with locus tag because you'll recall this is uh this is one of the repeating portions of our file. GP also needs to know what file to look in. So I will tell it we want to read Atlanta.gvfff and uh because this is uh going to because I know this is going to output a lot of uh individual lines as a result of this GP command uh I'm instead going to catch the results uh with the WC command and transform it into something a little more informative. So using the pipe I just I demonstrated earlier, I'm going to take all of the matches that GP produces, I'm going to feed these into the WC command to count uh what to count the number of lines that are coming out of the output. And in order to get only the lines uh I will use the -L flag. Now to save in nano you can the best practice is to hit control O where it'll give you the option to file name to write uh we're writing to count tags.sh uh if you were for instance to uh hit controll x without saving it is still going to give you the option to save a modified buffer. Okay. So now we have this count tags.sh file. If we want to see what's inside, we can use cat and this will show us this will show us the contents of the file, but it won't actually run it. We can see that our echol line is there and that our grap line is there. If we actually want to run this script, we have to we have to tell the shell that uh we want it to interpret the contents of this file in bash. The way that we do that is by first typing bash. And then uh we could say we want to run our count tags.sh. I hit count underscore and tab complete. And now when I hit enter, you can see that uh the output uh of our script is written to our shell here. Uh the text.gbf is there and we can see this number 12,237. So, um the to demonstrate um that the script is working properly, I want to show you just the GP portion of it. And so, uh instead of running the script again, I'm going to type the GP command. So rep, we're doing uh slashlocus tag equals that Atlanta. Uh locus tag is the string we're trying to match. The file we're looking in is Atlanta.gbff. And if uh I feed this to the WC command with the L option, it also spits out 12,237. Uh for the sake of demonstration, I'm going to pull the previous command back up. I'm going to backspace the WC part and the pipe part because I think it's interesting to show you uh what is happening under the hood. Um, this is about to output 12,000 results. So, sorry if this lags, but we can see that um, Grep is spitting out all of these uh, locus tags to our standard out. Now this is well and good but um if we are relying on the count that we were given by the script we just made uh we may notice a problem which is that um there are duplicates in these locust tags and this is a consequence of the of the format of the original file. But um we can see that the one ending in 435 happens twice. The one h ending in 545 occurs twice. And this happens in a lot of cases. Presumably not all of the cases because we got an odd number from our original script. But uh enough that it's a problem. So there is something we can do to fix this which is to try and find only the unique um try to find only the unique results uh of the GP command. So to do this I'm going to clear my screen. I'm going to go back into our script by typing nano count tags.sh and hit enter. And we're back in our script. Now, uh oh, quick trick. Uh if you want to get to the end of a line, you can use the end button on your keyboard. So, uh I'm at the end of the first line here. If I hit down, it puts me in the middle of Atlanta because uh Nano treats each of these uh each of these characters as an individual column in the file. So, technically that's right below the end of the first line. If I want to go to the end of the second line, I can hit the end button on my keyboard and go to the end of WC-L. So what I want to do is find only the unique tags that are uh being found by the GP command or rather I want to filter the results of the GP command so that only the unique tags are remaining. So to do this, I'm going to add another pipe to uh Whoa, that wouldn't work. That wouldn't work because WC just outputs a number. Instead, I want to go back here to after the GP command. What I want to do first is sort the results. And any sorting here works. We're we're dealing with text. So the default lexographic sorting is just fine for us. We don't need any extra parameters. Um and this sort part is important because the next thing we're going to do is use the unique program which is just shortened to uniq. What unique is going to do is look at uh the adjacent lines in a file to see if there are any duplicates. So if if there were two duplicates in a file and they were uh separated by a few lines, unique wouldn't remove them because it's only looking at the neighbors of that line. But because we're sorting the file first, all all of the all of the things that are duplicates are going to be right next to each other because they've been sorted. So unique is going to be able to look at all the neighbors, remove the duplicates and then to finish up uh I want to count the number of lines that come out of this pipe. So first GP which for which the results are sorted for which the uh the unique uh tags are found and I want to count all of these with WC-L. So I will follow best practice here and then hit control O to save. I will hit enter and then Ctrl X to exit. So now when we run our script with bash, I'll do count tab tags.sh. If I hit enter now, we see that the number has changed. And this is because we were able to use sort and unique to uh to remove those duplicates. Okay. Do we have any questions thus far about the script or any of the pipes, any of the programs? Sorry, can I see your uh your accounts uh tagsh to see if I wrote correctly? >> I'm sorry. My uh speaker was muted for the first half of your comment there. >> Can I see your shell script? Just one sec to see if I wrote it correctly. >> Yep. >> Thank you. >> I can do that. I will go back into nano. I'll count tags.sh. And here. >> Perfect. Thanks. >> Yep. No worries. Why did I mute myself? I'm the instructor. Okay. So, now we can be confident that we're able to create a script. Now, we're going to really start to uh give the script a little more power. And that's by using something uh called a loop. And you can think of loops as as a routine that's contained within the script. I mean, the script is a set of instructions. If there's a part of the script that you want to be run multiple times, that's when you would use a loop. And in order to construct this, we're going to use some very uh specific uh language. And in bash uh and the syntax can vary depending on the programming language. But in bash, we're going to be using uh for and do. And so by I'll hit my up arrow to get the nano command again, hit enter, and we're back in our script. So let's say that I want to create a loop. What I'm going to do is first create some space at the beginning of my script by pressing enter a couple of times. The nice thing about um uh these bash scripts is they're generally agnostic to spaces and enters uh most of the time. Uh so I can add as many new line characters as I want in here and it's not going to create a problem. So as I mentioned the specific language we're going to use is uh for and do. And so you can think of these as the building blocks of our uh of our loop. So what comes what comes after four uh are the parameters that we want to give to our uh to our loop. So the things that we want the loop to use and then what comes after do are the things that we actually want the loop to do. So the set of instructions that uh that we want it to follow. So to create this for loop, uh what my goal is to essentially do do this uh echo and grap uh these echo and grap operations uh multiple times. Let's say I want to do it for uh two files instead of just one. We can use something uh we can use something called a uh a variable here where it think of it like a wild card but it's it's not a wild card that can match everything. It's a wild card that takes on the definitions that you give it and hence it is variable and we call it a variable. We can call this variable whatever we want but uh for for instruction sake I will say for file name file name is our variable. Um now we need to tell uh the script what we want these file names to be. So we're going to give it a couple of options um using uh the word in. and in is telling the program uh to look for uh to look for an array to give uh to give to the file name. So I'm going to give the file name two things that it can possibly be. Uh the first can be Atlanta. GBF and I'll put another item in the array here uh which will be London.gbff GBF which is another one of these genomic files that we have. So what we've done here is told the program that uh for this array of which we have two file names Atlanta and London uh to take on that file name and it's also keeping count of how many things are in this array. So because our array has two things in it, Atlanta and London, it's going to do this loop two times. Now we're going to construct the do section. And fortunately, we have a lot of this already created uh from our previous script, but there are some things that we have to change. I'm going to navigate down to the echo line. And you'll you'll notice that this says atlant.gbff in plain text. If we were to run this with uh both the Atlanta and the London file, um it would say Atlanta at the beginning of the count both times. So, what we want to do in this case is uh take advantage of the fact that we now have uh access to a variable. I'm going to remove Atlanta and instead I'm going to put our file name uh variable in within these quotes. So to uh access a variable that you've created at the beginning of your loop, you have to use uh this uh dollar sign which uh is telling bash to evaluate that variable uh in in the place where you've put it. So I will say file name I will add a space and then uh close the quotation marks. And then so we know this for every uh iteration through the loop uh we're first going to echo the name of the file that we're using. And now you'll notice in the GP section of our script uh it also says atlant.gbff. So we would get the same result twice. So, in order to fix this, I'm going to go down, navigating with my arrow keys. I'm going to remove Atlanta. GBF, and similar to above, I'm going to put uh file name starting with the dollar sign to tell bash to evaluate it. File name. And the reason we don't use quotation marks here uh be is because uh in the echo line we want it to be a string. Whereas in this gre line uh we want we want the the shell to interpret this as an actual as an actual file not just not just some text. Okay. I will go to the end of this line by hitting end on my keyboard. I accidentally hit page down which is why I jumped down. And in order to finish our loop and tell it that we're done, we just simply type done. I will save this by hitting control O. Uh we will write it to the same script and I can exit. So now if I've done this correctly, if I type bash and count tags, which we have just updated with our loop, it now tells us the counts of both the atlant.gbff file with the unique tags and the London.gbff file with unique tags. And we can tell that our variables worked because um the file name uh that we've output with the echo portion is different and the counts are different uh between uh the results of our GP line. So that was a lot to take in all at once. Do we have any any questions from the group or any things that I can clarify or explain a little bit better? Okay. >> I have a question about the um the unique >> Yep. >> So, will that work if it's if you have more than two identical um numbers in a row? Let's say if you have three or four, would it also just give you the unique ones? >> Yep, it absolutely would. So, um, I can I'll make a quick test file here. Let's at this test.txt. Let's say one one one g3 and I will get D or control D to exit cat. Uh, so now if we look inside test.txt dxt with cat and without using the uh arrow we can see these numbers are here. If I wanted to use the unique command now uh you can see that it uh it it was able to remove all of the consecutive ones and keep only one instance of uh of each number. And so it would do the same thing for the tags or for text. >> Great. Thanks. >> Yep. Uh I will remove this test file and clear my screen. Okay. What do we have going on now? Great. So quickly inspecting our script. I will use head in this case. uh count tags.sh. Now, because I used head and I'm not in nano, I don't get the nice color coordination. Um, but that's okay. Uh, we can see that the file names that we're using are actually hardcoded into this loop. So, so far, uh, we're only able to use this loop for files called Atlanta.gbf or London.gbff. If we wanted to use another two files, we would have to go in and change the script each time or we can use uh another uh another set of special variables that are included in bash uh which I will demonstrate now. So, we'll go back into editing our count tags uh file with nano. And I'm specifically looking at the uh uh the first line. So, for file name, we're going to keep that the same because that's okay. I'm going to use my delete key to remove uh Atlanta and London. And I'm going to use another another uh couple of variables here. And so in bash there are a couple of variables that are intrinsic to the shell. And so an example of this is the variable one and the variable two. And what these uh what these are telling our script is to look for when we when we ask uh the shell to run our program to look for the argument in position one or the first position after we name the script and position two after we name the script. So to demonstrate how this works, I will save our script. Let's get out of here with Crl X. And now if I bash count tags.sh, I can give it to different file names. Now I've forgotten what the file names are. So I'm going to hit ls to see what's in this folder. And so somewhat arbitrarily, I will do bash counts.sh. Let's do Nevada. And you can tab complete when you're doing this as well, uh, when you're adding arguments to your program. So I'll tab complete Nevada and I will tab complete Texas. And so now we know our program is looking for the uh the arguments in position one and position two uh after uh we tell it we're running count tags sh. So if I hit enter, we see that Nevada has 5600 and change uh in uh unique tags and Texas has 5424. Now, another thing I'll demonstrate is that because we've only told the script to look for the first two positions, if we were to give it a third position, I'm bringing back the command uh let's say or a third argument rather, London.gbff. It's not going to run uh it's not going to run our loop for that third file because uh it's not within the parameters that we set. Inversely and perhaps uh not intuitively, if I have only one of the genome files and I run our loop, it's going to run just fine. And so, uh, depending on how you've written your script, sometimes this can break things, but in this case, it works. It just runs on the single file. And so we can do this to um run two files at the same time. But as I've shown you, maybe you want to run three. How are we going to do this? Well, we can use another one of these special variables in bash. And this one is a little more akin to a wild card. So I will go back into count tags through the nano program. And for uh the uh and for the array of things that we want file name our variable to be able to take on I'm going to use one of another one of the special bash uh variables which is dollar sign telling uh the the shell to interpret this and the at sign. And the at is a special variable which can represent uh any position uh or any argument uh in the program you're running. So uh it can be from 1 to two to 1 2 3 and so on and so forth. You can put as many file names as you care to type in here and it should still work. So control O to save. I'll press enter to write it to this file. Crl X to exit. And I'm going to clear our screen just to clean things up. And I will use ls to see what our files are. So because we've made our script now able to accept any number of arguments, we can use one of the wild cards I've shown you before. So if I run our bash script uh count tags.sh for for an argument I can use this star and then period orgff which will match all of the file names that you see above this line uh from Atlanta to Texas uh that end in the GBF uh extension here. So now that we've modified our script, when I press enter, it's going to work its way through that loop and uh for each of the files in the directory uh that match this regular expression. Uh we can see now that the script has counted the number of unique tags within that file. Okay. Um, that was throwing a couple bigger concepts at you rather quickly. Uh, do we have any questions or is there anything I could go over again? All right. Um there are only a couple things that I want to uh demonstrate to uh finish up this module. The first is uh the idea of comments. And so if we are working in this uh nano editor on our counts or on our account tag script, can we include the GB FFF in the bash script? That's a good question. We can try it. I expect it's not going to work and I should be able to explain why, but we can try it. Let's nano count tags.sh SH and so if we do GBF here and we can save it. Uh control O uh file name to write count tagsh enter and we'll exit. I'm going to run this script again by hitting up arrow a couple times till I see the command I want. Uh so I want to bash run count tags.sh SH was bash matching all the GBF files. Let's see what happens. Okay, something happened. Let's see if we can figure out what it's trying to tell us. I think. Okay. So, I think what happened here is that um this created uh this created an array of all of the GBF files that uh that are in this directory. But in the last item of the array, which was the Texas.gbff file name, bash uh appended this extra GBF string. So if I get out of nano bytrl x, we can see that uh from this error here, the script was looking for Texas.gbff. GBF. This isn't necessarily a behavior I would have expected. Could you do something? I think you probably could. Okay, let's try it. We have a minute. Yeah. So, um I'm going to combine uh or yes for file name in star gbff as David just said this would work. Could you do something like uh dollar sign add and dollar sign gbff? I believe you could. And I think what this is going to do is run every file twice. But we can try it. and then see if we can figure out why. So I will save this and get out of here and we can bash it again. Right. Okay. So we can see that it worked through the loop twice. We started at Atlanta, went down to Texas, and then Atlanta all the way through to Texas again. And the reason this happened is um what we've done here is after the word in we are creating an array of things to give to uh to give to the file name variable. when um when we use the dollar sign at to match the GBF in uh in our directory that's creating one part of the array where you get each of the file names and then in addition to that if we hardcodegF into our script uh it uh it's creating a second array kind of that is again contains all of the file names names that are within the uh within the directory you're working in. Now, I mean, if you were working in a directory with no GBF files, uh this wouldn't uh give you anything. Um but because we are uh it essentially created the first array from the wild card that we gave the program and the second array from being hard-coded in there looking for those GBF files. If we hardcoded just GBF files in here, I'll save this. Oh, it's the wrong button. If I run the script again, it should only run once, right? And if I take out the wild card here, right, it knows to look for GBF files just with just from the script. Okay. I'm going to put this back to uh dollar sign at for that special wild card. And what I'm going to do now is introduce the idea of a comment. And a comment are things that you can add to your scripts to help uh yourself better understand them, to help anyone who's reading them better understand them. and you can use it to have code in there that you may be working on but you don't necessarily want to run right then. Generally, it's bad practice to leave a [snorts] lot of code commented out. So, don't do that. Uh do as I say, not as I do. But for instance, if for every time in our loop, uh let's say I'll add another echo here. I'll echo with a new line. Something I think to do in the loop. And I will save this. Exit with control X. We're going to clear our screen. And then now if I bash app tags.sh SH uh matching all of our GVF files in this directory. It's adding I'm doing a loop to the beginning of each of the uh each of the lines of the output of our standard out. So count tags.sh if I don't want it to do that. Oh, I said something wrong earlier. By default, echo gives you a new line and the dash n tells it not to do a new line. I think I told you the inverse earlier. Anyway, um if instead we didn't want it to give us this message every time, we can use the hashtag character and a space. And this creates a comment. And a comment is something when when the when the when the kernel sees uh the hashtag, it just decides, oh, this isn't for me. I'm not going to read it. I'm not going to interpret it. So now that our comment is in there, if I save this file and exit, and then we run the command again with bash, we can see now that that line that was originally outputting the extra text uh is no longer being interpreted because um there because the I'm doing a loop text isn't in our final output. And so you can leave comments wherever you want in your code. Uh as long as it's not at the front. You could leave it for instance here at the end of this echol line and I want to say uh this outputs the file name. I will save it and exit. Similarly, uh if I run the bash program again, you'll see there's not the program h excuse me, the program hasn't changed because as soon as it sees the hashtag, it's reading it line by line, character by character, left to right. As soon as it sees the uh as soon as it sees the hashtag, uh the kernel says, "This is a comment. I'm not going to bother with it." The last thing I'll show you is how to uh convert this into an executable script. And so we've been executing it by telling the shell that we want to run it with bash. Um but there is a shorthand for running scripts. Um which is a dot and a slash. And so for exec for things that are executable, which I'll show you how to create in a moment, this is shorthand for I want to run this thing. So, it'll say you typing bash uh every time. If you are trying to run a script with uh something, excuse me, that exists in a file path that's far away, it can be annoying to have to uh type that every time before your program. So, we're going to first change the permissions of this uh of our script, our account tags.sh sh script. We're going to do this with a uh a program called chmod to change the permissions. I'll quickly show you the manual for chmod. Change file mode bits. We can see that uh ch takes options and then uh is looking for a file. It changes according to the mode. Um there are different ways that we can feed uh chod uh uh certain parameters. Uh but the way that I'm going to demonstrate it is not shown in this manual. Annoyingly, I'm going to use chod. I'm going to use this argument plus x. This is slightly different than a flag in that this is a specific um this is a specific argument that's specific. This is an argument that's specific to the chod program. Plus x is short for um uh plus as in add in I want to make it do this thing and the x stands for executable. So chod I'm changing the permissions. I want to make the thing executable. What do I want to make executable? That's going to be our count tags.sh script. Now, because we're in a bash in Oh, and once you do this, if we list our directory, you can see that count tags.sh is now green as opposed to uh earlier when it was uh just uh it looked like a text file to the shell. So it was it had white flavor text. This can differ between uh your uh your shell environment or your console. Uh but in our case it should be green. So to execute this now I can use that shortcut I showed you earlier which is the dot and the slash. And I will say count tags.sh. And this might work. This should work. I'll hit enter. It didn't work. The reason it didn't work is because the shell is again uh agnostic to file types. You and I know that this is a bash script because it ends insh, but the computer doesn't know that. So, one final thing we have to do uh is give the computer some information as to how it's supposed to work with this file if we don't specify bash in the command we use to uh run the program. So, one last time, we're going to nano the count tags program and I'm going to add an important piece of information to the top of the file here. So, first I'll press enter a couple times. and I'll use my up arrow to go back to the top. Um, this is something that has to be uh at the top of the file. I'm going to add something called a shebang. That's short for hashtag and exclamation point like hashbang exclamation point loud. Um, and now I need to tell the shell uh what uh what language to use to interpret this uh this program that we've written. So um the bash the bash executables are stored in a place called /bin /bash. This is something that we can uh this is something we can add to the top of the file. I will save it with control o hit enter. Uh exit this with control or controll x. And now if I try doing the dot slash notation again for count tags.sh. I think the reason it didn't work last time is because I didn't give it any arguments. Someone should have caught me on that. I think someone did was just being too polite to say anything. GB FFF. Uh we will match all things ending in GBF. And now uh and now our script works properly. Because this is going to bother me, I'm going to back back in and see if removing uh the shebang line is going to do anything very quickly. Aha, it would have worked. And the reason it would have worked is because we are in a bash environment. Uh but sometimes you're going to be using other languages. This could include uh R, which we'll be using in the workshop. You may be using pearl from time to time which is common to combine uh in some bash programs. Uh you could be using Python. Um you need to tell uh you need your script to tell the uh to tell the kernel how to read your program.