Organizing Genomics Projects

Getting your project started

Project organization is one of the most important parts of a sequencing project, but is often overlooked in the excitement to get a first look at new data. While it’s best to get yourself organized before you begin analysis, it’s never too late to start.

You should approach your sequencing project in a very similar way to how you do a biological experiment, and ideally, begins with experimental design. We’re going to assume that you’ve already designed a beautiful sequencing experiment to address your biological question, collected appropriate samples, and that you have enough statistical power. For all of those steps, collecting specimens, extracting DNA, prepping your samples, you’ve likely kept a lab notebook that details how and why you did each step, but documentation doesn’t stop at the sequencer!

Every computational analysis you do is going to create many files, and inevitability, you’ll want to run some of those analysis again. Genomics projects can quickly accumulates hundreds of files across tens of folders. Do you remember what PCR conditions you used to create your sequencing library? Probably not. Similarly, you probably won’t remember whether your best alignment results were in Analysis1, AnalysisRedone, or AnalysisRedone2; or which quality cutoff you used.

Luckily, recording your computational experiments is even easier than recording lab data. Copy/Paste will become your best friend, sensible file names will make your analysis traversable by you and your collaborators, and writing the methods section for your next paper will be a breeze. Let’s look at the best practices for documenting your genomics project.

Your future self will thank you.

Exercise

In this exercise we will setup a filesystem for the project we will be working on today. We will also introduce you to some helpful shell commands/programs/tools:

mkdir
history
tail
|
nano
>>

A. Create a file system for a project

We will start by creating a directory that we can use for the rest of the workshop:

First, make sure that you are in your home directory:

$ pwd
/home/dcuser
# Hopefully you got the above output '/home/dcuser'

Tip If you were not in your home directory, the easiest way to get there is to enter the command cd which always returns you to home.

Next, try making the following directories using the mkdir command

dc_workshop
dc_workshop/data
dc_workshop/docs
dc_workshop/results
dc_workshop/src

Verify that you have created the directories;

$ ls -R dc_workshop

if you have created these directories, you should get the following output from that command:

dc_workshop/:
data  docs  results

dc_workshop/data:

dc_workshop/docs:

dc_workshop/results:

dc_workshop/src:

B. Document your activity on the project

The history command is a convenient way to document the all the commands you have used while analyzing and manipulating your project. Let’s document the work we have done to create these folders.

View the commands that you have used so far during this session using history:
```
 $ history
```
The history likely contains many more commands that you have used just for these projects. Let’s view the last several commands that focus on just what we need for the project.
View the last n lines of your history (where n = approximately the last few lines you think relevant - for our example we will use the last 7:
```
 $ history | tail -n7
```
As you may remember from the shell lesson, the pipe | sends the output of history to the next program, in this case, tail. We have used the -n option to give the last 7 lines.
Using your knowledge of the shell: use the append redirect >> to create a file called dc_workshop_log_XXXX_XX_XX.txt (Use the four-digit year, two-digit month, and two digit day, e.g. dc_workshop_log_2015_07_30.txt)
You may have noticed that your history may contain the history command itself. To remove this redundancy from our log, lets use the nano text editor to fix the file:
```
$ nano dc_workshop_log
```

From the nano screen, you should be able to use your cursor to navigate, type, and delete any redundant lines.

Add a dateline and comment to the line where you have created the directory e.g.
```
 # 2015_07_30 
 # Created sample directories for the Data Carpentry workshop
```
Next, remove any lines of the history that are not relevant. Just navigate to those lines and use your delete key.
Close nano by hitting ‘Control’ and the ‘X’ key at the same time; notice in nano this is abbreviated ‘\^X’; nano will ask if you want to save; hit ‘Y’ for yes. When prompted for the ‘File Name to Write’ we can hit ‘Enter’ to keep the same name and save.
Now that you have created the file, move the file to ‘dc_workshop/docs’ using the mv command.

C. Intro to Markdown

As you start to build your notebook, there is an easy way to format these notes - Markdown.

Markdown is a simplified version of HTML, the code that is behind most webpages on the internet.

One reason to markdown is that it is a simple way to create documents that anyone can access. You may be very use to using word processing programs like Microsoft Word. While these are very nice, they are not free and not everyone has access. Also, when we are doing work on the computer usually using Linux we would not be able to read these Microsoft documents.

To write something in Markdown, all you need to do is write text like you always do, but you will format it using some special symbols.

Using an editor

It helps to have an editor so that you can see what your markdown will look like when it is rendered (final version that people see). Suggestions for free software include:

Or online editors such as

and many others.

Markdown basics

See a really good guide at: https://help.github.com/articles/basic-writing-and-formatting-syntax/

Paragraphs

Regular paragraphs are no different than typing on any text editor/word processor. Just type.

Adding Headings

Use the ‘#’ key to make a line a heading (subtitle). For really big fonts use just one ‘#’ the more ‘#’s’ used, the smaller the subtitle.

# Data Carpentry workshop
will become

Data Carpentry workshop

## Organization of genomics projects
becomes

Organization of genomics projects

### 2017-03-23 becomes

2017-03-23

#### Created a simple directory structure for the Data Carpentry workshop becomes

Created a simple directory structure for the Data Carpentry workshop

Bolding and emphasis

Surrounding a word or phrase with one set of asterisks (*makes it italic*) makes it italic. Using two sets (**makes it bold**) makes it bold.

Lists

Lists can be

* Unordered

* Start with a single asterisk

* Have one item per line.

Which will be displayed as:

Unordered
Start with a single asterisk
Have one item per line.

To make an ordered list:

\1. Start the list with one number, followed by a period

\2. On the next line, just use the next number.

Which will be displayed as:

Start the list with one number, followed by a period
On the next line, just use the next number.

Links and Images

A link is properly written in markdown using a pair of square brackets ‘[]’ followed by round parentheses ‘()’. The text of the link will go in the square brackets, and the actual URL will go in the parentheses:

[Good Enough Practices in Scientific Computing]](https://arxiv.org/pdf/1609.00037.pdf) will become:

Good Enough Practices in Scientific Computing

If you want an image, just follow the link text in the brackets with an exclamation mark ‘![ ]( )’

![Kitten image](./img/kittens.jpg) will become:

Kitten image

Questions:

What is the default number of lines that tail displays?
What is the difference between ‘>’ and ‘»’

References

Good Enough Practices in Scientific Computing