Manipulating files with grep
What is grep?
grep is a powerful command-line search tools that is included as part of most Unix-like systems. It is one of the most useful tools in bioinformatics! At the most basic level, grep searches for a string of characters that match a pattern and will print lines containing a match. Basic syntax is:
grep 'pattern' file_to_search
By default, grep will match any part of the string, so for example:
echo 'my dog is brown' > sample.txt grep 'dog' sample.txt grep 'do' sample.txt grep 'd' sample.txt
The above three grep commands will all match the text in the example file and will print the line. However, there are a huge number of arguments that can modify how grep behaves. Here are a few useful examples!
grep -w matches entire words.
- So in the above example:
grep -w 'dog' sample.txt would match the string, but
grep -w 'do' sample.txt would not.
grep -i allows case-insensitive matches
- In the above example,
grep -i 'DOG' would still match the line
grep -v inverts, returning lines that do not match the pattern.
grep -o returns only the matching words, not the entire line.
grep -c counts the number of lines that match the pattern.
- Equivalent to
grep 'pattern' file | wc -l
Print lines before/after a match:
grep -A [n] returns matching line and n lines after match
grep -B [n] returns matching line and n lines before match
grep -C [n] returns matching line and n lines before and after match
grep -A is very useful for pulling out specific lines from a FASTA file...just make sure your FASTA file is single line and not multi-line!
grep -f pattern.txt matches a list of patterns contained in pattern.txt against target file.
- I.e. the command
grep -f patterns.txt file.txt will match all patterns in patterns.txt against file.txt
- Pattern file has one pattern per line
Remember that we can combine different grep arguments with each other! E.g.:
Say we want to pull out a specific sequence from a FASTA file, like the X chromosome. We use the
-w flag to match whole words, combined with
-A to get the line after a match:
grep -w -A 1 '>X' dmel-all-chromosome-r6.20.fasta
There are many other functions of grep! When in doubt, remember you can check the help page using
man grep...or just by using google :)
Pattern matching with regular expressions
Regular expressions, aka "regex", are patterns that describe sets of strings. In other words, they allow you to match complex patterns with grep, not just exact matches. Regex is extremely powerful, but can also get (very) complicated, so we'll just stick to a few basic uses.
Regex has certain meta-characters that are reserved for special uses:
^: matches pattern at start of string $: matches pattern at end of string .: matches any character except new lines : matches any of enclose characters [^]: matches any characters *except* ones enclosed (note: is different from ^) \: "escapes" meta-characters, allows literal matching
Note that if we want to match any of these special characters literally (e.g. matching a period character "."), we would need to use a "\" to escape it first:
grep '\.' dmel-all-no-analysis-r6.20.gff will literally match a period.
One example of how regex can come in handy is using the ^ special character to quickly count how many sequences are in a FASTA file, which we would do as follows:
grep -c '^>' mel-all-chromosome-r6.20.fasta
This command matches lines in the FASTA file that start with a ">" character, i.e. the header lines, and uses the -c argument to count how many matches!
Here are a few more examples to show you how grep can help you wrangle your files!
- Use the
-vflag to invert grep and filter out lines matching unassigned contigs (chrUn) in the file hg19.genome and direct the output to a file:
grep -v 'chrUn' hg19.genome > hg19_noUn.genome
- Use grep to pull out the header line of only the 2R chromosome arm from dmel-all-chromosome-r6.20.fasta, using
-wto avoid partial matches:
grep -w '^>2R' dmel-all-chromosome-r6.20.fasta > 2R_header.txt
- Use grep from a list of patterns with
-fto extract the lines of only the major chromosome arms (2L, 2R, 3L, 3R, and X) from the file dmel-all-no-analysis-r6.20.gff:
printf '^>%s\n' 2L 2R 3L 3R X > major_arms.txt grep -w -f major_arms.txt \ dmel-all-no-analysis-r6.20.gff | less