Skip to content

Instantly share code, notes, and snippets.

@rjweiss

rjweiss/regex

Created November 21, 2013 06:44
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save rjweiss/7577022 to your computer and use it in GitHub Desktop.
Save rjweiss/7577022 to your computer and use it in GitHub Desktop.
Download ZIP
Regex
Raw
regex
{
"metadata": {
"name": "regex_lecture.ipynb"
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"source": [
"# Regex tutorial\n",
"\n",
"You write regular expressions (regex) to match patterns in strings. When you are processing text, you may want to extract a **substring** of some predictable structure: a phone number, an email address, or something more specific to your research or task. You may also want to clean your text of some kind of junk: maybe there are repetitive formatting errors due to some transcription process that you need to remove.\n",
" \n",
"In these cases and in many others like them, writing the right regex will be better than working by hand or using a magical third-party library/software that claims to do what you want.\n",
"\n",
"Please refer back to the slides to see the building blocks of regex.\n",
"\n",
"## Character classes\n",
"\n",
"- Used to match any one of a specific set of characters\n",
"- Defined using the **[** and **]** metacharacters\n",
"- Within a character class, **^** and **-** can have special meaning (complement and range), depending on their position in the class\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"import re #the regex module in the python standard library\n",
"\n",
"#strings to be searched for matching regex patterns\n",
"str1 = \"Aardvarks belong to the Captain\"\n",
"str2 = \"Albert's famous equation, E = mc^2.\"\n",
"str3 = \"Located at 455 Serra Mall.\"\n",
"str4 = \"Beware of the shape-shifters!\"\n",
"test_strings = [str1, str2, str3, str4] #created a list of strings"
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"outputs": [],
"prompt_number": 28
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"for test_string in test_strings:\n",
" print 'The test string is \"' + test_string + '\"'\n",
" match = re.search(r'[A-Z]', test_string)\n",
" if match:\n",
" print 'The first possible match is: ' + match.group()\n",
" else:\n",
" print 'no match.'"
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"The test string is \"Aardvarks belong to the Captain\"\n",
"The first possible match is: A\n",
"The test string is \"Albert's famous equation, E = mc^2.\"\n",
"The first possible match is: A\n",
"The test string is \"Located at 455 Serra Mall.\"\n",
"The first possible match is: L\n",
"The test string is \"Beware of the shape-shifters!\"\n",
"The first possible match is: B\n"
]
}
],
"prompt_number": 29
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's go through the code above line by line:\n",
"\n",
" for test_string in test_strings:\n",
"\n",
"`test_strings` is a list, and so it is iterable in a for loop. Every element in this list is a string. So for the rest of the for loop, we will be referring to the current element as `test_string`\n",
"\n",
" print 'The test string is \"' + test_string + '\"'\n",
" \n",
"This just prints out the current object we're iterating over\n",
"\n",
" match = re.search(r'[A-Z]', test_string)\n",
"\n",
"Remember the basic approach to using regex in Python. You give a searcher (in this case, the function `re.search()` a pattern and a string in which to find matches. That's exactly what this line does. `re.search()` returns either an object of type `SRE_Match` or `None`. \n",
" \n",
" if match:\n",
" print 'The first possible match is: ' + match.group()\n",
" else:\n",
" print 'no match.'\n",
"\n",
"`match` is an object that has two possible states: `SRE_Match` or `None`. `None` is a type of object that returns `false` in a logical test. In this for loop, we've basically told the Python interpreter to check whether match is `NoneType` or not. If it isn't, we return a string plus `match.group()`. `group()` is a method that `SRE_Match` objects have. By default, it returns the 0th group; we'll get to what that means later. For now, just know that it will return the substring that matched the pattern defined.\n",
"\n",
"Note that since we are using `re.search`, only a single character is returned. That's because of the following:\n",
"\n",
"1. We only defined a single character pattern and\n",
"2. `re.search` finds the first possible match and then doesn't look for any more.\n",
"\n",
"If you wanted to find **all** of the possible matches in a string, you can use `re.findall()`, which will return a list of all matches:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"for string in test_strings:\n",
" print re.findall(r'[A-Z]', string)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"['A', 'C']\n",
"['A', 'E']\n",
"['L', 'S', 'M']\n",
"['B']\n"
]
}
],
"prompt_number": 30
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can also compile your regex ahead of time. This will create `SRE_Pattern` objects. There are many performance reasons to do this. Additionally, you can create lists of these objects and iterate over both strings and patterns more easily. Here's an example:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"patterns = [re.compile(r'[ABC]'),\n",
"re.compile(r'[^ABC]'),\n",
"re.compile(r'[ABC^]'),\n",
"re.compile(r'[0123456789]'),\n",
"re.compile(r'[0-9]'),\n",
"re.compile(r'[0-4]'),\n",
"re.compile(r'[A-Z]'),\n",
"re.compile(r'[A-Za-z]'),\n",
"re.compile(r'[A-Za-z0-9]'),\n",
"re.compile(r'[-a-z]'),\n",
"re.compile(r'[- a-z]')]\n",
"\n",
"def find_match(pattern, string):\n",
" match = re.search(pattern, string)\n",
" if match:\n",
" return match.group()\n",
" else:\n",
" return 'no match.'\n",
" \n",
"for test_string in test_strings:\n",
" matches = [find_match(pattern, test_string) for pattern in patterns]\n",
" \n",
" for pattern in patterns:\n",
" print 'The first potential match for \"' + pattern.pattern + '\" in \"' + test_string + '\" is: ' + matches[patterns.index(pattern)]"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"The first potential match for \"[ABC]\" in \"Aardvarks belong to the Captain\" is: A\n",
"The first potential match for \"[^ABC]\" in \"Aardvarks belong to the Captain\" is: a\n",
"The first potential match for \"[ABC^]\" in \"Aardvarks belong to the Captain\" is: A\n",
"The first potential match for \"[0123456789]\" in \"Aardvarks belong to the Captain\" is: no match.\n",
"The first potential match for \"[0-9]\" in \"Aardvarks belong to the Captain\" is: no match.\n",
"The first potential match for \"[0-4]\" in \"Aardvarks belong to the Captain\" is: no match.\n",
"The first potential match for \"[A-Z]\" in \"Aardvarks belong to the Captain\" is: A\n",
"The first potential match for \"[A-Za-z]\" in \"Aardvarks belong to the Captain\" is: A\n",
"The first potential match for \"[A-Za-z0-9]\" in \"Aardvarks belong to the Captain\" is: A\n",
"The first potential match for \"[-a-z]\" in \"Aardvarks belong to the Captain\" is: a\n",
"The first potential match for \"[- a-z]\" in \"Aardvarks belong to the Captain\" is: a\n",
"The first potential match for \"[ABC]\" in \"Albert's famous equation, E = mc^2.\" is: A\n",
"The first potential match for \"[^ABC]\" in \"Albert's famous equation, E = mc^2.\" is: l\n",
"The first potential match for \"[ABC^]\" in \"Albert's famous equation, E = mc^2.\" is: A\n",
"The first potential match for \"[0123456789]\" in \"Albert's famous equation, E = mc^2.\" is: 2\n",
"The first potential match for \"[0-9]\" in \"Albert's famous equation, E = mc^2.\" is: 2\n",
"The first potential match for \"[0-4]\" in \"Albert's famous equation, E = mc^2.\" is: 2\n",
"The first potential match for \"[A-Z]\" in \"Albert's famous equation, E = mc^2.\" is: A\n",
"The first potential match for \"[A-Za-z]\" in \"Albert's famous equation, E = mc^2.\" is: A\n",
"The first potential match for \"[A-Za-z0-9]\" in \"Albert's famous equation, E = mc^2.\" is: A\n",
"The first potential match for \"[-a-z]\" in \"Albert's famous equation, E = mc^2.\" is: l\n",
"The first potential match for \"[- a-z]\" in \"Albert's famous equation, E = mc^2.\" is: l\n",
"The first potential match for \"[ABC]\" in \"Located at 455 Serra Mall.\" is: no match.\n",
"The first potential match for \"[^ABC]\" in \"Located at 455 Serra Mall.\" is: L\n",
"The first potential match for \"[ABC^]\" in \"Located at 455 Serra Mall.\" is: no match.\n",
"The first potential match for \"[0123456789]\" in \"Located at 455 Serra Mall.\" is: 4\n",
"The first potential match for \"[0-9]\" in \"Located at 455 Serra Mall.\" is: 4\n",
"The first potential match for \"[0-4]\" in \"Located at 455 Serra Mall.\" is: 4\n",
"The first potential match for \"[A-Z]\" in \"Located at 455 Serra Mall.\" is: L\n",
"The first potential match for \"[A-Za-z]\" in \"Located at 455 Serra Mall.\" is: L\n",
"The first potential match for \"[A-Za-z0-9]\" in \"Located at 455 Serra Mall.\" is: L\n",
"The first potential match for \"[-a-z]\" in \"Located at 455 Serra Mall.\" is: o\n",
"The first potential match for \"[- a-z]\" in \"Located at 455 Serra Mall.\" is: o\n",
"The first potential match for \"[ABC]\" in \"Beware of the shape-shifters!\" is: B\n",
"The first potential match for \"[^ABC]\" in \"Beware of the shape-shifters!\" is: e\n",
"The first potential match for \"[ABC^]\" in \"Beware of the shape-shifters!\" is: B\n",
"The first potential match for \"[0123456789]\" in \"Beware of the shape-shifters!\" is: no match.\n",
"The first potential match for \"[0-9]\" in \"Beware of the shape-shifters!\" is: no match.\n",
"The first potential match for \"[0-4]\" in \"Beware of the shape-shifters!\" is: no match.\n",
"The first potential match for \"[A-Z]\" in \"Beware of the shape-shifters!\" is: B\n",
"The first potential match for \"[A-Za-z]\" in \"Beware of the shape-shifters!\" is: B\n",
"The first potential match for \"[A-Za-z0-9]\" in \"Beware of the shape-shifters!\" is: B\n",
"The first potential match for \"[-a-z]\" in \"Beware of the shape-shifters!\" is: e\n",
"The first potential match for \"[- a-z]\" in \"Beware of the shape-shifters!\" is: e\n"
]
}
],
"prompt_number": 31
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's go over this code line by line: \n",
"\n",
" patterns = [re.compile(r'[ABC]'),\n",
" re.compile(r'[^ABC]'),\n",
" re.compile(r'[ABC^]'),\n",
" re.compile(r'[0123456789]'),\n",
" re.compile(r'[0-9]'),\n",
" re.compile(r'[0-4]'),\n",
" re.compile(r'[A-Z]'),\n",
" re.compile(r'[A-Za-z]'),\n",
" re.compile(r'[A-Za-z0-9]'),\n",
" re.compile(r'[-a-z]'),\n",
" re.compile(r'[- a-z]')]\n",
"\n",
"This creates a list of `SRE_Pattern`s.\n",
" \n",
" def find_match(pattern, string):\n",
" match = re.search(pattern, string)\n",
" if match:\n",
" return match.group()\n",
" else:\n",
" return 'no match.'\n",
"\n",
"I defined a function `find_match` that expects some variables called `pattern` and `string`. Notice that this function is very similar to the logical condition testing from the code above. Note also that this function returns either the match.group() or a string `\"no match.\"` \n",
" \n",
" for test_string in test_strings:\n",
" matches = [find_match(pattern, test_string) for pattern in patterns]\n",
"\n",
"By defining the `find_match()` function above, I can then call it from within a list comprehension. In words, for each string `test_string` that is in `test_strings`, I want to compare against the list of patterns and return matches. The resulting list of `matches` should be the same length as `patterns`; one match per pattern tested. \n",
"\n",
" for pattern in patterns:\n",
" print 'The first potential match for \"' + pattern.pattern + '\" in \"' + test_string + '\" is: ' + matches[patterns.index(pattern)]\n",
"\n",
"Because I wanted to print some diagnostic code, I need to iterate over each `pattern` in `patterns` (a list and thus iterable) and print it out, along with the test string. If you want to get the pattern out of an `SRE_Pattern` object, you can call its member method `.pattern` and it will return the regex pattern as a string. Since we are nesting this loop within the bigger loop above, this loop will go over every pattern in the `patterns` list for each string, and then repeat for the next string in the list `test_strings`.\n",
"\n",
"However, note that I am dynamically referring to the index of the `matches` list. By this, I mean the following code:\n",
"\n",
" matches[patterns.index(pattern)]\n",
"\n",
"Make sure this makes sense to you. Remember, `matches` and `patterns` are the same length. That means that if I want to return the match that correspondes to the current pattern, I have to call the match at the same index as the current pattern for their respective lists. Every list has an `.index()` method, and you can find the corresponding index number in the list for a given element passed to the method as an argument. So if I wanted *where* in `patterns` was the regex `r'[^ABC]'`, I could use `patterns.index(re.compile(r'[^ABC]'))`. This will return an `int`, which corresponds to the position of `r'[^ABC]'` in `patterns.`"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"print patterns.index(re.compile(r'[^ABC]'))"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"1\n"
]
}
],
"prompt_number": 26
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pre-defined character classes: shorthand"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"patterns2 = [re.compile(r'.'),\n",
"re.compile(r'\\w'),\n",
"re.compile(r'\\W'),\n",
"re.compile(r'\\d'),\n",
"re.compile(r'\\D'),\n",
"re.compile(r'\\n'),\n",
"re.compile(r'\\r'),\n",
"re.compile(r'\\t'),\n",
"re.compile(r'\\f'),\n",
"re.compile(r'\\s')]\n",
"\n",
"test_strings.append('Aardvarks belong to the Captain, capt_hook')\n",
"\n",
"for test_string in test_strings:\n",
" matches = [find_match(pattern, test_string) for pattern in patterns2]\n",
"\n",
" for pattern in patterns2:\n",
" print 'The first potential match for \"' + pattern.pattern + '\" in \"' + test_string + '\" is: ' + matches[patterns2.index(pattern)]"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"The first potential match for \".\" in \"Aardvarks belong to the Captain\" is: A\n",
"The first potential match for \"\\w\" in \"Aardvarks belong to the Captain\" is: A\n",
"The first potential match for \"\\W\" in \"Aardvarks belong to the Captain\" is: \n",
"The first potential match for \"\\d\" in \"Aardvarks belong to the Captain\" is: no match.\n",
"The first potential match for \"\\D\" in \"Aardvarks belong to the Captain\" is: A\n",
"The first potential match for \"\\n\" in \"Aardvarks belong to the Captain\" is: no match.\n",
"The first potential match for \"\\r\" in \"Aardvarks belong to the Captain\" is: no match.\n",
"The first potential match for \"\\t\" in \"Aardvarks belong to the Captain\" is: no match.\n",
"The first potential match for \"\\f\" in \"Aardvarks belong to the Captain\" is: no match.\n",
"The first potential match for \"\\s\" in \"Aardvarks belong to the Captain\" is: \n",
"The first potential match for \".\" in \"Albert's famous equation, E = mc^2.\" is: A\n",
"The first potential match for \"\\w\" in \"Albert's famous equation, E = mc^2.\" is: A\n",
"The first potential match for \"\\W\" in \"Albert's famous equation, E = mc^2.\" is: '\n",
"The first potential match for \"\\d\" in \"Albert's famous equation, E = mc^2.\" is: 2\n",
"The first potential match for \"\\D\" in \"Albert's famous equation, E = mc^2.\" is: A\n",
"The first potential match for \"\\n\" in \"Albert's famous equation, E = mc^2.\" is: no match.\n",
"The first potential match for \"\\r\" in \"Albert's famous equation, E = mc^2.\" is: no match.\n",
"The first potential match for \"\\t\" in \"Albert's famous equation, E = mc^2.\" is: no match.\n",
"The first potential match for \"\\f\" in \"Albert's famous equation, E = mc^2.\" is: no match.\n",
"The first potential match for \"\\s\" in \"Albert's famous equation, E = mc^2.\" is: \n",
"The first potential match for \".\" in \"Located at 455 Serra Mall.\" is: L\n",
"The first potential match for \"\\w\" in \"Located at 455 Serra Mall.\" is: L\n",
"The first potential match for \"\\W\" in \"Located at 455 Serra Mall.\" is: \n",
"The first potential match for \"\\d\" in \"Located at 455 Serra Mall.\" is: 4\n",
"The first potential match for \"\\D\" in \"Located at 455 Serra Mall.\" is: L\n",
"The first potential match for \"\\n\" in \"Located at 455 Serra Mall.\" is: no match.\n",
"The first potential match for \"\\r\" in \"Located at 455 Serra Mall.\" is: no match.\n",
"The first potential match for \"\\t\" in \"Located at 455 Serra Mall.\" is: no match.\n",
"The first potential match for \"\\f\" in \"Located at 455 Serra Mall.\" is: no match.\n",
"The first potential match for \"\\s\" in \"Located at 455 Serra Mall.\" is: \n",
"The first potential match for \".\" in \"Beware of the shape-shifters!\" is: B\n",
"The first potential match for \"\\w\" in \"Beware of the shape-shifters!\" is: B\n",
"The first potential match for \"\\W\" in \"Beware of the shape-shifters!\" is: \n",
"The first potential match for \"\\d\" in \"Beware of the shape-shifters!\" is: no match.\n",
"The first potential match for \"\\D\" in \"Beware of the shape-shifters!\" is: B\n",
"The first potential match for \"\\n\" in \"Beware of the shape-shifters!\" is: no match.\n",
"The first potential match for \"\\r\" in \"Beware of the shape-shifters!\" is: no match.\n",
"The first potential match for \"\\t\" in \"Beware of the shape-shifters!\" is: no match.\n",
"The first potential match for \"\\f\" in \"Beware of the shape-shifters!\" is: no match.\n",
"The first potential match for \"\\s\" in \"Beware of the shape-shifters!\" is: \n",
"The first potential match for \".\" in \"Aardvarks belong to the Captain, capt_hook\" is: A\n",
"The first potential match for \"\\w\" in \"Aardvarks belong to the Captain, capt_hook\" is: A\n",
"The first potential match for \"\\W\" in \"Aardvarks belong to the Captain, capt_hook\" is: \n",
"The first potential match for \"\\d\" in \"Aardvarks belong to the Captain, capt_hook\" is: no match.\n",
"The first potential match for \"\\D\" in \"Aardvarks belong to the Captain, capt_hook\" is: A\n",
"The first potential match for \"\\n\" in \"Aardvarks belong to the Captain, capt_hook\" is: no match.\n",
"The first potential match for \"\\r\" in \"Aardvarks belong to the Captain, capt_hook\" is: no match.\n",
"The first potential match for \"\\t\" in \"Aardvarks belong to the Captain, capt_hook\" is: no match.\n",
"The first potential match for \"\\f\" in \"Aardvarks belong to the Captain, capt_hook\" is: no match.\n",
"The first potential match for \"\\s\" in \"Aardvarks belong to the Captain, capt_hook\" is: \n"
]
}
],
"prompt_number": 32
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Matching character sequences\n"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"test_strings2 = [\"The Aardvarks belong to the Captain.\",\n",
" \"Bitter butter won't make the batter better.\",\n",
" \"Hark, the pitter patter of little feet!\"]\n",
"\n",
"patterns3 = [re.compile(r'Aa'),\n",
"re.compile(r'[Aa][Aa]'),\n",
"re.compile(r'[aeiou][aeiou]'),\n",
"re.compile(r'[AaEeIiOoUu][aeiou]'),\n",
"re.compile(r'[Tt]he'),\n",
"re.compile(r'^[Tt]he'),\n",
"re.compile(r'n.'),\n",
"re.compile(r'n.$'),\n",
"re.compile(r'\\W\\w'),\n",
"re.compile(r'\\w[aeiou]tter'),\n",
"re.compile(r'\\w[aeiou]tter'),\n",
"re.compile(r'..tt..')]\n",
"\n",
"for test_string in test_strings2:\n",
" matches = [find_match(pattern, test_string) for pattern in patterns3]\n",
"\n",
" for pattern in patterns3:\n",
" print 'The first potential match for \"' + pattern.pattern + '\" in \"' + test_string + '\" is: ' + matches[patterns3.index(pattern)]"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"The first potential match for \"Aa\" in \"The Aardvarks belong to the Captain.\" is: Aa\n",
"The first potential match for \"[Aa][Aa]\" in \"The Aardvarks belong to the Captain.\" is: Aa\n",
"The first potential match for \"[aeiou][aeiou]\" in \"The Aardvarks belong to the Captain.\" is: ai\n",
"The first potential match for \"[AaEeIiOoUu][aeiou]\" in \"The Aardvarks belong to the Captain.\" is: Aa\n",
"The first potential match for \"[Tt]he\" in \"The Aardvarks belong to the Captain.\" is: The\n",
"The first potential match for \"^[Tt]he\" in \"The Aardvarks belong to the Captain.\" is: The\n",
"The first potential match for \"n.\" in \"The Aardvarks belong to the Captain.\" is: ng\n",
"The first potential match for \"n.$\" in \"The Aardvarks belong to the Captain.\" is: n.\n",
"The first potential match for \"\\W\\w\" in \"The Aardvarks belong to the Captain.\" is: A\n",
"The first potential match for \"\\w[aeiou]tter\" in \"The Aardvarks belong to the Captain.\" is: no match.\n",
"The first potential match for \"\\w[aeiou]tter\" in \"The Aardvarks belong to the Captain.\" is: no match.\n",
"The first potential match for \"..tt..\" in \"The Aardvarks belong to the Captain.\" is: no match.\n",
"The first potential match for \"Aa\" in \"Bitter butter won't make the batter better.\" is: no match.\n",
"The first potential match for \"[Aa][Aa]\" in \"Bitter butter won't make the batter better.\" is: no match.\n",
"The first potential match for \"[aeiou][aeiou]\" in \"Bitter butter won't make the batter better.\" is: no match.\n",
"The first potential match for \"[AaEeIiOoUu][aeiou]\" in \"Bitter butter won't make the batter better.\" is: no match.\n",
"The first potential match for \"[Tt]he\" in \"Bitter butter won't make the batter better.\" is: the\n",
"The first potential match for \"^[Tt]he\" in \"Bitter butter won't make the batter better.\" is: no match.\n",
"The first potential match for \"n.\" in \"Bitter butter won't make the batter better.\" is: n'\n",
"The first potential match for \"n.$\" in \"Bitter butter won't make the batter better.\" is: no match.\n",
"The first potential match for \"\\W\\w\" in \"Bitter butter won't make the batter better.\" is: b\n",
"The first potential match for \"\\w[aeiou]tter\" in \"Bitter butter won't make the batter better.\" is: Bitter\n",
"The first potential match for \"\\w[aeiou]tter\" in \"Bitter butter won't make the batter better.\" is: Bitter\n",
"The first potential match for \"..tt..\" in \"Bitter butter won't make the batter better.\" is: Bitter\n",
"The first potential match for \"Aa\" in \"Hark, the pitter patter of little feet!\" is: no match.\n",
"The first potential match for \"[Aa][Aa]\" in \"Hark, the pitter patter of little feet!\" is: no match.\n",
"The first potential match for \"[aeiou][aeiou]\" in \"Hark, the pitter patter of little feet!\" is: ee\n",
"The first potential match for \"[AaEeIiOoUu][aeiou]\" in \"Hark, the pitter patter of little feet!\" is: ee\n",
"The first potential match for \"[Tt]he\" in \"Hark, the pitter patter of little feet!\" is: the\n",
"The first potential match for \"^[Tt]he\" in \"Hark, the pitter patter of little feet!\" is: no match.\n",
"The first potential match for \"n.\" in \"Hark, the pitter patter of little feet!\" is: no match.\n",
"The first potential match for \"n.$\" in \"Hark, the pitter patter of little feet!\" is: no match.\n",
"The first potential match for \"\\W\\w\" in \"Hark, the pitter patter of little feet!\" is: t\n",
"The first potential match for \"\\w[aeiou]tter\" in \"Hark, the pitter patter of little feet!\" is: pitter\n",
"The first potential match for \"\\w[aeiou]tter\" in \"Hark, the pitter patter of little feet!\" is: pitter\n",
"The first potential match for \"..tt..\" in \"Hark, the pitter patter of little feet!\" is: pitter\n"
]
}
],
"prompt_number": 33
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Matching character sequences"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"def find_all_matches(pattern, string):\n",
" matches = re.findall(pattern, string)\n",
" if matches:\n",
" return matches\n",
" else:\n",
" return None\n",
"\n",
"for test_string in test_strings2:\n",
" matches = [find_all_matches(pattern, test_string) for pattern in patterns3]\n",
" \n",
" for pattern in patterns3:\n",
" if matches[patterns3.index(pattern)]:\n",
" print 'All potential matches for \"' + pattern.pattern + '\" in \"' + test_string + '\" is/are: ' + ', '.join(matches[patterns3.index(pattern)])\n",
" else:\n",
" print 'There were no matches for \"' + pattern.pattern + '\" in \"' + test_string + '\".'\n"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"All potential matches for \"Aa\" in \"The Aardvarks belong to the Captain.\" is/are: Aa\n",
"All potential matches for \"[Aa][Aa]\" in \"The Aardvarks belong to the Captain.\" is/are: Aa\n",
"All potential matches for \"[aeiou][aeiou]\" in \"The Aardvarks belong to the Captain.\" is/are: ai\n",
"All potential matches for \"[AaEeIiOoUu][aeiou]\" in \"The Aardvarks belong to the Captain.\" is/are: Aa, ai\n",
"All potential matches for \"[Tt]he\" in \"The Aardvarks belong to the Captain.\" is/are: The, the\n",
"All potential matches for \"^[Tt]he\" in \"The Aardvarks belong to the Captain.\" is/are: The\n",
"All potential matches for \"n.\" in \"The Aardvarks belong to the Captain.\" is/are: ng, n.\n",
"All potential matches for \"n.$\" in \"The Aardvarks belong to the Captain.\" is/are: n.\n",
"All potential matches for \"\\W\\w\" in \"The Aardvarks belong to the Captain.\" is/are: A, b, t, t, C\n",
"There were no matches for \"\\w[aeiou]tter\" in \"The Aardvarks belong to the Captain.\".\n",
"There were no matches for \"\\w[aeiou]tter\" in \"The Aardvarks belong to the Captain.\".\n",
"There were no matches for \"..tt..\" in \"The Aardvarks belong to the Captain.\".\n",
"There were no matches for \"Aa\" in \"Bitter butter won't make the batter better.\".\n",
"There were no matches for \"[Aa][Aa]\" in \"Bitter butter won't make the batter better.\".\n",
"There were no matches for \"[aeiou][aeiou]\" in \"Bitter butter won't make the batter better.\".\n",
"There were no matches for \"[AaEeIiOoUu][aeiou]\" in \"Bitter butter won't make the batter better.\".\n",
"All potential matches for \"[Tt]he\" in \"Bitter butter won't make the batter better.\" is/are: the\n",
"There were no matches for \"^[Tt]he\" in \"Bitter butter won't make the batter better.\".\n",
"All potential matches for \"n.\" in \"Bitter butter won't make the batter better.\" is/are: n'\n",
"There were no matches for \"n.$\" in \"Bitter butter won't make the batter better.\".\n",
"All potential matches for \"\\W\\w\" in \"Bitter butter won't make the batter better.\" is/are: b, w, 't, m, t, b, b\n",
"All potential matches for \"\\w[aeiou]tter\" in \"Bitter butter won't make the batter better.\" is/are: Bitter, butter, batter, better\n",
"All potential matches for \"\\w[aeiou]tter\" in \"Bitter butter won't make the batter better.\" is/are: Bitter, butter, batter, better\n",
"All potential matches for \"..tt..\" in \"Bitter butter won't make the batter better.\" is/are: Bitter, butter, batter, better\n",
"There were no matches for \"Aa\" in \"Hark, the pitter patter of little feet!\".\n",
"There were no matches for \"[Aa][Aa]\" in \"Hark, the pitter patter of little feet!\".\n",
"All potential matches for \"[aeiou][aeiou]\" in \"Hark, the pitter patter of little feet!\" is/are: ee\n",
"All potential matches for \"[AaEeIiOoUu][aeiou]\" in \"Hark, the pitter patter of little feet!\" is/are: ee\n",
"All potential matches for \"[Tt]he\" in \"Hark, the pitter patter of little feet!\" is/are: the\n",
"There were no matches for \"^[Tt]he\" in \"Hark, the pitter patter of little feet!\".\n",
"There were no matches for \"n.\" in \"Hark, the pitter patter of little feet!\".\n",
"There were no matches for \"n.$\" in \"Hark, the pitter patter of little feet!\".\n",
"All potential matches for \"\\W\\w\" in \"Hark, the pitter patter of little feet!\" is/are: t, p, p, o, l, f\n",
"All potential matches for \"\\w[aeiou]tter\" in \"Hark, the pitter patter of little feet!\" is/are: pitter, patter\n",
"All potential matches for \"\\w[aeiou]tter\" in \"Hark, the pitter patter of little feet!\" is/are: pitter, patter\n",
"All potential matches for \"..tt..\" in \"Hark, the pitter patter of little feet!\" is/are: pitter, patter, little\n"
]
}
],
"prompt_number": 35
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have a new function and some new code. Let's go over it:\n",
"\n",
"First, I wrote a function called `find_all_matches`:\n",
" \n",
" def find_all_matches(pattern, string):\n",
" matches = re.findall(pattern, string)\n",
" if matches:\n",
" return matches\n",
" else:\n",
" return None\n",
"\n",
"There are only two differences between `find_matches` and `find_all_matches`. First, `find_all_matches` uses `re.findall` not `re.search`. So matches is a list of all possible matches. Thus, instead of return a single string in either condition, `find_all_matches` can return either a list of strings or `None`.\n",
"\n",
" for test_string in test_strings2:\n",
" matches = [find_all_matches(pattern, test_string) for pattern in patterns3]\n",
" \n",
" for pattern in patterns3:\n",
" if matches[patterns3.index(pattern)]:\n",
"\n",
"Remember the use of `.index()` from the previous code walkthrough. Also, remember that `None` returns false in a logical condition test. In this `if` statement, I'm testing to see if there were any matches for the current pattern in the loop. If there were any matches, the code will execute the next line. Otherwise, it will go to the `else` block. \n",
" \n",
" print 'All potential matches for \"' + pattern.pattern + '\" in \"' + test_string + '\" is/are: ' + ', '.join(matches[patterns3.index(pattern)])\n",
" \n",
"If `matches` at the index of the current pattern is not `None`, it will be a list of strings. Because I'm printing these results, I wanted to nicely format them for diagnostic purposes. So we use the standard list-to-string Python expression of `''.join(list)`. In this case, I wanted the results to be comma-separated. \n",
" \n",
" else:\n",
" print 'There were no matches for \"' + pattern.pattern + '\" in \"' + test_string + '\".'\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Quantification and grouping"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"test_strings3 = ['Now Mr. N said, \"Nooooooo!\"',\n",
" 'Then she told him he had to be quiet.']\n",
"\n",
"patterns4 = [re.compile(r'No*'),\n",
"re.compile(r'No+'),\n",
"re.compile(r'No?'),\n",
"re.compile(r'No{7}'),\n",
"re.compile(r's?he'),\n",
"re.compile(r'(she|he)')]\n",
"\n",
"for test_string in test_strings3:\n",
" matches = [find_all_matches(pattern, test_string) for pattern in patterns4]\n",
" \n",
" for pattern in patterns4:\n",
" if matches[patterns4.index(pattern)]:\n",
" print 'All potential matches for \"' + pattern.pattern + '\" in \"' + test_string + '\" is/are: ' + ', '.join(matches[patterns4.index(pattern)])\n",
" else:\n",
" print 'There were no matches for \"' + pattern.pattern + '\" in \"' + test_string + '\".'"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"All potential matches for \"No*\" in \"Now Mr. N said, \"Nooooooo!\"\" is/are: No, N, Nooooooo\n",
"All potential matches for \"No+\" in \"Now Mr. N said, \"Nooooooo!\"\" is/are: No, Nooooooo\n",
"All potential matches for \"No?\" in \"Now Mr. N said, \"Nooooooo!\"\" is/are: No, N, No\n",
"All potential matches for \"No{7}\" in \"Now Mr. N said, \"Nooooooo!\"\" is/are: Nooooooo\n",
"There were no matches for \"s?he\" in \"Now Mr. N said, \"Nooooooo!\"\".\n",
"There were no matches for \"(she|he)\" in \"Now Mr. N said, \"Nooooooo!\"\".\n",
"There were no matches for \"No*\" in \"Then she told him he had to be quiet.\".\n",
"There were no matches for \"No+\" in \"Then she told him he had to be quiet.\".\n",
"There were no matches for \"No?\" in \"Then she told him he had to be quiet.\".\n",
"There were no matches for \"No{7}\" in \"Then she told him he had to be quiet.\".\n",
"All potential matches for \"s?he\" in \"Then she told him he had to be quiet.\" is/are: he, she, he\n",
"All potential matches for \"(she|he)\" in \"Then she told him he had to be quiet.\" is/are: he, she, he\n"
]
}
],
"prompt_number": 8
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Capturing groups\n",
"\n",
"In Python, `SRE_Match` objects have `.groups` and `.group` methods. These correspond to the capturing groups established in the regex, if you chose to indicate groups. By default, the 0th group is the entire match to the whole regex. To access the result for a capturing group, you pass the capturing group index to the `.group` method. "
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"test_strings4 = ['The benefit is being held for Mr. Kite and Mr. Henderson.',\n",
" 'Tickets cost $5.00 for adults, $3.50 for children.',\n",
" 'Over 9000 attendees are expected, up from 900 attendees last year.',\n",
" 'Over 9,000 attendees are expected, up from 900 attendees last year.']\n",
"\n",
"patterns5 = [re.compile(r'Mr\\. (\\w+)'),\n",
"re.compile(r'\\$(\\d+\\.\\d\\d)'),\n",
"re.compile(r'(\\d+) attendees'),\n",
"re.compile(r'((\\d+,)*\\d+) attendees')]"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 37
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"# simple example\n",
"\n",
"matches = re.search(patterns5[3], test_strings4[3])\n",
"print 'Group 0: ' + matches.group(0)\n",
"print 'Group 1: ' + matches.group(1)\n",
"print 'Group 2: ' + matches.group(2)\n",
"#print 'Group 3: ' + matches.group(3) # what happens if you uncomment this?"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Group 0: 9,000 attendees\n",
"Group 1: 9,000\n",
"Group 2: 9,\n"
]
}
],
"prompt_number": 10
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This example searched for r'((\\d+,)*\\d+) attendees' in the string \"Over 9000 attendees are expected, up from 900 attendees last year.'\" There are two groups, one nested inside the other. Groups are indexed outer-most left parens. This is why Group 1 is `9,000` and Group 2 is `9,`."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"for test_string in test_strings4:\n",
" for pattern in patterns5:\n",
" for result in re.finditer(pattern, test_string):\n",
" for i in range(pattern.groups+1):\n",
" \n",
" print 'In \"' + test_string + '\", ' + 'given pattern \"' + pattern.pattern + '\", the group ' +str(i)+ ' match is ' + str(result.group(i))"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"In \"The benefit is being held for Mr. Kite and Mr. Henderson.\", given pattern \"Mr\\. (\\w+)\", the group 0 match is Mr. Kite\n",
"In \"The benefit is being held for Mr. Kite and Mr. Henderson.\", given pattern \"Mr\\. (\\w+)\", the group 1 match is Kite\n",
"In \"The benefit is being held for Mr. Kite and Mr. Henderson.\", given pattern \"Mr\\. (\\w+)\", the group 0 match is Mr. Henderson\n",
"In \"The benefit is being held for Mr. Kite and Mr. Henderson.\", given pattern \"Mr\\. (\\w+)\", the group 1 match is Henderson\n",
"In \"Tickets cost $5.00 for adults, $3.50 for children.\", given pattern \"\\$(\\d+\\.\\d\\d)\", the group 0 match is $5.00\n",
"In \"Tickets cost $5.00 for adults, $3.50 for children.\", given pattern \"\\$(\\d+\\.\\d\\d)\", the group 1 match is 5.00\n",
"In \"Tickets cost $5.00 for adults, $3.50 for children.\", given pattern \"\\$(\\d+\\.\\d\\d)\", the group 0 match is $3.50\n",
"In \"Tickets cost $5.00 for adults, $3.50 for children.\", given pattern \"\\$(\\d+\\.\\d\\d)\", the group 1 match is 3.50\n",
"In \"Over 9000 attendees are expected, up from 900 attendees last year.\", given pattern \"(\\d+) attendees\", the group 0 match is 9000 attendees\n",
"In \"Over 9000 attendees are expected, up from 900 attendees last year.\", given pattern \"(\\d+) attendees\", the group 1 match is 9000\n",
"In \"Over 9000 attendees are expected, up from 900 attendees last year.\", given pattern \"(\\d+) attendees\", the group 0 match is 900 attendees\n",
"In \"Over 9000 attendees are expected, up from 900 attendees last year.\", given pattern \"(\\d+) attendees\", the group 1 match is 900\n",
"In \"Over 9000 attendees are expected, up from 900 attendees last year.\", given pattern \"((\\d+,)*\\d+) attendees\", the group 0 match is 9000 attendees\n",
"In \"Over 9000 attendees are expected, up from 900 attendees last year.\", given pattern \"((\\d+,)*\\d+) attendees\", the group 1 match is 9000\n",
"In \"Over 9000 attendees are expected, up from 900 attendees last year.\", given pattern \"((\\d+,)*\\d+) attendees\", the group 2 match is None\n",
"In \"Over 9000 attendees are expected, up from 900 attendees last year.\", given pattern \"((\\d+,)*\\d+) attendees\", the group 0 match is 900 attendees\n",
"In \"Over 9000 attendees are expected, up from 900 attendees last year.\", given pattern \"((\\d+,)*\\d+) attendees\", the group 1 match is 900\n",
"In \"Over 9000 attendees are expected, up from 900 attendees last year.\", given pattern \"((\\d+,)*\\d+) attendees\", the group 2 match is None\n",
"In \"Over 9,000 attendees are expected, up from 900 attendees last year.\", given pattern \"(\\d+) attendees\", the group 0 match is 000 attendees\n",
"In \"Over 9,000 attendees are expected, up from 900 attendees last year.\", given pattern \"(\\d+) attendees\", the group 1 match is 000\n",
"In \"Over 9,000 attendees are expected, up from 900 attendees last year.\", given pattern \"(\\d+) attendees\", the group 0 match is 900 attendees\n",
"In \"Over 9,000 attendees are expected, up from 900 attendees last year.\", given pattern \"(\\d+) attendees\", the group 1 match is 900\n",
"In \"Over 9,000 attendees are expected, up from 900 attendees last year.\", given pattern \"((\\d+,)*\\d+) attendees\", the group 0 match is 9,000 attendees\n",
"In \"Over 9,000 attendees are expected, up from 900 attendees last year.\", given pattern \"((\\d+,)*\\d+) attendees\", the group 1 match is 9,000\n",
"In \"Over 9,000 attendees are expected, up from 900 attendees last year.\", given pattern \"((\\d+,)*\\d+) attendees\", the group 2 match is 9,\n",
"In \"Over 9,000 attendees are expected, up from 900 attendees last year.\", given pattern \"((\\d+,)*\\d+) attendees\", the group 0 match is 900 attendees\n",
"In \"Over 9,000 attendees are expected, up from 900 attendees last year.\", given pattern \"((\\d+,)*\\d+) attendees\", the group 1 match is 900\n",
"In \"Over 9,000 attendees are expected, up from 900 attendees last year.\", given pattern \"((\\d+,)*\\d+) attendees\", the group 2 match is None\n"
]
}
],
"prompt_number": 46
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Before we go over this code block, let's establish the purpose of the code. I wanted to return all the matches for each group. But there are a few concerns:\n",
"\n",
"1. The *number* of groups is different for each pattern. So I can't hardcode the number of times to loop over. In other words, the number of times my loop should iterate has to be *dynamically* assigned, conditioning on *which regex pattern* is the comparison regex in the loop.\n",
"2. `.findall' return a list of matches, and if there are groups, it will return a list of tuples, where each tuple is the length of the number of capturing groups."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"matches = re.findall(patterns5[3], test_strings4[3])\n",
"matches"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 47,
"text": [
"[('9,000', '9,'), ('900', '')]"
]
}
],
"prompt_number": 47
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can refer to the index of a tuple within a list of tuples through indexing a second index:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"matches[0][0]"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "pyout",
"prompt_number": 48,
"text": [
"'9,000'"
]
}
],
"prompt_number": 48
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"But there are other ways of constructing this kind of loop."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" for test_string in test_strings4:\n",
" for pattern in patterns5:\n",
" for result in re.finditer(pattern, test_string):\n",
" \n",
"`re.finditer` returns an iterator, which is a new Python concept to you. This loop means that for every pattern and for each string we're testing, instead of creating a list of matches, we're going to create a iterator object that contains the results.\n",
" \n",
" for i in range(pattern.groups+1):\n",
"\n",
"The `.groups` method will list the number of capturing groups in the regular expression. `range` is a function that will return a list of integers ranging from a start or a stop value and by a step value. If you just give it a int, by default it will treat that value is a stopping value and start from 0. Now, we add 1 to this value because the end point is omitted in `range`. If we want to return *all* the groups, we have to add that end point back.\n",
"\n",
" print 'In \"' + test_string + '\", ' + 'given pattern \"' + pattern.pattern + '\", the group ' +str(i)+ ' match is ' + str(result.group(i))\n",
"\n",
"Because `i` is established as the index value of the current regex match produced by the iterator, we can use `i` as the index value for which group we'd like to return. That's why we can call `result.group(i)`. \n",
"\n",
"In no way was this the *only* way to accomplish this task! I wanted to show you a few different functions in this tutorial, as well as introduce you to the more examples where typical \"Pythonic\" code constructions are useful, such as list comprehensions and `join`. There are many ways of replicating all of these diagnostic printout examples."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Final exercise\n",
"\n",
"Let's see how much you've learned. We're going to give you three strings that have a phone number in them. Your job is to write a regex that will return the full form of all of them."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"phone_strings = ['Call Empire Carpets at 588-2300',\n",
"'Does Jenny live at 867 5309?',\n",
"'You can reach Mr. Plow at 636-555-3226']"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 12
}
],
"metadata": {}
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment