Regular expressions are : › A language or syntax that lets you specify patterns for matching e.g. filenames or strings › Used to identify the files

Introduction to Regular ExpressionsChristine Moulen

MIT LibrariesELUNA 2014

What is a regular expression?

Regular expressions are :› A language or syntax that lets you specify

patterns for matching e.g. filenames or strings

› Used to identify the files or lines you want to work with

› Used inside of substitution functions to change the contents of a string

Command line examples

ls 14* › * is a wildcard here, not regex› 14 followed by zero or more of any character

ls 14[0-1][0-9]* › [0-1] and [0-9] are regex character classes,

specifying a single character within the the list of characters from 0 to 1, and 0 to 9, respectively

ls 14[0-1][0-9][0-3][0-9]*› 6 digits that look like a date YYMMDD, mostly

More command line examples

mv [b-z]* $data_scratch› An alphabetical class, which depending on

your system might match the lower case letters from b through z, OR a mix of upper and lower case: b C c D d ... Z z

grep 'MIT01$' sysnos.txt› Find lines that end ($) with MIT01› ^ can be used to match at the beginning

of a line

UNIX/Linux editors

In vi, you can use regular expressions with the s/// substitution operator

With emacs, use M-x query-replace-regexp› Replace $ with MIT01› Take a list of system numbers and make it

valid input to an Aleph service by adding the library code to the end of each line

Matching example in Perl

Look through a MARC file in Aleph sequential format for lines with tag 260› 001234567 260 L $$aCambridge$$bMIT Press

if ($matched =~ m/^\d{9}\s260.+/) { ... }› $matched is the while loop variable

representing the line we're working on› =~ is a pattern operator used with the

matching (m), substitution (s), and translation (tr) functions

› m// is the pattern matching function

m/^\d{9}\s260.+/

^ start at the beginning of the line \d Perl-speak for the digits character

class {9} a quantifier. Find exactly 9 of \d \s Perl-speak for the whitespace char

class 260 the MARC tag I'm looking for . any character + a quantifier. Find 1 or more of .

m/^\d{9}\s260.+/

^ start at the beginning of the line

\d Perl-speak for the digits character class

{9} a quantifier. Find exactly 9 of \d

\s Perl-speak for the whitespace char class

260 the MARC tag I'm looking for

. any character

+ a quantifier. Find 1 or more of .

Working with MARC fields

Look for deleted records › LDR position 05 is d› $my_LDR =~ /LDR L .....d/

Look for e-resource records› $my_245 =~ /\$\$h\[electronic resource\]/

Look for OCLC numbers› $my_035 =~ /($OCoLC$\d{8,10})/› Note the double use of () here

Counting up records at the end of a script

if ($hash{$tmp} =~ m/SKIP/ || $hash{$tmp} =~ m/NEW/) { $new_count++ if (m/ FMT L /); $skip_count++ if (m/ FMT L / && $hash{$tmp} =~

m/SKIP/); $bre_count++ if (m/ FMT L / && $hash{$tmp} =~ m/SKIP

Brief/); $bks_count++ if (m/ FMT L / && $hash{$tmp} =~ m/SKIP

Books24x7/); $eebo_count++ if (m/ FMT L / && $hash{$tmp} =~

m/SKIP EEBO/); $epda_count++ if (m/ FMT L / && $hash{$tmp} =~

m/SKIP EPDA/); $sta_count++ if (m/ FMT L / && $hash{$tmp} =~ m/SKIP

STA/);}

Substitution example in Perl

We have a browse index of URLs An Aleph browse index only sorts the

first 69 characters of the field When we have many URLs from the

same site, we need to get the unique part closer to the beginning

Following is an SFX OpenURL from the MARCit! service

This OpenURL ...

http://owens.mit.edu/sfx_local? url_ver=Z39.88-2004&ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&rfr_id=info:sid/sfxit.com:opac_856&url_ctx_fmt=info:ofi/fmt:kev:mtx:ctx&sfx.ignore_date_threshold=1&rft.object_id=3710000000092335&svc_val_fmt=info:ofi/fmt:kev:mtx:sch_svc&

... becomes this.

http://owens.mit.edu/sfx_local?rft.object_id=3710000000092335&url_ver=Z39.88-2004&ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&rfr_id=info:sid/sfxit.com:opac_856&url_ctx_fmt=info:ofi/fmt:kev:mtx:ctx&sfx.ignore_date_threshold=1&svc_val_fmt=info:ofi/fmt:kev:mtx:sch_svc&

The substitution expression

$my_856 =~ s/(^.*sfx_local\?)(.*)(rft\.object_id\=\d{1,}\&)(.*$)/$1$3$2$4/;

s is the substitution operator› substitute/this/for this/

Parentheses used here to group different sections of the pattern, and then re-arrange them

s/(^.*sfx_local\?)(.*)(rft\.object_id\=\d{1,}\&)(.*$)/$1$3$2$4/

$1 The first matched parenthetical section

^.*sfx_local\? From the beginning, anything up to and including sfx_local?

? is a special character and is escaped here to get a literal question mark

$2 The 2nd matched parenthetical section

.* Any number of any character, until it reaches the next match string

s/(^.*sfx_local\?)(.*)(rft\.object_id\=\d{1,}\&)(.*$)/$1$3$2$4/

Now change the order from $1$2$3$4 to $1$3$2$4

$3 The 3rd parenthetical section

rft\.object_id\=\d{1,}\&

rft.object_id= followed by one or more digits and an ampersand

. = and & are escaped with \ because they are special characters

{1,} is like + a quantifier meaning one or more

$4 The 4th and final parenthetical section

.*$ Any number of any character to the end

Parsing thesis notes

Thesis degree, year, and department are stored in a single free text MARC field 502

We have applied some structure to this, but it has varied over time

In DSpace, we want to get these 3 bits into separate fields, so the note is parsed on the way from MARC to Dublin Core


$MIT = 'Massachusetts Institute of Technology\.?|M\.\s?I\.\s?T\.';› ? is the zero or one quantifier. › | match the pattern alternative before or

after this $Dept = '[Dd]epartment\s[Oo]f|

[dD]ept\.\s+[Oo]f';› A few small character classes, to allow for

case variation, and Department vs Dept.


$Month = 'January|February|March|April|May|June|July|August|September|October|November|December';› match any one month name when $Month

is used inside a pattern

Thesis. 1975. Sc.D.--Massachusetts Institute of Technology. Dept. of Mechanical Engineering

/^Thesis\.\s+(\d+)\.?\s+([\w\.\s]+)--($MIT)\.?\s+($Dept)?\s*(.+)$/o

/^Thesis\. Begin with Thesis.

\s+ 1 or more spaces

(\d+) 1 or more digits = $1

\.? 0 or 1 period


([\w\.\s]+) 1 or more word chars, periods, spaces = $2

-- --

($MIT) something matching $MIT = $3

Thesis. 1975. Sc.D.--Massachusetts Institute of Technology. Dept. of Mechanical Engineering

/^Thesis\.\s+(\d+)\.?\s+([\w\.\s]+)--($MIT)\.?\s+($Dept)?\s*(.+)$/o

\.? 0 or 1 period


($Dept)? 0 or 1 strings matching $Dept = $4

\s* 0 or more spaces

(.+)$ anything left to the end = $5

/o An option. Compile the expression only once. The variables, $MIT and $Dept are not going to change

More thesis examples

Massachusetts Institute of Technology. Dept. of Economics. Thesis. 1968. Ph.D.

Massachusetts Institute of Technology, Dept. of Civil Engineering, Thesis. 1965. Sc. D.

/^($MIT)(\.|,)?\s+($Dept)?\s*([\w\s\.,]+)\s+Thesis.\s*(\d{4})\.?\s*(.*)$/o


Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Aeronautics and Astronautics, 1973.

Thesis (Sc. D.)--Massachusetts Institute of Technology, Dept. of Aeronautics an Astronautics.

Thesis. (M.S.)--Sloan School of Management, 1983. Thesis (Sc. D.)--Massachusetts Institute of Technology,

Dept. of Mechanical Engineering, 1951. Thesis (Ph. D.)--Massachusetts Institute of Technology,

Dept. of Linguistics and Philosophy, February 2004.

/^Thesis\.?\s*$([^$]*)\)(\s*--?\s*|\s+)?(($MIT)[\.,]?)?\s*($Dept)?\s*(.*)(,\s+(\d{4}))?\.?$/o


Thesis (Ph. D.)--Joint Program in Oceanography/Applied Ocean Science and Engineering (Massachusetts Institute of Technology, Dept. of Earth, Atmospheric, and Planetary Sciences; and the Woods Hole Oceanographic Institution), 2013.

/^Thesis\.?\s*$([^$]*)\)(\s*--(Joint Program in ([\w\.\s]+)$(($MIT)[\.,]?)?\s*($Dept)?\s*([\w,;\s]+)$))(,\s+(\d{4}))?\.?$/o

Questions?

[email protected]

Documents

Regular expressions are : › A language or syntax that lets you specify patterns for matching e.g. filenames or strings › Used to identify the files