Regex… and "economy of accuracy" (Regular Expressions – Part 2)

In Regular Expressions – Part 1 I wrote a summary of where regular expressions could be used in SDL Trados Studio, and I covered a couple of examples.  I also referred to RegexBuddy quite a lot as this is a really useful tool in helping you write and understand regular expressions.  But in case learning another application is something you don’t want to do I thought it would be handy to go through what I think are the most useful applications of regular expressions for every day use in SDL Trados Studio, and also share a few tips on how to use Studio to verify the expressions are finding what you need as well as introduce a little “economy of accuracy“.
So first of all, and an important thing to note that takes away much of the mystery of regex is to look at what is actually happening.  It is basically a search tool, so if I wanted to find the word “multifarious” using regex amongst these sentences:

I don’t have to do anything complicated, I simply put “multifarious” into the display filter like this:

This is because all regex does is to search systematically through the content of each segment looking for a match.  So in this case it looks for the letter m, then when it finds it it looks for the letter u and so on until it matched the whole word.  If it failed at any point then the display filter would not display the segment.
Now, the interesting thing about regex is that it also comes with a number of special patterns you can use to make your search more powerful.  Let’s start with something called an anchor.  There are two anchors, the ^ (caret) symbol and the $ (dollar) symbol.  These symbols don’t match anything at all, they just tell Studio where to anchor the position of the search.  So for example, if I put a caret at the start I see this where segment #43 is no longer visible:

This is because the caret says, start searching from the start only.  Segment #43 starts with an I so the search fails immediately.  If I put a dollar at the end instead I see this:

This time segment #41 is not visible, but #43 is.  This is because now Studio is only looking for the case where the word multifarious is the last thing in the segment.  If I wanted to see only segments that contained multifarious and nothing else I could use the caret and the dollar at the same time, now I see this:

That was pretty easy… but what happens if I want to find something containing the caret symbol?  This is a good time to introduce the range of characters that have a special meaning and need to be escaped in the regex expression if you want to find the literal character in the text.  There are eleven characters that have a special meaning, and once you understand them regular expressions start to take on a whole new meaning… and almost addictive quality (… or maybe that’s going a little too far ;-)):
^  $  [    .  |  ?  *  +  (  )
I’m not going to cover what all of these are for in this blog, but it’s important to understand that you have to escape them if you are looking for them as literal text.  To escape them you simply place a backslash () before the character in the regex expression.  So for example, if I wanted to find $2.45 in a list like this $2.67, $2.72, $3.37, $2.45, $2.67, $2-45, $2.92, $2145, $2,45 and I just type in the literal characters I get this no match:

The first reason is because the dollar means anchor at the end of the string and then start looking.  Of course there is nothing after the end so the search pattern can’t find anything.  So I have to escape this special character and now I see this:

This time  I have a few results, but still not what I need.  This is because the dot is a special character that means match anything at all, apart from a line feed character (or hard break).  So it matches the dot, the hyphen, the one and the comma.  So to be precise I need to escape the dot as well like this:

You probably see these backslashes used a lot in regular expressions and they can have the effect of confusing the heck out of you..!  But hopefully after reading this you’ll see it’s not so hard and when used for the simple sort of search operations, that are still extremely powerful and useful, you might do in Studio they’re quite easy to learn as well.
Another thing you might hear people referring to are “character classes/sets”.  These are used to match only one out of several options.  So for example, if you wanted to be able to find all segments containing the word localisation, or where it was spelled localization, you can do this using a single search containing a character set:
locali[sz]ation
So by putting the square brackets (note the opening square bracket was also a special character so needs to be escaped if you want to search for the opening square bracket as a literal text) around the letters s and z within the word Studio knows it should look for words that contain either s or z but not both.

A quick tip here is that Studio can be quite helpful when you are trying to get your regex expression perfected.  If you use the search facility rather than the display filter then the specific terms that are found are highlighted in the editor and this helps you to make sure all the things you want are picked up correctly:

You then hit the “Find Next” button and get this:

You can quite quickly see whether the search is going to find the correct things and then put it back into the display filter if you trying to filter rather than find specific terms.
This ability to use a character class like this might be quite useful as a QA check for example (another place you can use regular expressions in Studio).  You can build up expressions of your own that check the document for commonly mispelled words, or simply to help ensure consistency throughout your document, before or after translation.  “Before” is quite useful as you have an opportunity to make sure the correct translation will be used where the mispelling could lead to to subtle changes in the meaning:

Perhaps not the greatest example of a QA check, but I wanted to illustrate the possibility with the same expression we just learned… it would then show up in Studio as you worked like this so you knew there was a possible misspelling to be aware of through the small warning symbol that is shown in the status column:

Another useful thing about character sets are what ‘s known as “shorthand” character sets.  So for example, if you wanted to find any number between 0 and 9 you can use this:
[0-9]
Or if you wanted to find letters, you can use these:
[a-z]
[A-Z]… or even
[a-zA-Z]
Where the first one looks for any letter between a to z as a lowercase letter.  The second looks for uppercase letters, and the third doesn’t care whether they are uppercase or lowercase.  You have the same concept for numbers like this:
[0-9] [2-5] [3-8]
Here the regex expression will try to match a number within the range specified, so 0 to 9, 2 to 5 or 3 to 8.  These two patterns for finding any number or letter are very useful and well worth becoming familiar with them.  If you are looking for Product codes for example that follow a particular pattern then you can use these to build up the patterns.  For example I could do this:

In in reality this is not a bulletproof regex because it will find anything that starts with a capital letter followed by a number.  But showing you this as an example also allows me to illustrate another useful point.  As a Translator using Studio the phrase “economy of accuracy” is a good one to adopt.  We’re not writing computer code here that needs to be very specific… we are just trying to use regex to make our lives a little easier and find the things we need to address.  So in this example, based on all the other content of my document this simple expression was enough to find exactly what I needed.  So why would I spend time trying to write something like this that would also achieve the same result?
[A-Z]d{2}[A-Z]+d{1}.d{1}
So, “economy of accuracy” is a good term to remember.  But also a word of caution… think of the consequences when being economic.  If you are filtering or just searching then this principle applies perfectly.  If you are replacing text throughout an entire document then the more accurate you can be the better… in fact when you are replacing you may have to be accurate depending on what you are doing.  But I’ll cover search and replace in part 3.
To finish this article off I want to mention another useful expression I used in the more complicated regex above:
d{2}
The d is another way of writing [0-9].  Technically they mean different things, but for our purposes they do the same thing.  The 2 in curly brackets means find me the number exactly 2 times.  You can also use something like this:
d{2,4}
Match a number between 2 and 4 times.  Now these are really useful for handling number based patterns like IP addresses and dates that can be autolocalised by Studio when you don’t want them to be.  So the idea would be that you find the segments with these numbers in them, lock them and safely prevent them from being changed.  So consider the following where I used d{3}.

Clearly this regex isn’t accurate enough yet to serve my purpose because I only want to see segments that only contain IP addresses.  But by inspecting at the content of this file it looks as though I can resolve the problem with a little “economy of accuracy” by adding a caret to the start:

I can now easily lock all of these segments in one go and then translate without worrying about them being automatically translated incorrectly because dots have become commas or spaces for example depending on the language I am translating into.  Certainly this is easier than using an expression like this where the checks also validate the IP numbers themselves… as a translator or project manager you may not care whether they are valid or not..!:
^(?:(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9]).){3}(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])$
There are many more things to learn about using regular expressions in Studio, and if you do find them interesting and have other applications where you can use them then there are many more things to learn generally..!  But I hope I have managed to achieve my main objective of removing the wizardry a little, explaining “economy of accuracy” and how this principle will allow you to have a go with regex in Studio yourself and become more productive as a result.
There are a couple of other articles that are worth reviewing;
Regex….. what regex..! (Regular Expressions – Part 1)
Search and replace with Regex in Studio – Regular Expressions Part 3
DOGS and CATS… Regular Expressions Part 4!