Spot the difference!

001I don’t know if you can recall these games from when you were a kid?  I used to spend hours trying to find all the differences between the image on the left and the one on the right.  I never once thought how that might become a useful skill in later life… although in some cases it’s a skill I’d rather not have to develop!

You may be wondering where I’m going with this so I’ll explain.  Last weekend the SFÖ held a conference in Umeå, Sweden… I wasn’t there, but I did get an email from one of my colleagues asking how you could see what changes had been made in your bilingual files as a result of post-editing Machine Translation.  The easy answer of course is to do the post-editing with your track changes switched on, then it’s easy to spot the difference.  That is useful, but it’s not going to help with measurement, or give you something useful to be able to discuss with your client.  It’s also not going to help if you didn’t work with tracked changes in the first place because you’d need some serious spot the difference skills to evaluate your work!

Now, I’ve discussed this sort of thing, how to be productive with MT and ways to measure it, in great detail in the past in these articles:

So why talk about them again?  Well, I think in the past I went into so much detail I think I probably frightened many people off and they may not realise just how simple this can be.  So I’m going to very quickly look at how to measure your effort when you do post-edit work using two different plugins and explain the difference between them.  To do this I have a simple text in German which I ran through a machine translation engine and then post-edited it without track changes on.  I then have two versions for a bit of “spot the difference” where you can see from the AT status that I edited all but the first two segments:

002

I’ll start with Post-Edit Compare.

Post-Edit Compare

This plugin is available on the SDL AppStore and the developer, Patrick Hartnett, has versions for Studio 2009, 2011, 2014 and 2015.  I’m only going to look at 2015 where it also has the additional advantage of being nicely integrated into the product as you’ll see in the video.  It’s only about 7-minutes long and worth a watch to see how the plugin can be used to compare the differences between the two files and provide an analysis to summarise the effort you had to use:

Next I’ll take a quick look at Qualitivity.

Qualitivity

This plugin is also available on the SDL AppStore and the developer, also Patrick Hartnett, has versions for Studio 2014 and 2015.  I’m only looking at Studio 2015 again, but the principle is exactly the same for both versions.  The video is also about 7-minutes long and shows a small part of the features that are actually available with this tool, specifically related to the comparison we’re looking for here.  Of the two plugins this is my favourite simply because it is completely automated and you don’t actually have to compare anything.  It is tracking everything you do instead of simply comparing files, although this does mean there are interesting differences between the tools and why you would choose one over the other (see the summary at the end):

Summary

Both of these plugins do a really good job of quantifying the changes you have had to make to turn a machine translated file into a polished and customer ready target language file.  But there are differences between them that are worth knowing so you can use the one most appropriate for your needs.  I’ll just discuss these a little here to round off the article:

  1. If you prepare machine translated files and send them off for post-editing then Post-Edit Compare may be the easiest plugin to use.  You can save each version of the project as a new Post-Edit Project and easily compare between any two versions you like.  You also have the ability to easily restore your Project to whichever version you like.
  2. Qualitivity does not compare two files.  It tracks everything you do while post-editing and allows the post-editor to generate extremely rich reports that can be used for post-edit analysis.  But as a Project Manager you would need to ask the Post-Editor for these reports in order to be able to see the changes yourself.
  3. If you are doing academic research, or analysing Post-Editing techniques then Qualitivity is the tool for you.
    1. It records the data in the background quietly without interfering with the work of the post-editor at all
    2. It tracks everything the Post-Editor is doing within Studio, and only within Studio, and this can be very useful.  Both tools calculate a similar PEM (Post Edit Modification) score but Qualitivity will show you what steps the Post-Editor had to make in order to get to the post-edited version, for example:
      003
      If you look closely you’ll see the differences.  Qualitivity is actually track changes based on what you actually did and how you edited the segment.  Post-Edit Compare shows you the changes, but without any knowledge of how the changes were actually made keystroke by keystroke.  Qualitivity can capture these keystrokes and how long it took to make them!

In a nutshell these are the differences as far as quantifying Post-Editing in its simplest form is concerned.  Both tools are excellent and very simple to use.  I hope this article and the videos demonstrate this, but if you want more details don’t forget you can always investigate these articles to learn more:

3 comments
  1. Slightly off-topic (sorry!), but your article has brought an old grievance back to my mind.
    I wish the translation results window in Studio, which gives you a side-by-side comparison between a match from the TM and the active segment, would show differences as clearly as Qualitivity does above, instead of just shoving one deleted word between two new ones as in the Post-Edit-Compare example, which makes the whole thing more or less unreadable and useless.
    There has been an entry on ideas.sdl.com addressing this issue (http://ideas.sdl.com/ideas/detail.asp?i=3971) for a few years now, but sadly it has been ignored by SDL. The fix would be easy: make several crossed out words in a row be followed by several new inserted words, instead of one word deleted, one word inserted and so on.
    Other than that, Studio has become better with every version and I use it all the time, except I don’t look at the results window any more, as I find the way differences are shown to be confusing.

    Like

    • Thanks Alexander. I agree with you on the markup and would even prefer to see the old Trados markup back. I think it was much easier to see at a glance than the track changes, even when it’s written in a more usable way. We did look at creating an app for this to allow more customisation, but it’s not possible without completely replacing the current TM Results window and the TM Provider. So a lot of work. Maybe this is why the core development team have not tackled it yet.

      Like

      • So not as easy as I thought… OK, thanks for your replay anyway!

        Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: