Qualitivity… measuring quality and productivity

In the last year or so I’ve had the pleasure of watching Patrick Hartnett use the SDL OpenExchange (now RWS AppStore) APIs and SDK to develop SDLXLIFF Compare, then Post-Edit Compare, the Studio Timetracker and a productivity tool that combined all of the first three into one and introduced a host of productivity metrics and a mechanism for scoring the quality of a translation using the Multidimensional Quality metrics (MQM) framework.  This last application was never released, not because it wasn’t good, but because it keeps on growing!

Then last month I got to attend the TAUS QE Summit in Dublin where we had an idea to present some of the work Patrick had done with his productivity plugin, get involved in the workshop style discussions, and also learn a little about the sort of things users wanted metrics for so we could improve the reporting available out of the box.  At the same time TAUS were working on an implementation around their Dynamic Quality Framework (DQF) and were going to share a little during the event about their new DQF dashboard that would also have an API for developers to connect.

For Patrick this was another challenge and before the event he added quality metrics for SAE J2450 and TAUS DQF.  After the event in less than a week he added to this with the old LISA QA Metric as we do still have customers using this model, and he also added the ability to make up your own and add them to the tool.  Then, he amazingly built an integration to the TAUS DQF Dashboard that was launched at LocWorld in Berlin the week before last week.  This part required considerable work to make sure the application was capable of mirroring the required workflows whilst still neatly slotting into Studio 2014, and of course Studio 2015 which is due in the next few weeks.

You may be getting the feeling that I really like this application and I’m very impressed by the work Patrick has done, and you’d be right.  The capability this brings within the normal working environment for Studio is quite amazing and it becomes possible for the users to collect a level of detail I don’t think has been possible in this way until now.  To give you a quick summary of the sort of capability available I’ve listed a few things below:

• The Translation Memory or Machine Translation suggestions that were used by the translator are recorded
• Tracking of whether the translation was then later adapted by the translator before finishing the translation.
• Records every single action that qualifies as an addition, deletion or adaptation of any of the properties in a segment.
• The time taken to translate the complete file
• The time taken to translate each segment (segment by segment)
• The key strokes typed by the translator/reviewer (optional)
  The number of words (source) in the segments that were updated
• The Quality Metric applied to a segment during a quality review
• The PEM% (Post Edit Modification) rating and associated cost analysis
• The Track Changes related to what was modified in the segment
• The flat in context comparison of what was modified in the segment
• Cascading comparison results, when a segment is modified more than once (i.e. if a translator visits a segments more than once)
• Ability to quality review a document with your chosen metrics and report on a pass/fail along with a record of every result
• Export the metrics as XML or to Excel where you can easily manipulate the data as you see fit
• Export a full project report for your client containing all the metrics and reports in a way they can use them
• Ability to create TAUS DQF projects from within Studio
• Ability for a translator to import DQF project settings and work on a project, finally updating the TAUS dashboard themselves
• and much much more…

The best thing of all is that all of this is pretty much managed from within Studio without the translator having to do anything more than just work.  So for any organisation working collaboratively with their translators to improve the quality of their Machine Translation engines, or help identify what can be done to improve the training for post-editors, or translators, for example, then this is the perfect platform to do it.  Note that I say “collaboratively” which is the key word here.  The idea of an application like this is not to beat anyone with a stick, and that means anyone because the granularity provided can be useful for translators, reviewers, post-editors and their clients… it’s not just there to provide a one-sided view.  It’s all about working together in an informed way to improve the use of Translation Environment Tools and the processes around them for the translator and the client.

Now, there is so much in this application I don’t even now where to start with a single blog article, so I thought the best idea would be to keep it to the basic summary I provided above and then create a few videos covering the processes and what’s possible.  So I’ve created four videos dealing with the following:

1. Getting started as the Project manager and how to set up the TAUS DQF workflow if you wish to use it (not mandatory)
2. Working as a translator/reviewer with Qualitivity using Studio as a standalone tool, then apply this to a TAUS workflow
3. Working with the Quality Metrics and scoring the translations
4. Taking a look at the reports… built in, exported and TAUS DQF

I think these give a pretty good overview of what’s possible, and might even be useful for anyone testing the tools themselves so they are clear how to use them in the workflows I have described.  It’s probably also worth mentioning that we have created a project group in the SDL Community for this development as there has been quite a bit of interest from users looking for these types of features.  It was used during the development of the toolset but if you’re interested in this kind of application please feel free to join the community now and feedback your experiences in there.  It’s called the Studio Time Tracker Professional project because the early incarnation of the tool started as an enhanced version of the Timetracker plugin that is available on the OpenExchange.  Patrick renamed it Qualitivity because it’s a lot more than a time tracker now!  You can also find some handy videos created by Patrick on the Qualitivity youtube pages which cover the functionality in nice bite sized chunks.

.

Getting started as the Project manager and how to (optionally) set up a TAUS DQF workflow

This video takes you through the creation of a Project and what you need to set up in order to start working with the Qualitivity application in Studio.  It also covers the setting up of a TAUS DQF project and explains what’s needed to engage the editors so they are updating it as they work… from a technical enablement perspective.

https://youtu.be/1lwaX0PKrxY

.

Working as a translator/reviewer with Qualitivity using Studio as a standalone tool, then (optionally) apply this to a TAUS DQF workflow

This video starts with the project package created in the previous video, explains how to use the various settings in the application, and once the post-edit is complete explains how you would then get your data into the TAUS Project where appropriate.  It’s important to note that the application is fully functional without using TAUS at all, but the TAUS integration does have the potential to offer some unique insights into what everyone else is doing too!

https://youtu.be/kptrXirUqLg

.

Working with the Quality Metrics and scoring the translations

This is all about using the Quality features in the application.  Qualitivity comes with four recogninsed standards preconfigured for use and an easily customisable interface if you want to use your own.  The video shows how to set this up, and also how to use it during the review process.  Finally it briefly covers the Quality reporting built into the interface.

.

Taking a look at the reports… built in, exported and TAUS DQF

This is what it’s all about!  You’ve got your data, now what are you going to do with it?  The video runs though the possible reporting options starting with the built in UI features, but ending in the fantastic reporting hand-off packages which top off the collaborative principles of this application quite nicely.  If all you wanted to know was what value you can get from using this application then just watch this one!

.

That’s it in a nutshell!  That was easy to say, but this is by far the most comprehensive OpenExchange plugin I have seen developed to date, and it covers such a wide range of features that are only made possible through the extensibility of the Studio platform and it’s APIs… plus of course the ingenuity of the developer!!  If you’d like to know what’s possible this website has all the information to point developers in the right direction… and it’s free!

Oh yes… where can you find this application?  It’s on the SDL OpenExchange (now RWS AppStore) (aka SDL AppStore) under QUALITIVITY.

.

.