Why data transformations are hard (part 2)

Previously, on Palagpat Coding...

In Part 1 of this post, I talked about how I was going to attack my task of converting an XML-based format for file transcription (XCES) into the flat-but-structured EFR-segfile format. At the end of the day, I'd gone with a quick-and-dirty solution in Visual Basic 6, as both XSLT and my Bat'leth tool seemed to fall short. Here's a sample of the input and output of that effort:

  
    
    Lunchtime
    !
    
  

// EFR-segfile output:
0-1164 (00:00:00:00 - 00:00:38:24)
Scene 1: T1S
//T: Lunchtime!

As I mentioned last time, I took this EFR-segfile output and loaded it up in my EFR authoring software to test the segment boundaries, and found that it was off... not hugely off at the outset, just a few seconds... but things very quickly went off the rails. For example, by the end of Chapter 2 (just 13 minutes into the film), my segfile segments were already off by over 7 minutes!

37020-37231 (00:20:34:00 - 00:20:41:01)
Scene 417: T303S
//T: Come on, before you catch a real cold.
//N: actual film time: 00:13:27-00:13:30

Clearly, this won't do.

What happened?

Looking things over, I realized a couple of mistakes right away: First, I had assumed the XCES files covered the full film, so I started at frame 0. However, the first dialog in the film doesn't occur until about 40 seconds in, so my output started off wrong by that much. Secondly, I started each subsequent clip from the endpoint of the previous one, thus ignoring any segments of the movie that didn't involve spoken dialog, so as the film progressed, the timestamps got more and more wrong.

At this point I took another look at my original XCES file, and it hit me that there are actually two, overlapping data structures at play:

  
     
    Get
    down
    ,
    guys
    .
  
  
    Get
    down
    .
     
  

The <s> tags, which I thought represented segments, instead appear to represent individual sentences. (with the <w> tags representing individual words... seems rather obvious in retrospect). When I looked at the format last time, I didn't give any thought to why the <time> elements sometimes came at the beginning of a sentence and sometimes at the end, but in my review I noticed a consistent trend: the instances that occur as the first child of an <s> element have an id ending in S, and those that are the last child of an <s> element have an id ending in E... S for "Start", and E for "End." So the XML structure of <s>entences and <w>ords has a sort of a superstructure that is made up of T(ranscript?) blocks of one or more sentences each. I'm guessing XCES doesn't support this kind of 3-tier structure, hence the ad-hoc markup, but as a long-time user of XML, I really would have preferred something like this:

  
    Get
    down
    ,
    guys
    .
  
  
    Get
    down
    .
  

But, beggars can't be choosers, and anyway, my parser isn't a pure-XSLT solution anyway, so in the long run it doesn't matter all that much, even if it does make the data purist in me cringe a little.

Shadow data

As the title of this post asserts, it's been my experience that data transformations are only rarely as straightforward as they look. Several years ago, I worked for a B2B e-commerce company that handled a lot of EDI data, both reading it and writing it. In theory, this should be a standard format: if you know the specification, you know what the data will look like. In practice, however, we found that every new company we brought on the system had their own "shadow" data structures hidden in the standard fields, just like the invisible <t>ranscript blocks in this XCES data.

The takeaway is this: when you're writing a parser for a new-to-you data format, no matter how "standard" it professes to be, never take your data's structure at face value. There may be hidden complexities, just waiting to be tripped over.

Dear Visual Basic: It's not you, it's me

I just read a great article over at O'Reilly, 7 reasons I switched back to PHP after 2 years on Rails. It's a few years old, and the comments thread is practically a flamewar between pro- and anti- Ruby on Rails people, but the article itself raises some great points — not about how awesome PHP is, but about the art and science of software design. The author, Derek Sivers (founder of CD Baby) had this to say about a developer's perennial urge to rewrite old code:

But the main reason that any programmer learning any new language thinks the new language is SO much better than the old one is because he’s a better programmer now! You look back at your old ugly PHP code, compared to your new beautiful Ruby code, and think, "God that PHP is ugly!" But don’t forget you wrote that PHP years ago and are unfairly discriminating against it now. It’s not the language (entirely). It’s you, dude. You’re better now. Give yourself some credit.

I love that! This week I've been re-visiting a lot of my old Visual Basic code to get some things freshened up for my Downloads page, so I can absolutely relate. Looking over the code, it's easy to dismiss it as simplistic and over-reliant on hacks to accomplish its aims. So Sivers' article was a nice reminder to me that the language itself isn't the problem (well, mostly), it's just the way I wrote in it. So if time allows, I'll be revisiting a few of these tools in the coming weeks with an eye to make them better.

View-Source is Good?

Alex Russell, founder and president of the Dojo Foundation, recently wrote on his blog about the importance of "view source" to the evolution and growth of the World Wide Web:

Web developers get started by taking some code, pasting it into a file, loading it in a browser and switching between editor and browser between even the most minor changes. This is a stark contrast with other types of development, notably those that impose a compilation step on development, in which the process of seeing what what done requires an intermediate action. In other words, immediacy of output helps build an understanding of how the system will behave, and ctrl-r becomes a seductive and productive way for developers to accelerate their learning in the copy-paste-tweak loop. The only required equipment is a text editor and a web browser, tools that are free and work together instantly. That is to say, there’s no waiting between when you save the file to disk and when you can view the results. It’s just a ctrl-r away.

As I read the above quote, it hit me: this is exactly why I love developing in old-school Visual Basic, and why even now VB.NET doesn't feel natural to me. For almost 10 years, my work was primarily done in Visual Basic 5 (and later 6), where the normal process of development runs something like:

1. Design something in the UI
2. Write an event handlers for the UI component
3. Run the app in debug mode and test the new feature
4. Break program execution and tweak the event handler
5. Continue program execution and test the revised code
6. Repeat steps 1-5 as necessary

This is precisely the same reason why I love web development so much: the immediacy of the feedback loop, the ability to see your changes almost as you make them, and to, essentially, evolve the code to achieve your end goals. Tools like Firebug have made this process even easier, allowing me to actually poke and prod UI elements in runtime without having to change and refresh the underlying code. And I absolutely love the ability to see a cool effect coded by someone else, and do the old "right-click, inspect element" to see how they did it.

Interestingly, the parallels between VB and the web don't end there. From time to time you hear developers of languages like Java and C++ pooh-pooh both Visual Basic and Javascript as being somehow "inferior". The simple syntax, lack of a compiler and need for a runtime are somehow seen as negatives, even though the trend from the beginning has always been towards more, not less, abstraction.

Now, do I use VB6 for all of my Windows desktop development? No, I also code in C++, C#, and sometimes even batch scripts. Do I use JavaScript for all of my web development? No, all of my back-end and template code is typically written in Python, PHP, and the like. But for when I need a simple tool to do a simple job, simple code wins every time.