Palagpat Coding

Fun with JavaScript, game theory, and the occasional outbreak of seriousness

Monday, January 25, 2010

Why data transformations are hard (part 1)

Per my plan for January, last Friday was the day I'd planned to have several EFR segmentation files ready for use later on in testing the software I'm going to build for my Linguistics MA capstone project.

My task for that week was to take several XML movie transcripts (in segmented, time-aligned XCES format) and spit out text files in a rather simple format I'll call EFR-segmentation (or segfile for short), which can be parsed by other pre-existing tools in the EFR toolchain. Here's a sample of the input and target formats, so we can clearly define our start and end points:

XCES format (the source):
  
    
EFR-segfile format (the target):
2460-2520 (00:01:20:20 - 00:01:22:11)
Scene 20: T14S
//T: So, you do like it, don't you?

Since the starting point was an XML format, my first thought was to use XSLT transformations to accomplish the reformatting. I created a stylesheet, segfile.xsl, to transform the XCES data files into something closely resembling the EFR-segfile format... but it was lacking in a couple of minor-but-not-insignificant ways. Observe:

Almost-but-not-quite-segfile format:
00:01:20,664
Scene 20: T14S
//T: So , you do like it , don' t you ?

The major problem with this output is the timestamp, which in XCES is formatted like hh:mm:ss,ms. EFR-segfile's timestamp format, on the other hand, is start_frame-end_frame (hh:mm:ss:frame - hh:mm:ss:frame). The milliseconds-to-frames part is easy:

var old_format = "00:02:36,532";
var parts = old_format.split(',');
// X ms * (1 sec/1000 ms) * (30 frames/1 sec) = Y frames
var new_format = parts[0] + ':' + Math.round(parseInt(parts[1]) * 0.03);
// new_format == "00:02:36:16"

... but the rest of that format change is not so simple. We're lucky in the above segment, because there are two timestamps: one at the beginning and one at the end. This seems to be the exception, however, and not the rule, so we'll still need to keep some kind of "current timecode" variable on hand while our parser works its way through the file. Once this is addressed, a second task is to convert the segment start and end times from the hh:mm:ss:ff format into raw frame counts, since EFR segfiles require both in order to have a valid timestamp line. This is a simple enough calculation, but again, doing it in XSLT is like trying to cut a 2x4 with a hammer, when we have other tools in our toolbox.

Besides the timestamp-formatting issues, there was another glitch: the punctuation characters in the transcript line spit out by the XSLT are padded by unnecessary whitespace. For sentence terminators and splicers (periods, commas, etc), this doesn't present a huge problem. But inter-word apostrophes, indicating contractions, split up a word in such a way that any downstream term lookup routine will fail to match it properly. This kind of minor text cleanup would be a snap with my Bat'leth tool; it's designed with text parsing in mind, and has a lot of flexibility. But unfortunately, the timestamp conversion is more complex than Bat'leth's search-and-replace regular expressions can handle, so even if I used it for transcript cleanup, I'd still need another tool to do the timestamps.

Given these shortcomings, I ruled out both XSLT and Bat'Leth as monolithic solutions, although each could contribute a step to the overall parsing workflow. I really wanted to have a single piece of software that could do both parts, however, so I decided to try doing it in Javascript. Then, I fell down the hole that is cross-browser XML — which is a whole different blog post I'll have to write soon. (executive summary in one word: complicated). Given the difficulty of that approach and the pressing deadline, I ended up writing something quick and dirty in Visual Basic 6. Not my first choice, but it got the job done.

The story doesn't end there, though. One of the movies I wanted to treat was 1985's The Goonies, something of a cultural touchstone for my generation. I found an XCES alignment file for the movie in the OPUS OpenSubtitles corpus that I'm using, fed it through my parser, and got this EFR-segfile. Looked pretty good, I thought... so I fired up my EFR authoring software to verify the transcript alignment. And.... it didn't match up! Tune in later this week for part 2 of this post, when we try to figure out why.

Labels: , , , ,

Sunday, January 17, 2010

Updated download: Bat'leth 3.1

This weekend I've been working with Bat'leth, a tool I wrote a number of years ago for text processing and data format manipulation, on a project for my Computational Linguistics graduate work that I'll be blogging about soon. As I worked with it, I noticed that it was looking quite dated. It didn't support XP-style controls, some of the button images were inconsistently styled and not properly masked, and so forth. (I also found a bug or two!)

As I wrote a few days ago, I have a love/hate relationship with my old Visual Basic code: I love how easy it is to hack together a "quick and dirty" tool when I need one, and I hate how hard it is to make it actually look polished and modern when finished. Then I stumbled across an add-on for VB called vbAdvance. It extends old-school VB in some very welcome ways, including the ability to compile an XP manifest file into your apps so that they support the newer visual styles (other cool features include the ability to create DLLs with function exports, console apps with no GUI, and several other nifty bits). Though originally commercial, it's now classed as "unsupported freeware," so if you or anyone you know does any work in classic VB, this is well worth your time.

Anyway, thanks to vbAdvance and a little surplus time this weekend, I've updated Bat'leth to version 3.1, now available on my downloads page as both an install package and full VB6 source code. Enjoy, and let me know if you find this at all useful.

Labels: , ,

Thursday, January 14, 2010

Dear Visual Basic: It's not you, it's me

I just read a great article over at O'Reilly, 7 reasons I switched back to PHP after 2 years on Rails. It's a few years old, and the comments thread is practically a flamewar between pro- and anti- Ruby on Rails people, but the article itself raises some great points — not about how awesome PHP is, but about the art and science of software design. The author, Derek Sivers (founder of CD Baby) had this to say about a developer's perennial urge to rewrite old code:

But the main reason that any programmer learning any new language thinks the new language is SO much better than the old one is because he’s a better programmer now! You look back at your old ugly PHP code, compared to your new beautiful Ruby code, and think, "God that PHP is ugly!" But don’t forget you wrote that PHP years ago and are unfairly discriminating against it now. It’s not the language (entirely). It’s you, dude. You’re better now. Give yourself some credit.

I love that! This week I've been re-visiting a lot of my old Visual Basic code to get some things freshened up for my Downloads page, so I can absolutely relate. Looking over the code, it's easy to dismiss it as simplistic and over-reliant on hacks to accomplish its aims. So Sivers' article was a nice reminder to me that the language itself isn't the problem (well, mostly), it's just the way I wrote in it. So if time allows, I'll be revisiting a few of these tools in the coming weeks with an eye to make them better.

Labels: ,

Wednesday, January 13, 2010

Downloads: now with source code

I've fixed the broken links on my Downloads page. So far there are mostly only zip archives with the VB6 source code, with only one app that actually includes an installer. I intend to upload more binaries and/or installers soon; tonight I'm working on an update to Bat'leth that will help me do some work on my grad school project (I'll post more on that effort on my root blog in a day or two)

Labels: , ,

Friday, January 08, 2010

View-Source is Good?

Alex Russell, founder and president of the Dojo Foundation, recently wrote on his blog about the importance of "view source" to the evolution and growth of the World Wide Web:

Web developers get started by taking some code, pasting it into a file, loading it in a browser and switching between editor and browser between even the most minor changes. This is a stark contrast with other types of development, notably those that impose a compilation step on development, in which the process of seeing what what done requires an intermediate action. In other words, immediacy of output helps build an understanding of how the system will behave, and ctrl-r becomes a seductive and productive way for developers to accelerate their learning in the copy-paste-tweak loop. The only required equipment is a text editor and a web browser, tools that are free and work together instantly. That is to say, there’s no waiting between when you save the file to disk and when you can view the results. It’s just a ctrl-r away.

As I read the above quote, it hit me: this is exactly why I love developing in old-school Visual Basic, and why even now VB.NET doesn't feel natural to me. For almost 10 years, my work was primarily done in Visual Basic 5 (and later 6), where the normal process of development runs something like:

  1. Design something in the UI
  2. Write an event handlers for the UI component
  3. Run the app in debug mode and test the new feature
  4. Break program execution and tweak the event handler
  5. Continue program execution and test the revised code
  6. Repeat steps 1-5 as necessary

This is precisely the same reason why I love web development so much: the immediacy of the feedback loop, the ability to see your changes almost as you make them, and to, essentially, evolve the code to achieve your end goals. Tools like Firebug have made this process even easier, allowing me to actually poke and prod UI elements in runtime without having to change and refresh the underlying code. And I absolutely love the ability to see a cool effect coded by someone else, and do the old "right-click, inspect element" to see how they did it.

Interestingly, the parallels between VB and the web don't end there. From time to time you hear developers of languages like Java and C++ pooh-pooh both Visual Basic and Javascript as being somehow "inferior". The simple syntax, lack of a compiler and need for a runtime are somehow seen as negatives, even though the trend from the beginning has always been towards more, not less, abstraction.

Now, do I use VB6 for all of my Windows desktop development? No, I also code in C++, C#, and sometimes even batch scripts. Do I use JavaScript for all of my web development? No, all of my back-end and template code is typically written in Python, PHP, and the like. But for when I need a simple tool to do a simple job, simple code wins every time.

Labels: , ,