Our Domain-Specific Language is probably more domain-specific than you might be used to. Frameworks like Ruby-on-Rails or ScalaTest are DSLs for very broad domains (web-app development, and testing, respectively). Our DSL’s domain is our web product. Not any web product, not any Spring app, our web app. This is important, as it allows us to creating something that is very focused and specific to what we’re doing. In general, the more specific the domain, the more concise and powerful you can make your DSL.

The domain of application: its architecture

To understand how it works, you need to understand how the apppilcation is architected. While it’s a Spring MVC web app, in the general sense, there’s a lot more convention around how it’s built. If you’ve used a “web framework” like Spring MVC, you know that it’s really a giant library for responding to HTTP requests at its core, and that it’s not very opinionated; it’s generally just as easy to do things one way as it is another (check out this if you disgree). As such, you need conventions around how to use it. Developers shouldn’t spend time asking how to wire up URLs, or where code should go; they should be getting to work and building features.

Our app is built along two main concepts: pages, and modules. A Page is made up of one or more modules. Each module contains a basic bit of information needed to display a page. Think about websites that say “Hi, Dave!” in the upper corner; that “Hi, Dave” would be controlled by a module. Each page is backed by a controller whose job it is to expose only what’s needed to configure the modules on the page. All stateful information (e.g. “who’s logged in?”) is in these controllers. Modules, on the other hand, are stateless; they get all the information they need to render from the URL and query parameters. Modules are made up of two parts: a resource, which identies the data to display, and a view, which renders that data. A single resource can have many views, and together, they form a module.

This is all glued together via a souped-up version of a <jsp:include> tag. A page jsp-includes a bunch of urls, that happen to be modules, and conform to these conventions. We have scaffolding scripts to generate the massive boilerplate required to make this happen.

Testing with Scala

Which brings us to our DSL for testing this stuff. Essentially, a module’s url can be requested and tested independently of any page; we don’t need to navigate through the application to test modules, since they are almost entirely stateless (typically, they might require a login, but this a cross-cutting concern). By requesting the module’s URL, and forcing it to render a view, we also get test coverage of our JSP pages, and can push our app to QA with high confidence that all pages will at least render, and that the correct information will be somewhere on the page.

This can all be tested with HTMLUnit, however the tests began to look like bloated assembly-language. Enter Scala. I started by writing out the ideal pattern of a test for a module. Let’s consider a module that renders a person’s most recent bill. Suppose we have two views of this module; in the “large” view, we want to see the bill’s cost, KWh used, and an account number. In the “small” view (that we might use on a dashboard page), we want to see just the bill’s cost. How might we test this?

resource for bill 34 from customer 123:
  requires login as customer 123
  contains "45.67" as the cost
  has a view "small"
  has a view "large"
    that contains "156" as the kwh
    that contains "655321" as the account number

I decided I wanted to write this in code, and have it work. Not having time to write a parser (or invent a new language), I decided an internal DSL would be the way to go, and that Scala would let me get as close as possible to this:

This code creates several data structures that our test can now walk. Essentially, what this will test:

• That accessing the url without being logged in gets an error
• That, once logged in, requesting a view named “small” will not generate an error
• That, once logged in, requesting a view named “large” will not generate an error
• That neither view has message properties that are missing their values (we aggresively use message properties to allow localization)
• That both views contain a div with the id price that contains the text “45.67″
• That the “large” view also contains a div with id kwh that contains “156″ and a div with id “accountNumber” that contains “655321″

Getting fancy

Since this is just Scala code, you can use this to do more sophisitcated things:

I don’t know if the developers are totally in love with this DSL, but there’s so much you get for free, I’m certain they’d hate testing that by hand even more (or, worse, simply not test some of these things):

• You never have to remember to check for authentication requirements
• You never have to remember to check for missing message properties
• You can dynamically generates tests via test-data (as demonstrated above)
• Because making a new data structure in Scala is so easy, you can make very fluent tests and test data.

Of course, this DSL is entirely useless to anyone but our web team. It’s tailor-made to test our web product and our web product only. The win, other than conciseness, readability, and brevity, was that this could be implemented and documented very quickly; I didn’t have to make the ultimate web-based tesitng DSL; just one that worked for our product.

If you are considering creating a DSL, keep this in mind: make it exactly right for you, and fight the urge to make it more general. Also, don’t be afraid to use Scala for this; it’s much easier than Java, and you can fully type-check and document it very easily. I should note that I intentionally used very few of Scala’s features to do this; developers only need to learn a few new concepts to understand what this code does.

One thing I’ve noticed about successful candidates is polish. This isn’t a word thrown around when discussing developers a whole lot, but I’ve found it to be a strong indicator of a true software professional. The OPOWER application is more than just a résumé; it’s a written quiz and a coding assignment as well. Sloppiness at any point is a negative; it shows that the candidate just doesn’t give a shit about getting the job. It’s not a deal-breaker, but it has a huge effect on how excited we get about a candidate, and it could be the deciding factor between two otherwise identical candidates. It’s our version of “hire the better writer.”

So, what do I mean by “polish”? Here’s a few examples:

• Make your code better than you would “at work.” Don’t make me download the JUnit jar and type javac `find . -name *.java` to compile your code; give me a pom.xml or Ant file with Ivy support.
• Send PDFs instead of MS-Word; they are so much easier on the eyes (and processor). This is dead simple on OS X, there’s some free options for Windows, and there’s always LaTeX (which will definitely earn you some geek cred
• Don’t send me the functional spec of your last project and call it a résumé. Use bullet lists, short sentences, and formatting to create an easy-to-scan CV.
• Format your writing; compare this, to this. Even though they both are excerpts from correctly-answered quizzes, which one is going to be easier and more engaging for me to read?
• English: do you speak it?. Being able to communicate effectively using the written word is absolutely crucial for being a good developer, especially at OPOWER. It doesn’t need to be your first, or even best, language; you just need to wield it as effectively as you do Java

A lack of polish won’t get you refused, but putting an extra shine on your application can make you stand out quite a bit…

But first, let’s talk about defensive coding and how we do it. For example:

/**
 * frobs the bar and baz.
 * @param bar the bar to frob; may not be null
 * @param baz the baz to from; must be positive
 */
public String frob(String bar, int baz) {
  if (bar.length() > baz) {
    return bar.substring(baz);
  }
  else {
    return bar;
  }
}

If the contract is violated by the caller, we will get a NullPointerException or some other StringIndexOutOfBoundsException. Not too helpful. Instead, we use an internal validator helper class to provide useful messages. This is crucial for properly understanding bugs that occur in production; there’s nothing worse than coming into work to find a NullPointerException in your inbox and no clue what went wrong.

/**
 * frobs the bar and baz.
 * @param bar the bar to frob; may not be null
 * @param baz the baz to from; must be positive
 */
public String frob(String bar, int baz) {
  Validate.notNull(bar,"bar must not be null to frob");
  Validate.isTrue(baz > 0,"baz must be positive");
  if (bar.length() > baz) {
    return bar.substring(baz);
  }
  else {
    return bar;
  }
}

Validate.* methods essentially throw IllegalArgumentException if what they are checking for fails.
We use this pattern extensively, even in constructors of simple data-structure style objects. This (plus keeping this things immutable) allows users of these objects to safely rely upon the get methods behaving properly. I had need of a class to hold both a Person (representing a website user) and a Customer (representing a utility-company customer). This class (called, naturally, PersonAndCustomer), takes both objects in its constructor and requires that they are both non-null. As a further sanity check, I wanted to make sure that the Person‘s customerId matched the id of the Customer that was being passed in (i.e. that they represented the same actual human in the real world):

public PersonAndCustomer(Person p, Customer c) {
  Validate.notNull(p,"Person may not be null");
  Validate.notNull(c,"Customer may not be null");
  Validate.isTrue(p.getCustomerId() == c.getId(),
    "Person %d's customer id %d didn't match passed-in customer's id %d",
    p.getId(),
    p.getCustomerId(),
    c.getId());
  ...
}

Looks pretty inocuous, right? Well, because these objects are Hibernate-managed, all of our ids are Long and not long. So, my isTrue validation is really checking that the person’s customerId object is the same object as the customer’s id object. What’s worse, in several months of development and production deployment, these two objects happened to be the same.

Until this week; we were testing our application for a new client and I got the error posted above (“Person 1001′s customer id 109032 didn’t match the passed-in customer’s id of 109032″). It seems that much like how the JVM will re-use string objects, it also will sometimes re-use boxed objects as well. But not always. The fix for this is obvious, but I wanted to make sure I could actually recreate this situation. So, I created a test that gave both the Person and Customer the same value for the customer id, but as different objects. and verified that the class could still be constructed. Sure enough, the test failed. The fix:

public PersonAndCustomer(Person p, Customer c) {
  Validate.notNull(p,"Person may not be null");
  Validate.notNull(c,"Customer may not be null");
  Validate.isTrue(p.getCustomerId().equals(c.getId()),
    "Person %d's customer id %d didn't match passed-in customer's id %d",
    p.getId(),
    p.getCustomerId(),
    c.getId());
  ...
}

What’s interesting is that if one end of the == is a primitive, the JVM will unbox the other one and the test succeeds:

Long l1 = new Long(45L);
Long l2 = new Long(45L);

System.out.println(l1 == l2);      // false
System.out.println(l1.equals(l2)); // true
System.out.println(l1 == 45L);     // true!

The moral of the story is to always know what you are comparing. Might even be best to always use .equals() unless the compiler complains.

My first thought was: there’s lots of blocking reading and writing from the database, so adding some threads should speed things up. While one thread was analyzing Bob’s energy data, another could be fetching Mary’s, while another could be updating Joe’s meta-data with the results. Or so I thought.

The more threads I added, the slower the entire thing became. It turned out that the fastest implementation was a single-threaded one. But why? It all has to do with the diminishing returns one gets from scaling out.

If you think of a task as having a serial component, which cannot be broken up concurrently, and then multiple tasks which can be done concurrently, we can analyze the returns we get by increasing the number of available processors (threads, in my case). This is Amdahl’s Law and is exemplified by the following equation (where “x” is the number of processors or threads, and “s” is the percentage of your overall task that must be serialized; “y” is the increase in speed you will see from scaling).

When you graph this, it’s pretty obvious that there are diminishing returns to adding more threads/processors (the graph below assumes that 90% of the overall job can be done concurrently). As we add threads, we get less and less of a gain in speed.
But there’s still a gain to get, so what happened to me?

Amdahl’s law is actually pretty optimistic. It doesn’t account for the overhead required to synchronize the shared data. If we account for this, with a new value “k” ( the percentage penalty for maintaining consistency), we see that increasing our processors/threads actually starts to hurt us (this equation is in red)!

In my case, almost all of this synchronization is happening within the database; it turns out that almost all the time my process needs is accessing data from the database. So, I dialed it down to only one thread, and we should be done early next week.

Nacho Cheese Doritos.

Honorable Mentions: Fritos and Cool Ranch Doritos.

The easiest way to do that is to look at the bugs our QA staff find in the product updates. Although we have a lot of automated tests, we still need some eyes on the applications to check for things like text overruns of our printed reports, or JavaScript weirdness in some less-than-well-behaved) browsers.

After a few iterations, Anh (our beloved program manager) ran some queries on our bug-tracking database and, using some Excel magic, showed us numbers like “number of defects in the web application” and “average number of days a critical bug sat unassigned”. This is good stuff; seeing hard data about what we’re doing lets us focus on inefficiency. Since we release software every month (and launch new clients almost as frequently), this is a huge help in making each iteration better than the previous.

While Anh’s hand-jammed report was awesome, manually connecting SQLite and Excel is no fun for anyone. I whipped up a basic Ruby application using Sinatra to automate most of what we’d like to see. We can now see how long critical defects site open and unfixed, as well as how many overall defects we’re fixing and even the root source of our defects (e.g. did we get the wrong requirements, or just mess up implementing them?)

We still need to show trends, however. “Classic” thinking in software process improvement says to do that by dividing everything by lines of code; if we had 10 defects per 1,000 lines of code this month, but only 8 the following, we’re doing better. This is not a great measure for us, since we are changing and enhancing our product (as opposed to writing fresh code each time), so we decided to normalize our measurements by the story points we assigned to the iteration. This way, we can see charts like the one below (which says we are doing better each time):

This information is dead simple to collect; the burden on developers and testers when writing defects is almost nothing. With only a priority, severity, and “source of defect”, we can collect a lot of advanced information about our development process — and make it better iteration by iteration.