At Opower, we recognize that our team members are as complex as our customers, and our ability to create brilliant products depends on our ability to reimagine what’s possible. That’s why, twice a year, we shut out the world and focus on harnessing our collective talent to improve everything Opower—these are our Innovation Days. Similar to hackathons and 20 percent time, they go even further to promote community, creative thinking, and freedom from organizational hierarchy.

Innovating, on the shoulders of giants

In recent years a lot has been made over Google’s approach to innovation—20 percent time dedicated toward projects that further the company mission—which was similar to the approach 3M took as far back as 1948 with 15 percent time. Before Google, HP was encouraging employees to use 10 percent time—after lunch on Fridays—to work on whatever they wanted.

Some amazing products have come from those companies and their approach to innovation outside of normal product initiatives:

• Gmail, AdSense, and Google Earth
• from 3M—a sandpaper with rearranged particles that easily cuts through metal, known as the Cubitron II
• one HP employee apparently used his 10 percent time to help invent HTML

These products revolutionized how people around the world go about their work.

The format of those innovation initiatives was generally geared toward employees with technical abilities, and the actual percent time was not always formalized, either. At Google, if you wanted to work on something, you just did—there was no paperwork, no manager approval.

Many other companies take a different approach to innovation by hosting hackathons—companies like Yahoo!, Foursquare, Salesforce, and Yelp. This approach encourages public participation in a contest to build on top of a company’s platform. Many awesome products have come out of these events over the years, as well.

What we’ve learned

As we continue to improve on our Innovation Days, we’ve gleaned a couple key insights from these giants of innovation:

• innovation can come from anyone and anywhere, and it’s up to us to recognize opportunity and champion it
• without facilitating a time and place for unconstrained work, it’s incredibly difficult for people to make innovation happen on their own time without support and encouragement

As such, our guiding principles are simple: remove constraints and facilitate.

Making time for what could be

The only rule of Opower’s Innovation Day is that folks must present their projects at our Innovation Fair—a science-fair style event where everyone gets to share what they worked on and check out other folks’ projects. The rationale, from a behavioral psychology perspective, is that when people publicly proclaim their intentions, they make themselves accountable to people who trust them to follow through. We want our team members to not only pursue their best ideas, but realize their ideas in ways that allow their projects to come to life for others.

To remove constraints:

• everybody is included, and people can work with whoever they wish on whatever they want
• internal meetings, client meetings, or other regular commitments are not allowed

And to facilitate:

• we’ve created a simple digital space for folks to share ideas and join in groups
• about a week prior, we run Pitchathons in each office where anyone can propose and share ideas

Working with participants across three offices in two countries has presented its share of challenges. For years, we tried coordinating the events so that kickoff and summary speeches coincided, but we realized that allowing each office to embrace its own time zone, location, and unique culture was the best way forward.

In my reading, I came across a great article by Rebecca Wirfs-Brock [1] in which she discusses the notion of “habitable code”. This idea isn’t original to her (she credits Richard Gabriel in Patterns of Software [2]), but she provides a great explanation of the concept, and goes on to talk about its role in complex systems.

Paraphrasing some key points …

• Clarity for its own sake is an elusive goal. In my own experience, if I strive really hard to make some aspect of my code crystal clear (say, a challenging algorithm or data structure), it’s at the expense of clarity in some other area. It’s a bit like a game of Whac-A-Mole. Just as you can’t write code that’s infinitely flexible in the face of unknown future requirements, I don’t think you can write code that’s infinitely clear to an unknown future audience.
• Beauty becomes a constraint for future authors. It’s hard to extend or update excessively beautiful code without spoiling some of its inherent loveliness. An analogy to industrial design — the MacBook Air achieves a certain level of elegance by ruthlessly eliminating functionality like replaceable batteries, integrated ethernet or a DVD drive. How difficult would it be to add that stuff in later without compromising the original aesthetic? Probably very.
• Habitable code makes its structure and intentions easy to understand. Structure, sure, but as a professional developer, the “and intentions” part seems pretty important. You can read code to understand what it’s doing, but it’s not always obvious what problem the original author thought it was solving. This is a hard thing to do right — the best I’ve been able to do is document the problem being solved, and make the structure of the code appropriate to the solution. Code reviews are a huge help in identifying areas that need improvement in either direction.
• Habitable code makes developers feel at home. At Opower we have a pretty good-sized body of coding conventions, some of which are automatically enforced using tools like Checkstyle and PMD. To new hires, it sometimes seems totalitarian (it did to me). But once you’ve been around awhile, you notice a funny thing: you can go almost anywhere in our source base and see things done in a similar fashion. This helps us transition from artifact to artifact more easily, and reduces the overhead in sharing code. Habitable, indeed.

Just like any other principles, these can be reduced to absurdity. Giving up completely on clarity is obviously not the answer, nor is making your code horribly ugly so that future maintainers don’t feel bad about rewriting it. But it’s interesting to think about where habitability differs from clarity and beauty, and what it demands of us as developers.

So … who’s inhabiting your code today? Whose code are you inhabiting? How is that going to change in a month? Six months? A year? And what are you doing to make each other feel welcome and at home?

References

[1] http://www.wirfs-brock.com/PDFs/DoesBeautifulCodeImply.pdf
[2] http://dreamsongs.net/Files/PatternsOfSoftware.pdf

• 4 WAR applications
• 33 JAR libraries (1 open sourced @ https://github.com/opower/jpile)
• 14 “pom artifacts” that define groupings of projects and organize dependencies, versions, etc

Not all of that growth was brand new, however.  As we expanded our product lines and thought more about the best way to scale our core features we invested heavily in the decomposition of existing WARs and improving the “librarification” of our code.  In addition to supporting scaling, improving the modularity of our code base also supports a move towards S.O.A. and improves the velocity of our scrum teams.  After all, it’s a lot easier to focus on developing the stories in your iteration if you don’t need to worry about stepping on another team’s toes.  That’s more easily achievable when your team aligns to one specific code artifact.

A problem we quickly ran in to as we started decomposing WARs and JARs was the creation of a lot of redundant spring configuration files– especially in an integration test context.  Here’s a crude picture of our initial state following some decomposition:

One WAR depending on two JARs with duplicated spring bean configuration

Note that the WAR includes the two JARs and wires together beans declared in the JARs with an applicationContext.xml defined in the WAR.  We thought it would be a good idea to adopt the policy of “keep the tests close to the code that they stress.”  That’s a sensible policy, but resulted in duplication of context configuration found in the WAR for the purposes of spinning up Spring contexts during integration tests.

Reducing that duplication was done by continuing the theme of decomposition and applying it to the Spring context as well.  For example, we adopted the policy that a JAR should export a sensible “default wiring” of the classes that it encapsulates.  We adopted naming conventions to facilitate the need to either explicitly or implicitly include a Spring context snippet.

• If a JAR includes a context file ending in “-spring.xml” then it’s automatically included by any WAR that declares a dependency on that JAR.
• On the other hand, if a JAR includes a context file ending in “Context.xml” then it is only brought in to the WAR’s context with an explicitly declared inclusion of that file.

Here’s what the picture looked like when we migrated to that model:

One WAR with two JARs, but no duplicated bean configuration

Integration tests that lived in our various JARs liked living in this model because they could always import “*-spring.xml” and their integration test context would come up and run just fine.

This worked well until we encountered some situations where one JAR started depending on another JAR.  We wanted to ship sensible Spring context defaults from both JARs, but because of the “*-spring.xml” rule we ended up needing to duplicate all of the application properties that were needed to wire up the beans in the depended artifact from within the src/test/resources directory of the dependent JAR.  Here’s a picture to explain what I mean:

Application properties starting to proliferate

This problem just got worse as you got more and more descended from a given library– for example, if jar-A depends on jar-B depends on jar-C and they all ship with their own “*-spring.xml” files we needed to define properties to configure C within jar-B and we needed to define properties to configure jar-B and jar-C within jar-A.  It was getting hairy.

To solve this we adopted a new standard that basically amounts to “provide sensible defaults for our sensible defaults”: any bean in a “*-spring.xml” snippet that can be configured with a PropertyPlaceholderConfigurer must provide a sensible default for that property.  So a bean definition might have looked like this before:

    <bean name="foo" class="opower.Foo">
        <property name="locale" value="${locale}"/>
    </bean>

After:

    <bean name="foo" class="opower.Foo">
        <property name="locale" value="${locale:en_US}"/>
    </bean>

The defaulting of application properties allowed us to eliminate 90% of the boilerplate test properties and code that we would have otherwise needed and allows for a greater abstraction of thought when a developer decides to pull in a library.  Thus, our current setup looks something like this:

Defaulted properties eliminates almost all configuration

A couple of tricks to note when using this model:

• There’s a big difference in classpath context scanning between “classpath:/*-spring.xml” and “classpath*:/*-spring.xml”.  The first (with just one star) will pull in all the files that match “*-spring.xml” in the first directory on your classpath that contains any file matching that expression and then stop.  The second version (with classpath*) will do what you’d expect….it pulls in all files that match “*-spring.xml” from every directory in scans in your classpath.
• Application property defaulting accepts literals after the colon, so it’s safe to do something like “${path:file:///blah}” to default the property to the string “file:///blah”
  You can also change the default string separator in the PropertyPlaceholderConfigurer bean so your defaults look more like this: “${path?file:///blah}”  (using a “?” as a separator)
• It’s tempting to eliminate application.properties wherever there’s a default, but we avoided that for documentation reasons– we wanted to provide the implementers and users of our software with a one-stop location to discover and understand the configurability of our applications.  Thus, we adopted a standard that even if an application property is defaulted in a JAR’s spring context snippet, we still explicitly declare that property in a WAR’s top-level application.property and document how that property affects the system.

Our Domain-Specific Language is probably more domain-specific than you might be used to. Frameworks like Ruby-on-Rails or ScalaTest are DSLs for very broad domains (web-app development, and testing, respectively). Our DSL’s domain is our web product. Not any web product, not any Spring app, our web app. This is important, as it allows us to creating something that is very focused and specific to what we’re doing. In general, the more specific the domain, the more concise and powerful you can make your DSL.

The domain of application: its architecture

To understand how it works, you need to understand how the apppilcation is architected. While it’s a Spring MVC web app, in the general sense, there’s a lot more convention around how it’s built. If you’ve used a “web framework” like Spring MVC, you know that it’s really a giant library for responding to HTTP requests at its core, and that it’s not very opinionated; it’s generally just as easy to do things one way as it is another (check out this if you disgree). As such, you need conventions around how to use it. Developers shouldn’t spend time asking how to wire up URLs, or where code should go; they should be getting to work and building features.

Our app is built along two main concepts: pages, and modules. A Page is made up of one or more modules. Each module contains a basic bit of information needed to display a page. Think about websites that say “Hi, Dave!” in the upper corner; that “Hi, Dave” would be controlled by a module. Each page is backed by a controller whose job it is to expose only what’s needed to configure the modules on the page. All stateful information (e.g. “who’s logged in?”) is in these controllers. Modules, on the other hand, are stateless; they get all the information they need to render from the URL and query parameters. Modules are made up of two parts: a resource, which identies the data to display, and a view, which renders that data. A single resource can have many views, and together, they form a module.

This is all glued together via a souped-up version of a <jsp:include> tag. A page jsp-includes a bunch of urls, that happen to be modules, and conform to these conventions. We have scaffolding scripts to generate the massive boilerplate required to make this happen.

Testing with Scala

Which brings us to our DSL for testing this stuff. Essentially, a module’s url can be requested and tested independently of any page; we don’t need to navigate through the application to test modules, since they are almost entirely stateless (typically, they might require a login, but this a cross-cutting concern). By requesting the module’s URL, and forcing it to render a view, we also get test coverage of our JSP pages, and can push our app to QA with high confidence that all pages will at least render, and that the correct information will be somewhere on the page.

This can all be tested with HTMLUnit, however the tests began to look like bloated assembly-language. Enter Scala. I started by writing out the ideal pattern of a test for a module. Let’s consider a module that renders a person’s most recent bill. Suppose we have two views of this module; in the “large” view, we want to see the bill’s cost, KWh used, and an account number. In the “small” view (that we might use on a dashboard page), we want to see just the bill’s cost. How might we test this?

resource for bill 34 from customer 123:
  requires login as customer 123
  contains "45.67" as the cost
  has a view "small"
  has a view "large"
    that contains "156" as the kwh
    that contains "655321" as the account number

I decided I wanted to write this in code, and have it work. Not having time to write a parser (or invent a new language), I decided an internal DSL would be the way to go, and that Scala would let me get as close as possible to this:

This code creates several data structures that our test can now walk. Essentially, what this will test:

• That accessing the url without being logged in gets an error
• That, once logged in, requesting a view named “small” will not generate an error
• That, once logged in, requesting a view named “large” will not generate an error
• That neither view has message properties that are missing their values (we aggresively use message properties to allow localization)
• That both views contain a div with the id price that contains the text “45.67″
• That the “large” view also contains a div with id kwh that contains “156″ and a div with id “accountNumber” that contains “655321″

Getting fancy

Since this is just Scala code, you can use this to do more sophisitcated things:

I don’t know if the developers are totally in love with this DSL, but there’s so much you get for free, I’m certain they’d hate testing that by hand even more (or, worse, simply not test some of these things):

• You never have to remember to check for authentication requirements
• You never have to remember to check for missing message properties
• You can dynamically generates tests via test-data (as demonstrated above)
• Because making a new data structure in Scala is so easy, you can make very fluent tests and test data.

Of course, this DSL is entirely useless to anyone but our web team. It’s tailor-made to test our web product and our web product only. The win, other than conciseness, readability, and brevity, was that this could be implemented and documented very quickly; I didn’t have to make the ultimate web-based tesitng DSL; just one that worked for our product.

If you are considering creating a DSL, keep this in mind: make it exactly right for you, and fight the urge to make it more general. Also, don’t be afraid to use Scala for this; it’s much easier than Java, and you can fully type-check and document it very easily. I should note that I intentionally used very few of Scala’s features to do this; developers only need to learn a few new concepts to understand what this code does.

I’m currently reading Matthew Crawford’s Shop Class as Soulcraft who writes about how manual labor can make you spiritually whole. To get us to look at work in new ways, he describes work using the categories: assertiveness and attentiveness. Crawford writes that work is not exclusively assertive or attentive and that to obtain the spiritualism that comes from manual work, you need balanced quantities of both. Too much assertiveness, and you may destroy your work, while too much attentiveness and you could go out of business. Soulcraft is being able to apply just the right amount of each.

I believe that Crawford’s categories can extend beyond the realm of manual labor. His descriptions really got me thinking about how they would apply to software engineering. What is “assertive” and “attentive” software engineering? I’ve always felt that programming brings with it a sort of spiritualistic satisfaction. Now Crawford gives me tools to examine programming in new ways.

The assertive programmer is one who makes new or improves features. In this case, the engineer is on the cutting edge of product development and essentially makes something out of nothing. He or she “asserts” him or herself by creating new code or by combining or expanding infrastructure in new ways to make new things.

This type of programming is fun and rewarding because you get to see something made. You also get a craftsman-level satisfaction and the pride of ownership knowing that the software is built well and that other people find it useful. Contrast this with the attentive programmer. The attentive programmer generally works with software that he or she didn’t make. And is given a problem which has perplexed others.

I think that I am more akin to this type of programmer. I am a Database Architect who works on the OPOWER Scale Team. On the Scale team, we handle problems that reflect scalability issues with the software. We also get a first crack at designing new features that are scalable out of the box, and I think that puts us firmly in the assertive camp.

I want to reflect more upon our reactive scalability work because I think this sort of work takes a different mindset and possibly a different software engineering approach. To be an attentive programmer you almost have to have a Zen-like disposition and “listen” to the information that you find. You may think that you understand the problem but you must verify if the hypothesis is true. And if it’s not, you must reexamine your assumptions, read the data again, and develop a new hypothesis to test.

The attentive programmer knows that truth is the path to the right solution, and that’s exactly how we approach problems on the Scale team. I think the key to a good scalability engineer is attentiveness and checking preconceived notions at the door. This type of programmer needs lots of experience and must be open to a wide range of ideas where hunches based upon past experiences or fresh perspectives can be quickly dismissed or followed based upon the facts.

To me probably nothing is more satisfying than taking a piece of code, no matter how complicated, that no one has been able to make go fast and making it fly. I’m always ready for the next problem and always wondering what I’m going to learn from that puzzle.

Are you an attentive or assertive programmer, or some combination of? Why? Thinking about how you approach software engineering may help you raise your game and bring with it the satisfaction of rising above the status quo.

We focus on how we build the product to ensure it supports the business for the long haul. We take this part of our job very seriously and are constantly working to improve how we work. We use some of the obvious practices from agile like scrum teams, story pointing, and short-term iteration planning to keep ourselves on track with the functionality we need to build over a three iteration horizon. A more out-of-the-ordinary practice is having test team members fully embedded on our scrum teams. The software engineers in test work alongside software engineers building functionality, and a task is not considered finished until it is tested through our automated regression suite.

We do team retrospectives almost every iteration and keep a backlog of tasks to work on during regression weeks to keep improving our tools and process. When it comes to coding, we have formal code reviews every three weeks as part of the iteration cycle, and ad hoc code reviews happen during the build. Some of our reviews can get heated (though not personal), and everyone knows that the intent is to build the best product we can. The reviews are also a lot of fun.

The point of all of this work is to get everyone in line with building a product for the future. OPOWER is only four years old, and our product has been live and generating energy efficiency gains for over three years. We are pushing ourselves to build new products to improve those efficiency gains and to positively impact the environment in other ways. Our clients are pushing us to deliver more functionality to them faster and faster, and in the meantime, they keep giving us more and more data to learn from and use. If we took a short-term approach to our jobs as engineers, the platform we are building would collapse in on itself, and that would do no one any good.

Wil Shipley, of Delicious Monster fame, wrote an interesting article recently, drawing an analogy comparing building software companies to either farming or mining. When you mine, you take out a loan, purchase a ton of equipment, hire people to dig, strike gold, and profit. When the gold runs out, you retire. On the other hand, when you farm, you take out a loan, purchase some equipment and seed, and then you plant and tend. If things go well, you reap some reward during the harvest season, after which you repeat the cycle, reinvesting your money back into your land. If you are an adept farmer, you may figure out how to rotate crops and make the most of your land all year long, thus increasing profits. In his analogy, farming is a long-term approach to building a company, and mining is what you do with a social network for dogs — build it to flip it.

At OPOWER, this analogy is particularly apropos, because of our first bottom line. What is more sustainable — farming or mining? Clearly, OPOWER is a farming company, and the engineering team is working to ensure the harvest comes in season after season.

We’re looking for “Better Ways To Make Better Software”.

Agile software development is a great way to make good software that satisfies real needs and creates real value to those who build and use that software. Everyone on the team is concerned with the quality of the product. However, sometimes it can be an overwhelming task being the team member whose job it is to insure that the quality of the product is sufficient.

DCAST was created to develop a community of those interested in agile software testing who are also interested in testing agile products more effectively and more efficiently. By sharing our hard won knowledge we can all help make each other’s lives easier and more productive.

If you’re interested in sharing how you have raised the quality bar of your agile processes, we would love it if you were a part of this community. If you’re interested in learning (or want to all learn together), you’re even more welcome.

We’re going to be meeting again in about a month. Come check out the group if you’re in the DC area.

When you apply to OPOWER, we want to see how awesome YOU are in your natural habitat.  Our homework helps us do that — by giving you a project that you can complete on your own schedule, in your own environment.  Maybe you’re a “early-morning coffee and Eclipse” person, or an “up-all-night with pizza and vim” type.  Whatever it is, we want to see THAT.

Yes, we also have a more conventional part of our interview process.  You’ll talk to us on the phone and in person.  We’ll ask you about your current projects, make you answer some tough technical questions, and write code together on a whiteboard.  But the homework is our way of asking you to show us what you’re like, on your own terms.

We take our homework pretty seriously, and hope you do, too.  Think carefully about the problems.  Explain what you did (and what you didn’t do) using comments and READMEs.  And give us your best code — polish counts more than features.  Code that’s lean, appropriately commented, and thoroughly tested is going to earn high marks, guaranteed.

Our application process is hard.  We know, because every single person in OPOWER’s engineering group has been through it.  We do it this way because we want a team full of great people that can produce reliable, real-world solutions.  It may be more work for everybody up front, but we think it’s produced great results.  And we think you’ll agree when you join us.

One thing I’ve noticed about successful candidates is polish. This isn’t a word thrown around when discussing developers a whole lot, but I’ve found it to be a strong indicator of a true software professional. The OPOWER application is more than just a résumé; it’s a written quiz and a coding assignment as well. Sloppiness at any point is a negative; it shows that the candidate just doesn’t give a shit about getting the job. It’s not a deal-breaker, but it has a huge effect on how excited we get about a candidate, and it could be the deciding factor between two otherwise identical candidates. It’s our version of “hire the better writer.”

So, what do I mean by “polish”? Here’s a few examples:

• Make your code better than you would “at work.” Don’t make me download the JUnit jar and type javac `find . -name *.java` to compile your code; give me a pom.xml or Ant file with Ivy support.
• Send PDFs instead of MS-Word; they are so much easier on the eyes (and processor). This is dead simple on OS X, there’s some free options for Windows, and there’s always LaTeX (which will definitely earn you some geek cred
• Don’t send me the functional spec of your last project and call it a résumé. Use bullet lists, short sentences, and formatting to create an easy-to-scan CV.
• Format your writing; compare this, to this. Even though they both are excerpts from correctly-answered quizzes, which one is going to be easier and more engaging for me to read?
• English: do you speak it?. Being able to communicate effectively using the written word is absolutely crucial for being a good developer, especially at OPOWER. It doesn’t need to be your first, or even best, language; you just need to wield it as effectively as you do Java

A lack of polish won’t get you refused, but putting an extra shine on your application can make you stand out quite a bit…