At Opower, we recognize that our team members are as complex as our customers, and our ability to create brilliant products depends on our ability to reimagine what’s possible. That’s why, twice a year, we shut out the world and focus on harnessing our collective talent to improve everything Opower—these are our Innovation Days. Similar to hackathons and 20 percent time, they go even further to promote community, creative thinking, and freedom from organizational hierarchy.

Innovating, on the shoulders of giants

In recent years a lot has been made over Google’s approach to innovation—20 percent time dedicated toward projects that further the company mission—which was similar to the approach 3M took as far back as 1948 with 15 percent time. Before Google, HP was encouraging employees to use 10 percent time—after lunch on Fridays—to work on whatever they wanted.

Some amazing products have come from those companies and their approach to innovation outside of normal product initiatives:

• Gmail, AdSense, and Google Earth
• from 3M—a sandpaper with rearranged particles that easily cuts through metal, known as the Cubitron II
• one HP employee apparently used his 10 percent time to help invent HTML

These products revolutionized how people around the world go about their work.

The format of those innovation initiatives was generally geared toward employees with technical abilities, and the actual percent time was not always formalized, either. At Google, if you wanted to work on something, you just did—there was no paperwork, no manager approval.

Many other companies take a different approach to innovation by hosting hackathons—companies like Yahoo!, Foursquare, Salesforce, and Yelp. This approach encourages public participation in a contest to build on top of a company’s platform. Many awesome products have come out of these events over the years, as well.

What we’ve learned

As we continue to improve on our Innovation Days, we’ve gleaned a couple key insights from these giants of innovation:

• innovation can come from anyone and anywhere, and it’s up to us to recognize opportunity and champion it
• without facilitating a time and place for unconstrained work, it’s incredibly difficult for people to make innovation happen on their own time without support and encouragement

As such, our guiding principles are simple: remove constraints and facilitate.

Making time for what could be

The only rule of Opower’s Innovation Day is that folks must present their projects at our Innovation Fair—a science-fair style event where everyone gets to share what they worked on and check out other folks’ projects. The rationale, from a behavioral psychology perspective, is that when people publicly proclaim their intentions, they make themselves accountable to people who trust them to follow through. We want our team members to not only pursue their best ideas, but realize their ideas in ways that allow their projects to come to life for others.

To remove constraints:

• everybody is included, and people can work with whoever they wish on whatever they want
• internal meetings, client meetings, or other regular commitments are not allowed

And to facilitate:

• we’ve created a simple digital space for folks to share ideas and join in groups
• about a week prior, we run Pitchathons in each office where anyone can propose and share ideas

Working with participants across three offices in two countries has presented its share of challenges. For years, we tried coordinating the events so that kickoff and summary speeches coincided, but we realized that allowing each office to embrace its own time zone, location, and unique culture was the best way forward.

Lately I’ve been thinking about the notion of “beautiful code”, and finding it to be less and less satisfying as a goal. First off, beauty is elusive: everyone seems to have a different concept of what it means (and except for mine, they’re all wrong). And second, even when you can label a piece of code as “beautiful”, it’s not at all clear how that translates into other desirable characteristics like performance, maintainability, and so forth.

In my reading, I came across a great article by Rebecca Wirfs-Brock [1] in which she discusses the notion of “habitable code”. This idea isn’t original to her (she credits Richard Gabriel in Patterns of Software [2]), but she provides a great explanation of the concept, and goes on to talk about its role in complex systems.

Paraphrasing some key points …

• Clarity for its own sake is an elusive goal. In my own experience, if I strive really hard to make some aspect of my code crystal clear (say, a challenging algorithm or data structure), it’s at the expense of clarity in some other area. It’s a bit like a game of Whac-A-Mole. Just as you can’t write code that’s infinitely flexible in the face of unknown future requirements, I don’t think you can write code that’s infinitely clear to an unknown future audience.
• Beauty becomes a constraint for future authors. It’s hard to extend or update excessively beautiful code without spoiling some of its inherent loveliness. An analogy to industrial design — the MacBook Air achieves a certain level of elegance by ruthlessly eliminating functionality like replaceable batteries, integrated ethernet or a DVD drive. How difficult would it be to add that stuff in later without compromising the original aesthetic? Probably very.
• Habitable code makes its structure and intentions easy to understand. Structure, sure, but as a professional developer, the “and intentions” part seems pretty important. You can read code to understand what it’s doing, but it’s not always obvious what problem the original author thought it was solving. This is a hard thing to do right — the best I’ve been able to do is document the problem being solved, and make the structure of the code appropriate to the solution. Code reviews are a huge help in identifying areas that need improvement in either direction.
• Habitable code makes developers feel at home. At Opower we have a pretty good-sized body of coding conventions, some of which are automatically enforced using tools like Checkstyle and PMD. To new hires, it sometimes seems totalitarian (it did to me). But once you’ve been around awhile, you notice a funny thing: you can go almost anywhere in our source base and see things done in a similar fashion. This helps us transition from artifact to artifact more easily, and reduces the overhead in sharing code. Habitable, indeed.

Just like any other principles, these can be reduced to absurdity. Giving up completely on clarity is obviously not the answer, nor is making your code horribly ugly so that future maintainers don’t feel bad about rewriting it. But it’s interesting to think about where habitability differs from clarity and beauty, and what it demands of us as developers.

So … who’s inhabiting your code today? Whose code are you inhabiting? How is that going to change in a month? Six months? A year? And what are you doing to make each other feel welcome and at home?

References

[1] http://www.wirfs-brock.com/PDFs/DoesBeautifulCodeImply.pdf
[2] http://dreamsongs.net/Files/PatternsOfSoftware.pdf

Opower’s evolved rapidly in the last 24 months and nowhere is that more true than in our code base.  Two years ago we had 2 flagship applications that shared model objects, DAOs and a handful of utility classes with one common JAR.  Since then we’ve expanded to the point where (not even counting all the Ruby or Scala stuff) we’re managing:

• 4 WAR applications
• 33 JAR libraries (1 open sourced @ https://github.com/opower/jpile)
• 14 “pom artifacts” that define groupings of projects and organize dependencies, versions, etc

Not all of that growth was brand new, however.  As we expanded our product lines and thought more about the best way to scale our core features we invested heavily in the decomposition of existing WARs and improving the “librarification” of our code.  In addition to supporting scaling, improving the modularity of our code base also supports a move towards S.O.A. and improves the velocity of our scrum teams.  After all, it’s a lot easier to focus on developing the stories in your iteration if you don’t need to worry about stepping on another team’s toes.  That’s more easily achievable when your team aligns to one specific code artifact.

A problem we quickly ran in to as we started decomposing WARs and JARs was the creation of a lot of redundant spring configuration files– especially in an integration test context.  Here’s a crude picture of our initial state following some decomposition:

One WAR depending on two JARs with duplicated spring bean configuration

Note that the WAR includes the two JARs and wires together beans declared in the JARs with an applicationContext.xml defined in the WAR.  We thought it would be a good idea to adopt the policy of “keep the tests close to the code that they stress.”  That’s a sensible policy, but resulted in duplication of context configuration found in the WAR for the purposes of spinning up Spring contexts during integration tests.

Reducing that duplication was done by continuing the theme of decomposition and applying it to the Spring context as well.  For example, we adopted the policy that a JAR should export a sensible “default wiring” of the classes that it encapsulates.  We adopted naming conventions to facilitate the need to either explicitly or implicitly include a Spring context snippet.

• If a JAR includes a context file ending in “-spring.xml” then it’s automatically included by any WAR that declares a dependency on that JAR.
• On the other hand, if a JAR includes a context file ending in “Context.xml” then it is only brought in to the WAR’s context with an explicitly declared inclusion of that file.

Here’s what the picture looked like when we migrated to that model:

One WAR with two JARs, but no duplicated bean configuration

Integration tests that lived in our various JARs liked living in this model because they could always import “*-spring.xml” and their integration test context would come up and run just fine.

This worked well until we encountered some situations where one JAR started depending on another JAR.  We wanted to ship sensible Spring context defaults from both JARs, but because of the “*-spring.xml” rule we ended up needing to duplicate all of the application properties that were needed to wire up the beans in the depended artifact from within the src/test/resources directory of the dependent JAR.  Here’s a picture to explain what I mean:

Application properties starting to proliferate

This problem just got worse as you got more and more descended from a given library– for example, if jar-A depends on jar-B depends on jar-C and they all ship with their own “*-spring.xml” files we needed to define properties to configure C within jar-B and we needed to define properties to configure jar-B and jar-C within jar-A.  It was getting hairy.

To solve this we adopted a new standard that basically amounts to “provide sensible defaults for our sensible defaults”: any bean in a “*-spring.xml” snippet that can be configured with a PropertyPlaceholderConfigurer must provide a sensible default for that property.  So a bean definition might have looked like this before:

    <bean name="foo" class="opower.Foo">
        <property name="locale" value="${locale}"/>
    </bean>

After:

    <bean name="foo" class="opower.Foo">
        <property name="locale" value="${locale:en_US}"/>
    </bean>

The defaulting of application properties allowed us to eliminate 90% of the boilerplate test properties and code that we would have otherwise needed and allows for a greater abstraction of thought when a developer decides to pull in a library.  Thus, our current setup looks something like this:

Defaulted properties eliminates almost all configuration

A couple of tricks to note when using this model:

• There’s a big difference in classpath context scanning between “classpath:/*-spring.xml” and “classpath*:/*-spring.xml”.  The first (with just one star) will pull in all the files that match “*-spring.xml” in the first directory on your classpath that contains any file matching that expression and then stop.  The second version (with classpath*) will do what you’d expect….it pulls in all files that match “*-spring.xml” from every directory in scans in your classpath.
• Application property defaulting accepts literals after the colon, so it’s safe to do something like “${path:file:///blah}” to default the property to the string “file:///blah”
  You can also change the default string separator in the PropertyPlaceholderConfigurer bean so your defaults look more like this: “${path?file:///blah}”  (using a “?” as a separator)
• It’s tempting to eliminate application.properties wherever there’s a default, but we avoided that for documentation reasons– we wanted to provide the implementers and users of our software with a one-stop location to discover and understand the configurability of our applications.  Thus, we adopted a standard that even if an application property is defaulted in a JAR’s spring context snippet, we still explicitly declare that property in a WAR’s top-level application.property and document how that property affects the system.