nancy header

Why data science is a big deal for the utility industry: 9 insights from Opower’s VP of Analytics

Data is the fuel that drives Opower’s customer engagement platform. Every day, our software pulls in hundreds of millions of meter reads, weather measurements, and behavioral data points, and automatically processes them to generate timely, actionable insights for utility customers around the world.

How do we ensure our data insights are the most powerful in the industry? It all starts with data science.

To learn more, we sat down earlier this month with Opower’s VP of Analtyics, Nancy Hersh. She shared a deep look into Opower’s approach to data science, why it’s transforming utilities’ understanding of their customers, and her predictions for the future of the field.

Opower: Let’s start by talking about data science. The field has exploded onto the national scene; from Silicon Valley to political campaigning, it’s become core to the way leading organizations make decisions. What’s the driving force behind the rise of data science?

Nancy Hersh, VP of Analytics: I think there are two major trends at play. The first is obvious: there’s simply a lot more data available than there was 10, 15, 20 years ago. At an individual level, people are doing things, particularly digitally, that are imminently trackable and storable in a way that was never possible before. Our digital footprints are extremely well defined. And on a broader scale, there’s a proliferation of sensors that are passively capturing all kinds of new data about our world.

The end result is that there’s been an explosion of data over the past few years, and it’s just going to get bigger.

But all of that would be irrelevant if not for the second trend: we’ve revolutionized how we store, access, and analyze data. Our ability to keep all the information we generate has followed Moore’s Law, doubling and doubling every two years. And our ability to work with that information has fundamentally changed, mostly thanks to distributed storage systems like Hadoop. In the past, we were restricted to analyzing data in a serial fashion, on a single machine. Now we work in parallel, processing data across a whole bunch of systems at the same time.

O: Where do you think we’re headed? How do you see businesses tapping into data science in the near future?

NH: I think we’re just beginning to understand what’s possible. But broadly, I think we’ll see two seemingly contradictory, yet actually complementary paths emerge over the next five to ten years.


One of them is essentially the trail that Uber blazed: manipulating data in complex ways to fundamentally change some aspect of our lives. There will absolutely be more Ubers moving along that path in the coming decade. But there will also be a lot of companies that — trying to become the next Uber, the next Amazon — attempt to do too much, too fast, and they don’t get where they want to go because they’re being more sophisticated than they need to be. They’ll try to go from zero to 60 instead of zero to 20, 20 to 40, 40 to 60.

The contradictory-yet-complementary path that we’ll see companies follow is using data science in smart, simple ways not necessarily to reshape our lives, but to meaningfully improve them. There’s still a ton of low-hanging fruit that data scientists have yet to pick.

O: What will that look like in practice?

NH: I see a world where all the information around you is more relevant to that particular moment in your life.

Today, you have to actively seek out the data you need. You read the news. You look at Carfax. Tomorrow, that data will come to you automatically. It will seek you out when you know you need it, or even before you know you need it. It’s a world that realizes that age-old marketing adage: getting you the right information, at the right time, through the right channel.

O: These trends will obviously reach the utility industry, too. Smart meters have unleashed a tremendous amount of customer data. Are utilities putting it to use yet, personalizing customer communications based on all these new information streams?

NH: Well, historically, utilities’ messaging and marketing has been one-size-fits all; they haven’t done a great job delivering relevant insights when their customers need them. But today, we are starting to see a shift. The companies that use what they know about their customers to deliver great experiences — the Ubers, the Amazons, the Airbnbs — are raising the bar on great service. Utilities recognize that, and they’re beginning to deploy more advanced segmentation and targeting techniques to personalize their customer’s experiences.


They’ve got two main types of customer data to work with. First, there’s profile information: who you are. This is demographic and psychographic information like your age, income, geography, and so on. The second is behavior: what you do. How much energy do you use? Do you participate in energy efficiency programs? Do you log on the web? If so, where do you go?

To the extent that utilities are doing segmentation and targeting today, it’s almost all based on profile information. There is very little segmentation and targeting that’s based on behavior. But guess what? Behavior is actually much more predictive of what you’re likely to do in the future than profile data is. If utilities want to deliver truly relevant information that’s engaging and helpful and gets you to take action, they have to tailor their message for how you live your life.

O: Which is why Opower’s Data Science team built Load Curve Archetypes.

NH: Right. We took hundreds of thousands of load curves — which represent customers’ energy usage over a 24-hour period, their day-to-day behavior — and used machine learning algorithms to find hidden patterns in the data. The goal was to build an entirely new kind of customer segment that describes not who you are, but how you use energy.   It provides a way to get a “window into the home” and has been referred to as an energy personality.

O: And you found that there are five main energy personalities, right? Is there something behind that number?

NH: Yes and no. Yes, in the sense that our algorithms identified five distinct, clearly defined groups of energy consumers. Could the Data Science team have split people into eight, or ten, or 18 segments instead? Potentially. But the segments would be less distinct. And imagine if there really were 18 energy personalities. As a utility program manager, could you really develop unique marketing campaigns to reach each of them? What you’d end up doing is re-bucketing them — these three look similar, these five look alike, and so on.

load curves_600px

It’s about finding the right balance. You want to define segments in a way that reflects the nuances of the data but doesn’t become unwieldy. And after a lot of research, the magic number turned out to be five.

O: It’s the same reason you wouldn’t create segments for every 100 square-foot difference in customers’ home size, for example. That would become intractable.

NH: Exactly.

At the same time, saying that there are five major energy personalities doesn’t mean that there are only five major types of energy customers. After all, Load Curve Archetypes represent just one segment. Customers belong to a bunch of other segments, too — demographic and behavioral. What gets really powerful is when you use all of them in combination. You get that holistic view.

Take demand response programs, for example. Say I’m trying to create a segment of customers that will make a big dent in peak demand. Load Curve Archetypes are a great starting place: I can identify people who use most of their energy during the afternoon. I’d also target customers who use the most energy to begin with. If I want to install devices in customers’ homes, I’d only want to target homeowners. And I’d restrict it to single-family residences, rather than multi-family.

You keep going like that, and what you end up with is a very thoughtfully targeted group who can receive very personalized communications. And that leads to powerful outcomes.

O: Touch on outcomes for a moment.

NH: Well, for customers, this is the beginning of the end of junk mail. The ads you see and messages you get will be increasingly relevant to your life. And in the utility context, that means you’ll be able to save more energy and more money on your bills — through smart outreach about energy efficiency, demand response, time-of-use pricing, and so on.

For utilities themselves, data-driven segmentation and targeting is an incredible way to put smart meters to work. Nearly half of American homes have AMI, but most utilities aren’t using the data it generates to add value for customers. Those that do are seeing higher customer satisfaction, more effective demand-side management, and lower costs to serve.

O: Let’s talk about the future. Opower manages 40 percent of America’s residential energy data. What other applications can you imagine for that information?

NH: In the immediate term, our data scientists are doing a lot of work on algorithms to infer customers’ heating and cooling setpoints from smart meter data alone. Heaters and air conditioners use more energy than anything else in the home, and there’s a huge opportunity there to help people become more efficient and save a lot of money — without installing expensive devices.

wide shot

When it comes to machine learning, we have a lot of other irons in the fire. We’re spending time on usage disaggregation: decomposing customer’s energy use from a single blob into it’s component end uses.  We’re also investigating propensity modeling — using algorithms to better understand what actions utility customers are most and least likely to take in the future.

Looking further out, one data science application I can imagine is using Load Curve Archetypes to carefully reshape utilities’ entire load curves. In theory, a demand response program manager could identify a very large group of customers with different load archetypes — 50 percent of this type, 30 percent of this type, 20 percent of this type — motivate behavior change at a certain time, and, in aggregate, shift demand across their service territory. As the rise of renewables makes our energy supply more variable, that kind of approach could be extremely valuable.

O: Last question: outside the utility space, what data science projects are you most excited about?

NH: I think the notion of personalized medicine is really cool. I won’t pretend to know more than I actually do, but we’re all individualized machines, and all of us have unique genomes. A one-size-fits-all approach to medicine — which is what we have, particularly when it comes to prescriptions and procedures — doesn’t really make much sense. People process chemicals differently, so how we help them get well should happen differently, too.

O: The predictive aspect of it is interesting, too — the notion of propensity modeling, but for illness.

NH: And there’s a little bit of that already happening, right? Physicians make predictions about what will happen to you based on your family history. But they’re not doing it based on your genome, not just yet. That’s going to be big.

Want a deeper dive into Opower’s data analytics? Check out the algorithms that power Load Curve Archetypes.

Opower Author
Opower /