This month, David Socha discusses some of the encouraging and the less encouraging findings of a recent utilities analytics survey and comes to some surprising conclusions about the future of data & analytics.

A survey of North American utilities1 released last month suggested that in 2017, some had begun to take analytics a little more seriously. That’s good news you’d think, right? Headlines in the survey include the fact that there’s a 42% increase in utilities developing predictive analytics capabilities and that 63% have a data quality program in place. Even better, 62% plan major analytics projects in the coming 24 months. Those are certainly good things. But there’s more. And a lot of it’s not so great.

One million dollars

The survey also tells us that 47% will invest less that one million dollars in analytics this year. No, really. Less than a million. I know this might seem controversial and even counter-productive from someone who has a career (and a mortgage and a daughter in full-time education) reliant on businesses investing in analytics, but... that’s not what anyone could seriously call investment, is it?

How long ago was it that the world laughed as Dr Evil planned to hold the world ransom for “one million dollars”?2 I’ll give you a clue: it was over 20 years ago. (OK, that was quite a big clue. It wasn’t really a guessing game.)

To be fair, 20 years ago I would probably have applauded a utility investing a million dollars in analytics. But today, given the digitalisation agenda; the availability of data; the pressures on asset businesses to improve performance; and perhaps most tellingly, the proven value of analytics in so many industries, any utility larger than a Stadtwerke that’s investing less than a million dollars in these transformational capabilities isn’t going to be a business for much longer. Don’t say I didn’t warn you!

Do what thou wilt3

One more statistic tells us that 72% of those polled have no governance process in place for OT data. To me, that suggests those OT folks can still pretty much do what they want, citing business criticality and obscure complexity and all kinds of black-artsy arguments as to why those pesky governance types should just leave them alone to do their job.

Yes, we could spend a lot of time discussing how OT data is not the same as IT data and how there may or may not ever really have been the same data governance requirements in the OT world. But – however slowly and tentatively – the IT and OT worlds are converging, for all sorts of good reasons. As that convergence continues, having no data governance process for the OT part of the data will soon become unsustainable. So far, it seems 72% of us are still to subscribe to that view.

Data integration? Yawn

 I’m going to end on something of a high note, despite the subtitle above. Before that, let’s recap. So far, I’ve talked about a lack of serious investment and lack of governance. Both, you would think, impediments to an analytics programme that will deliver real value.

However, respondents actually cite data integration as the top impediment to success. Not only that, they state that availability / capabilities of analytics tools is the issue least likely to be impeding progress. This is great news!

The latest front-end analytics and BI tools are important. But what’s their value if they have no reliable and accessible data to perform their magic on? Zero. No matter how smart, how pretty, how cool they might be. And it’s great to see that utility companies are beginning to notice that in order to deliver value from analytics, they need integrated data sets to perform those analytics on. Because you know what? Integrating data in the best ways to enable analytics is a problem that has been solved. For a long time. It’s usually called something like…whisper it…a Data Warehouse.

The right foundation

While it may have been fashionable in recent years to suggest that the Data Warehouse is dead, that argument has already been lost. As an example, I give you Andrew Ng, founder of the Google Brain project; VP & Chief Scientist of Baidu; Co-Chairman and Co-Founder of Coursera; and an Adjunct Professor at Stanford University. I put it to you that Andrew knows what he’s talking about. And in a recent lecture on Deep Learning and AI, he discussed the complexities of gathering data from many sources to enable all that cool DL and AI to do what it does. His advice was clear and simple:

“If your boss asks you, tell them that I said build a Unified Data Warehouse”

 Doesn’t get much simpler than that, huh? And this is why I believe I can end on a high today. If, as the survey suggests, utilities have identified that data integration is their biggest problem in delivering value from analytics, then we’ve taken the first steps in the right direction. Yes, we still need to address investment levels. Yes, we still need to think about that OT governance thing. And there will be other issues along the way too. But hey, it’s a start. In fact it’s better than that. It’s the right start. I look forward to helping you along the journey.


  1. Bridge ask (entirely reasonably it has to be said) that we quote the source explicitly, ie:
    BRIDGE Energy Group® – 2018 BRIDGE Index™
  2. For those unfamiliar with the quote, it’s from the film Austin Powers: International Man of Mystery (1997)
  3. Since I’m being somewhat controversial this month, that one’s a quote from Aleister Crowley, notorious weirdo, occultist and subject of this great song