What Does “Data-Driven Company” Mean for a Developer?


At Pivotal Software, we frequently pitch about a virtuous cycle of of data-driven app development where:

  1. A cache of data is collected and stored.
  2. A team of analysts or data scientists discovers an insight or optimization opportunity
  3. This affects the app development which generates more data.
  4. The company continues through the cycle more and more agilely.

Wash. Rinse. Repeat.

However, the story above is more of an analytics story than a developer story.  This became clear to me while sitting with some crack developers at dinner at the Spring One 2GX conference.

One of my table mates asked me, “I understand what Pivotal is saying about the virtuous cycle, but what exactly happens between analytics and apps stages?”

He was asking for the requirements and discrete concrete actions that would result from a data driven company choosing to deploy an action based on insight’s discovered in step 2. In other words: “Hey marketing guy, nice story! Show me the code!”

While highly glossed over in the typical telling of this story, the role of developers in a data-driven company is extremely important.

First of all, this cycle often begins with app development.  In fact, application logging and transaction histories are usually the initial cache of data from which insights are initially discovered. Before an app in question is deployed, a company may have no ability to even measure whatever characteristics they are interested in tracking.

Next, the insights generated by analyzing this data can result in a wide range of new requirements such as:

  • Additional data collection from app or user behavior, or incorporation of other data sources
  • New app behaviors based on opportunities discovered, such as adding cross-selling functionality
  • Optimization of the app’s behavior or user experience
  • Adding event detection of cases, patterns, or exceptions using algorithms refined by analysts – perhaps allowing humans to be alerted to situations that need special handling.
  • Automation of app behavior using algorithms refined by analysts – going further than detection and intervention.
  • Net new apps are created to address new uses cases defined as a result of analyzing this data

The data scientists at Pivotal Data Labs are often fond of saying, “Power point is where insights go to die.” This is the fundamentally important role that developers in a data-driven company play: they turn insight into innovation.

Or paraphrasing a quote from Haiyo Miyazaki’s The Wind Rises:



For more ideas of how developers can help their companies become “data-driven companies”, check out the Big Data @Pivotal blog for ideas.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: