Are the Best Decisions Instinctual, or Data-Driven?

This post originally appeared on LinkedIn's engineering blog.

How do you develop products? Do you blindly trust your instinct, or do you hide behind a mountain of data to make decisions? Sure, these are two extremes, but I am always perplexed by the polarizing aspect of what is, in fact, a false dichotomy. Why do we have to choose one at the exclusion of the other? 

Instead, I argue that you need both. Building great products requires you to be a master of instinct and data-driven decision making. Ignoring either one will inevitably lead to failure. You may have one-off successes along the way, but you won’t escape the inevitable.

For the sake of argument, let’s first assume that you can build great products and innovate by experimentation alone. Within this approach, all innovations or product iterations would be tied to the ability to experiment on them. When I was growing up, a Walkman was a great product and innovation! Today it has disappeared, and my kids only experience it as a throwback device in Guardians of the Galaxy. After the first model, the product got better and better. Different companies fine-tuned its shape, buttons, headphones, weight, etc. It just got better and better, only to finally disappear in the wake of digital MP3 players.

Its coup de grace was the advent of the iPod. If the iPod is the “natural” extension of the portable music player, could you have experimented your way from the Walkman to the iPod? I argue, no. In the space of portable music players, a Walkman is and always will be a Walkman. The parameter space (the set of things you can experiment on) that you could explore starting with the Walkman would never lead you to an iPod. To paraphrase scientific philosopher Thomas Kuhn, the iPod is a paradigm shift.

At the other extreme, relying on an instinctual drive is similar to randomly drilling holes in the desert in order to find water. You may be lucky and actually find water, but more often than not you won’t. Instinct will either tell you to keep on digging forever at a given place or to jump to another random spot.

But if you start seeing the desert as an information rich environment – start noticing the contours of the land, or where certain plants are growing – it will provide data for your instincts and tell you where water is likely to be. At that time, you will start your methodical exploration by drilling experimental holes and tracking the results that either confirm or refute your hypotheses.  

Albert Einstein once said: “Imagination is more important than knowledge. For knowledge is limited to all we now know and understand, while imagination embraces the entire world, and all there ever will be to know and understand.” Instinct is of similar nature to imagination, while experimentation is more akin to knowledge. Without working from a gut feeling you can’t wildly innovate, and without experimentation or A/B testing you can’t really fine-tune, learn and grow your knowledge. From an optimization perspective, an A/B testing platform is essential to achieve local optima. At times, however, you will need instinct or gut reaction to take you out of a local optima to potentially achieve a global optimum.

There is a symbiotic, iterative loop between instinct and experimentations. Experimentation powered by a flexible A/B testing platform (to validate hypotheses) will ultimately fine-tune your instinct over time.  Instinct in turn will inform hypotheses your platform will test.

Thanks to Ya Xu and Craig Martell for having taken the time to review earlier versions of this Post.

The CIO and CMO: In pursuit of data symmetry

It is now accepted that the CMO and CIO positions are inter-dependent in the business of managing customer data in pursuit of demand and growth. The simple notion is that as more data signals can be applied to segmentation and media targeting it follows that the CIO needs to create a platform informed by, and deployed by, the CMO for the application of that data in the real world. We used to call it 'the single view of the customer.' Today we are as interested in the consumer we would like to know as much as the customer we do. 

The fundamental issue, albeit rarely described as such, is the creation of internal data symmetry; the central purpose of which is to enable multiple stakeholders such as marketing, sales, media, research, PR and others to work collaboratively and across functions while achieving multiple business and marketing metrics (sales, distribution, brand health, attitudes, usage, etc.).  It also forces key questions; is brand health a metric to be maximized or a minimum constraint while achieving sales KPI's? 

Internal symmetry allows for external asymmetry that in turn creates competitive advantage by knowing something about a customer or even an ad impression that is not known by competitors or others in the marketing supply chain, typically those that sell on shelf and on air access to the customer. The value this presents is obvious, if you get 100 calls from Spot A and 500 from Spot B, who is the last person you tell? 

Executing against the opportunity is less obvious and requires prioritizing. An easy way to think about this is to put data into two buckets: 

• data you own- typically about the customer you know
• data you rent, buy or accrue as the consequence of other actions - typically about the consumer you would like to know 

The first bucket; CRM and loyalty, direct transaction data (ex-factory or point of sale), site side analytics, e-mail and other databases are CIO territory. 

For his partner, the CMO, the focus is on third party data and community and campaign level information that can become appended to and conjoined, multiplying the value of bucket one, and, critically the application of all the data to segmentation and media trading - the sources of growth. 

Campaigns (and to an extent communities) by their nature are episodic and their data exhaust is often of more ephemeral value especially as few constants are available and the data tend to be less clean than the CRM or transaction file. Campaign and cookie data degrades over time, more so over multiple matches. Third party data is, for a price available to anyone. Neither require the level of corporate or personal security as bucket one. Publisher data, including the first party giants like Google and Facebook, is of potentially high value but more to seller than buyer if advertisers are denied third party ad delivery and tracking. 

In light of this brief, and admittedly incomplete analysis, it's interesting to consider which parts of the marketing tech / ad tech cloud (many just a minefield of acquired tools re-engineered and re-branded to create a complete stack with no guarantee of best for purpose in all components) that the enterprise should control. Some companies, particularly those with high value and abundant first party data that are in direct control of end user distribution want to own everything. That's not a bad idea as long as you allocate resources to deal with the inevitable breakage in components. Unlike diamonds, software is rarely forever. 

For others it's all about the owned data (see bucket one above) and the enterprise DMP (data management platform) that secures, harmonizes and organizes that data and creates a safe environment for it to co-mingle with the contents of bucket two. Aligning the enterprise by all of division, brand and geography on the DMP is THE key to internal data symmetry. Having brands disconnected from categories and disconnected again from countries significantly reduces the potential of operating across the portfolio. 

Of course, the application of data is most refined, and most developed of all closest to the point of a binary event like a sale. This only scratches the surface of data driven marketing. When so much of an advertising accelerated economy depends on performance at the broadest as well as the narrowest points of the purchase funnel, data is a key ingredient of success. At those points the broader attribution techniques developed by the agency community and specialists can actually be used to identify and price the most valuable third party data sets and apply using programmatic and other methodologies. 

Perhaps then it's better to talk not of convergent roles but about the complementary relationship between the CIO and CMO and the agencies that take the data to market. Inevitably we believe that media agencies have an important place in the supply chain, or perhaps, more accurately, the demand chain. 

This is not a question of contracts which has been a theme of 2015 nor is it a matter of system compatibility- praise the API. How those contracts are operated and connections executed to create a fluid stack around the DMP is most important and is a function of multiple factors: 

• Who has the greatest exposure to multiple stack components in both similar and different categories?
• Who can compare performance across category and geography?
• Who can on board new or replacement modules most nimbly?
• Who can take a horizontal view of the integration of the client DMP with digital asset management systems, a plethora of search, social and display bidders (often with their own DMPs), ad serving and campaign management platforms, social listening tools and the rest?
• Who has the most real world experience of applying systems and data to live markets? 

Agencies that have a genuine engineering group are good at this; agencies that combine that with their own original data sets sourced from their own technologies and original data collection are best of all. What's more it’s agencies that see every media channel, all the campaign level data and, agencies that are attribution and allocation agnostic that are best placed to serve marketers. For the most part, tech that is operated by media sellers is less likely to share this quality. 

The challenge for the CIO is to collect near perfect data and simultaneously keep it safe while unlocking it for CMO and agency application in the market often via third parties. In so doing the enterprise combines internal symmetry with external asymmetry and does so with partners that are exposed to all the tools, all the data, all the inventory, all the time. In there lies the knowledge from which advantage accrues. 

@robnorman

@robnorman with  thanks to Robert Schneider, Harvey Goldhersz and Colin Barlow

This post originally appeared on Mediavillage.com

What Social Data Can Tell You (And Why)

Pretty much everything. In a nutshell.

Take this fascinating piece of research: Facebook's data scientists were able to determine a very particular pattern when couples were courting, and when they had started a relationship simply. How? Simply by looking at the frequency of status updates between two people.

In essence:

During the 100 days before the relationship starts, we observe a slow but steady increase in the number of timeline posts shared between the future couple. When the relationship starts ("day 0"), posts begin to decrease. We observe a peak of 1.67 posts per day 12 days before the relationship begins, and a lowest point of 1.53 posts per day 85 days into the relationship. Presumably, couples decide to spend more time together, courtship is off, and online interactions give way to more interactions in the physical world.

Social data are powerful

These predictions comes from a new class of data called "social data". Broadly speaking, social data are the data that people create when they use social platforms like Facebook, Pinterest or LinkedIn. It is our likes, pins, favourites, retweets, status messages, content of those messages, people we are friends with.

Social data are normally voluntarily and informally created by the individual during the act of using a social platform. It is voluntarily made public (or semi-public)on the platform in question. It reflects their ordinary course of business, the stuff we care about. It provides a picture which is explicitly incomplete. It can be viewed in aggregate or at an individual (although not necessarily prima facie personally identified).

Social data as a class of data is new. No more than 10 or 15 years old, running back to the older social platforms (although arguably it stretches back to the earliest social protocols like Usenet).

As a class of data it is also very powerful. And can predict much more about individuals and groups that the non-specialist can imagine.

Birds of a feather, that's why

Take the pattern of courtship to relationship discovered by Facebook's data scientists. This pattern emerges because of similarities in the way people behave as they go through similar life stages. We can all be different, but in aggregate we might be able to evince some underlying pattern. It is these underlying patterns which allow us to predict things which seem unlikely - like whether a couple of entering a phase of courtship.

I write 'unlikely' but it's really only 'unlikely' if you don't work with social data, or other behaviour data, and don't hang around with data scientists and other statistical types.

In truth, that we can predict courtship from signals like this is hardly at all surprising. We can predict other things too. These include wealth, gender, sexual orientation, even whether ones parent's divorced before you were an adult. All of those predictions were based on Facebook Like data - commonly acknowledged as quite noisy. (You can read an earlier blog post I wrote: Three Insights from Social Data which covers some of these predictions.)

For a moment think about that.

It's a brave new world

For companies, social data allows them to know who they are dealing with, well before that person becomes a customer. If a potential customer follows you on Twitter, you can build a pretty full picture of who they are, what they might like and what you might want them as a customer. Well before you decide to despatch advertising or a sales person towards them. For example, at PeerIndex we've helped companies modify their product offerings in near real-time on the strength of behaviours we've picked up in social data. IBM is helping companies identify customer's personality types based on social data - for better customer service.

For governments, social data allows you to understand the over-arching trends and themes in your jurisdiction. It allows for the early detection of themes that you might need to respond to. (Disease outbreaks? Or sociological changes in household behaviour?)

With the increase in people sharing data with companies - a subject I touched on a few weeks ago - such predictions will become more precise, more accurate and more relevant.

I regular post links relating to social data on my Twitter feed. Do follow me there.

Further reading

Facebook's entire summary on love is available here. Makes for fascinating reading.

My blog post on insights from social data is here

We have more resources available at the PeerIndex website.

Image Credit: Pedro Simoes

2 Ways Big Data Can Make You Happier

Wait—what?

I’ve heard a lot of crazy things in Silicon Valley, but this one is new. Apparently, big data can make you happier.

That’s the new claim according to Stefan Weitz, Senior Director of Search at Microsoft, and author of the new book Search.

I’ve looked into Weitz’s points… and they make complete sense.

How?

Weitz explains that the digital world is currently being rebuilt inside our massive computing systems, which help run search engines. Every person, place, and thing—and all the relationships in between—are being cataloged.

Think about how much data that is. It’s nearly 4 Zetabytes this year alone – enough to fill 130 billion 32GB iPads. (Yes, you read that right: 130 billion.)

You see, this isn’t just big data—it’s comprehensive data. It provides a more complete picture of our physical world than ever before possible, and with that comes certain capabilities that computing systems can harness.

Suddenly, our phones:

• are telling us when to leave the house to make our friend’s concert in time
• can arrange a driver to take us there
• and remind us to pick up a hat at the store on 7th on our way to the venue

…all without us doing anything.

What happens when our minds are freed from all the minutiae of our daily lives?

Weitz’s claims that this newfound efficiency ushered by big data will make us happier, for two reasons:

###

Reason #1

Because we will no longer have to burden our already crowded minds with things to do or remember, we will be able appreciate the present more fully.

Buddhist monks talk about how happiness comes from being in the “present moment.” Well, the more big data frees up our to-do lists (now that apps like Humin can remember who you know, and how you know them), the more peace-of-mind we can have.

Reason #2

The second reason borrows from Barry Schwartz’s book The Paradox of Choice, in which he contends that the more choices we have, the less happy we are. It’s counter-intuitive. One would assume that more choices is a better situation—but the reality is that having 24 choices of organic toothpaste actually causes us to suffer from decision paralysis. And then when we do choose one, we are likely to suffer decision-regret.

New advances in big data and search technology will ease our decision making process, because eventually, the data may know your preferences better than you do yourself.

###

As we offload our basic tasks to our digital assistants, freeing our crowded minds and letting us focus more on things we love—we will be led to a new era of insight, efficiency, and ultimately, happiness.

But that’s just the tip of the iceberg.

If you’re curious about where the world of search tech and big data are going—and if you want to stay ahead of the next technological wave—take a look at Weitz’s new book Search: How the Data Explosion Makes Us Smarter.

In it, he chronicles the progression of search from something we use to find sites on the web, to a fully digital, omniscient version of the human brain (yes… search engines are currently being programmed to make decisions like the human brain).

I just finished reading the book, and I feel smarter already.

==============================================================

Alex Banayan is the author of a highly anticipated business book being released by Crown Publishers (Random House, Inc.) The book chronicles his five-year quest to track down Bill Gates, Lady Gaga, Warren Buffett, Steven Spielberg, and a dozen more of the world's most successful people to uncover the secrets of how they launched their careers.

To get exclusive content from the book and the latest from Alex's adventures,click here to join his Inner Circle email community.

The Audit Renaissance Is Here. Data Scientists Welcome.

In this series, professionals debate the state – and future – of their industry. Read more here, then write your own #MyIndustry post.

Public company auditing has long been a steady mainstay of our financial markets. Since the turn of the century, however, auditors have pushed into a new era — and it's a good one.

"We're in an audit renaissance," Deloitte & Touche LLP partner Dan Sunderland told me recently. Dan is the firm's National Leader of Audit and Assurance Services.

What's behind this audit rebirth? Dan says it's a confluence of factors, but two stand out as driving forces: (1) auditor relevance and (2) technology.

Let's have a look at these forces, which have implications for both the markets generally and for those considering a career in auditing.

Auditor Relevance in the 21st Century

Across the globe, investors, audit committees, and other key market participants have expressed their need and desire for more information regarding the work and views of public company auditors. This isn't surprising, given evidence of the robust confidencethat investors place in independent auditors.

The auditing profession is responding actively to this need on a number of fronts, notably development of audit quality indicators and rethinking the auditor's report.

The latter is a great example of the push for greater understanding. By way of background, the auditor's report is where the auditor opines on whether the financial statements are presented fairly, in all material respects, in accordance with a financial reporting framework like U.S. GAAP. The report is seen as a simple "pass-fail" statement, but policymakers, the profession, and others have been exploring how to make it more informative.

The United Kingdom has been in the vanguard on this issue. UK audit reports now provide more information, including a discussion of the application of materiality, the scope of the audit, and an assessment of risks of material misstatement. According to a report issued by the UK's Financial Reporting Council in January, "investors have welcomed extended auditor reporting, and greatly value the enhanced information it provides." For his part, James Doty, Chairman of the U.S. Public Company Accounting Oversight Board (PCAOB), recently hailed the new UK audit reports as bringing "new relevance to the audit."

As in the United Kingdom, U.S. public company auditors have been deeply engaged on the issue of auditor's report, providing extensive and constructive input on regulatory proposals. The profession will remain engaged as PCAOB continues work this year on rules to refashion the auditor's report.

Enhanced Understanding through Technology

As I've noted before on LinkedIn, technology is also a force for change in auditing. Firms are pushing into exciting new technological areas, like greater use of data analytics and artificial intelligence techniques. In the words of EY:

"Big data and analytics are enabling auditors to better identify financial reporting, fraud and operational business risks and tailor their approach to deliver a more relevant audit."

It's important to emphasize that the use of technology isn't just about being on the cutting edge. As Deloitte's Dan Sunderland told me, technology is "a depth of understanding play." High tech, in other words, is strengthening and deepening the auditor's analytical role and work — not simply speeding it up or adding bells and whistles.

Career Implications

The audit renaissance, of course, has career implications as well.

Generally speaking, accounting and auditing has a very favorable career profile, with a healthy hiring outlook driven by a global increase in demand. The profession's technological awakening should be of particular interest to jobseekers, as audit firms of all sizes will continue to hunt for those with expertise to use technology and the enthusiasm to learn it.

But remember, the audit renaissance is bigger than just adopting whiz-bang tech. In essence, this rebirth is driven by the need and desire, both inside and outside of the profession, for enhanced understanding. As such, the core skills and attributes that auditors have always had — independence, skepticism, objectivity — will remain as important and valuable as ever.

I welcome your thoughts in the comments.

A securities lawyer, Cindy Fornelli has served as the Executive Director of theCenter for Audit Quality since its establishment in 2007.