We have spent a fair amount of time thinking about why some problems faced by data driven marketers are so much harder than others, advertising's P vs. NP problems. We were looking to find a reason why these hard problems have not been solved.
In short, we realized that there are two ways of thinking about solving data problems, the first is by modeling (think algorithms) and the second was a new idea. The new idea is that there are problems in marketing whose solutions are emergent from the data itself. In other words it can be measured but not modeled. Effectively it means that some problems can only be solved by working out every possibility and measuring which work and which don't. It means that the way we design our data models really matters.
Emergent solutions arise when new properties and behaviors "emerge," with no one directing and no one able to foresee the new characteristics from knowledge of the constituents alone.
In chemistry, for example, the taste of saltiness is a property of salt, but that does not mean that it is also a property of sodium and chlorine, the two elements which make up salt. Thus, saltiness is an emergent property of salt.
17th century economist Adam Smith described an "invisible hand", an emergent property of markets, that guides markets to produce just the amount and variety of goods that the public needs. The stock market has its own "invisible hand." The purely self-interested actions of thousands of buyers and sellers result in the purely blind workings of the stock market—the sudden shifts in activity and valuations, the bubbles and crashes—as well as the market's notorious properties of stupendous intricacy and frustrating unpredictability.
There are some problems in media whose solutions will only emerge from data. The key challenge is usually measuring and identifying the source of the issue. Once a problem can be identified and measured, it’s solution emerges from the data. Patterns can be identified and changes can be made.
Solutions for marketing problems such as attribution are data-emergent in nature. As evidence, look to the current models used for attribution. Frankly, the market has just begun measuring attribution and the models in the market kind of suck. We can’t even model how and why people click on ads, just that they did, let alone understanding why they made a purchase.
To do that we need to change the way we understand attribution data. Books are a good analog to attribution. What if we stored the data for a library in such a way that every book's data contained its translation to every language on earth. Fundamentally, War and Peace is the same story in every language, but understanding all translations at the same time will surely yield understanding that we couldn't get from just the English version. This exemplifies how storing the same basic data in different ways unlocks knowledge that was simply inaccessible before. The old way of storing that information made it invisible. Every language has its own unique way of capturing and communicating ideas. If a story can be told in every language, the underlying fidelity of the ideas in the story are far more well defined. The improved definition is the emergent property in this case.
Many of the problems we face in marketing are as difficult as they are to solve because the way we describe the underlying information needs to be improved. We are simply unable to see a solution because we are unable to identify the source of the problem.