The Path to Discovery

Published on November 9, 2012 by Marc-Andre Morisset
Copyright: Standard copyright agreement

I have been part of a number of engagements lately that draw on technical capabilities to fulfill the promise of what is now referred to as Information Discovery. I have noticed that once the initial energy of the project launch begins to dissipate and development gears are set in motion, the following conversation inevitably emerges:

Technical Guru :  "So, what do you want to see?"

Business Visionary:  "I'll know it when I see it."

Technical Guru:  "... So, what do you want to see?"

And there it is: the deadlock, with no obvious route forward. The impact of this deadlock includes squandered ideas, increased confusion, and even disillusionment caused by a failure to start.

How could brilliant minds struggle so much to exploit the potential of an Information Discovery product?

To begin to answer this question we would need to take a detour and explore how Information Discovery differs from its close cousin Business Intelligence. The line between traditional Business Intelligence (BI) and Information Discovery can sometimes be blurry. This holds particularly true if the line was drawn by someone in Sales or Marketing over a few martinis.

One way to differentiate between BI and Information Discovery is based on the types of information that allegedly surface: structured or unstructured, respectively. Another way is to contrast the user experiences they deliver: static or interactive. Yet another way is to compare their underlying data models: defined or schema-neutral. As interesting as these points might be they downplay another significant difference: BI is used when we know an answer exists and we want to list it, slice it, aggregate it, compare, or contrast it. On the other hand, Information Discovery is used when we don't know what questions to ask.

In the case of BI, finding the answer to a known question (i.e. last quarter's revenue before taxes) lends itself to a division of concerns and expertise: the business defines the question and the technologists surface the data to answer it. However, uncovering what the question should be in the first place raises crosscutting concerns that are not neatly assigned to either the business or technical domains.

A crosscutting series of questions might be: can we overlay payroll data on the operations data? Wait, do we have payroll data? How does it compare to the operations data? Are these gaps relevant or material? Is there any actionable insight? No one role in the organization can answer these questions alone. As a result, the Information Discovery "requirement" or "specification" cannot be neatly packaged and thrown over the proverbial fence.

 Finding the questions first requires being bold enough to recognize that we don't know what the question is...yet. Finding the questions also requires being willing to misstep and find answers to the wrong questions from time to time. Finding the questions can also imply venturing beyond the perimeter we might have become comfortable with. Finally, finding the questions requires a partnership.

The description of the Information Discovery experience above suggests a stronger business involvement, tighter iterations to mitigate the impact of making a wrong turn, and redesigning the solutions as the need emerges. Some may recognize the traits of agile software development in this description of an Information Discovery experience. This is no mere coincidence: much like you can't deliver with agility without revisiting the level of collaboration between business and technology, you cannot simply deliver an Information Discovery experience using traditionally rigid and brittle tool chains. The level of uncertainty inherent in discovery engagements would not be efficiently managed using traditional BI workflows and infrastructures. In fact, the cost of changing direction midstream would cripple the data modeling effort, let alone the data transformation work.

In contrast, a strong Information Discovery platform lends itself to changing data structures and data type to gracefully adapt to course corrections. Additionally, an Information Discovery solution is deployable in small iterations without a massive solution built-out to mitigate the impact of tangents. Finally, Information Discovery technology lends itself to rebuilding to incorporate and extend the reach of information. With these points in mind, the real value of an Information Discovery platform begins to take shape, and in doing so, makes the difference between traditional BI and Information Discovery that much crisper.

Certainly, Information Discovery is a product name. Many will attest to that. Fewer will realize that Information Discovery is also engagement model that bootstraps the process of getting a glimpse of what you are looking for within your organization's information. When looking for a way forward in cases where "you don't know what you don't know," you will need both an engagement model and tooling that lend themselves to discovery.

Let's face it: if you knew what you were going to uncover before you started it wouldn't be "discovery" would it?

Marc-Andre Morisset | Architect | RealDecoy
phone: 613.234.9330 x1086


You need to be logged in to be able post a comment.