Data Maturity Valley

Identifying and Mitigating Inexorable Inefficiencies in the face of Multi-Platform Integration

Introduction

As data continually cements itself as ‘language of doing business’, the impetus is increasingly on firms to maintain their literacy in this language as much as possible. In this sense, ‘data maturity’ represents a shifting frontier, a benchmark that all firms are reaching for. It is a barometer for how entrenched proper analytics, management, and application are in a firm’s practice. Maturity is relative, and the degree in which a firm can be competitive with their data strategy depends on their respective industry, competition, and human capital.

An argument can be made that this creates inertia that forces firms to not only adapt but be innovative or die. Maturation is only as limited as the most recent product release; the next software rollout or the “new way of doing things” that lays to waste all the inefficiencies of the status quo. This impetus charges a perpetual race to the top, where firms will continually look to integrate and utilize the most current and best tools to keep up with competitors and maintain relevance. The platform or system that moves the needle for a firm represents a ladder rung to reach above for the sake of ‘better business’ in search of that illusive maturation.

The constant chase of technological innovation can look wildly different across different industries and sectors. However, there is an inexorable factor that emerges when these new platforms are rolled out that antagonizes even the most meticulous planning.

Concept

In the wake of these pursuits, inevitabilities begin to arise. As firms update data storage, management, or analytic applications, external effects begin to take hold. Implementation takes time, bandwidth, resources, and detailed planning. Because of their interdependent nature, it is often opportune for firms to implement multiple data solution platforms over a singular timeframe. This turns a single integration into a total technological transformation, keeping with the pursuit of a larger sense of data maturity.

With any such implementations, schedule delays, technological inconsistencies and scope creep begin to arise. Multi-platform implementations often expose systemic problems even in the face of ‘data maturity’. Often, a firm is in the process of integrating multiple platforms, but one is functional while another (or some element of it) is not yet usable. This scenario, where one platform is operational as some other lags, creates a period of increased inefficiency referred to as the Data Maturity Valley.

This phenomenon describes the operational hold-ups and productivity losses that occur when firms experience an incongruence in integration timelines. This creates a (sometimes substantial) time gap, where teams must continue working inefficiencies as operational bottlenecks begin to arise in the face of limited bandwidth.

In a data science or analytics practice, the Data Maturity Valley is an all-too-familiar pitfall that firms must navigate, especially during large-scale digital transformations, software updates, or migrations from legacy systems to new platforms. If the end goal is a total digital transformation by the firm, only providing half of the foundation ultimately affects the outputs and quality of work that is dependent on these platforms; this can sometimes be more detrimental than even the previous iteration.

For the sake of abstraction, this can be conceptualized further in terms of epochs- if a technological shift occurs in multiple phases, the gap between epoch (n) and epoch (n+1) creates risk in the form of inefficiencies, make-work solutions, shifted timelines, etc. Work continues but is hindered by the incomplete implementation of necessary tools, leading to a Valley of diminished productivity and increased frustration. The greater that gap is or becomes, the greater risk. While sometimes nebulous in definition, each Data Maturity Valley has its own causes and effects and organizational solutions for mitigating its impact.

Scope

Each data management platform has its respective nuances, strengths, and weaknesses that play as factors for each Data Maturity Valley. What’s important is not the prescription of success upon emerging from a Valley; moreover, this is the exploration, identification, and mitigation of a concept that arises time and time again when these elements are put into play by a particular firm.

To note: this problem can exist in many forms across many industries all the same, but the major focal point of this concept is, of course, data. The scope of the Data Maturity Valley is limited to analytics and data science, however this can be applied (in abstract) to other systems, such as web development or engineering management platforms.

Means of Identification

While it may look different across different industries and sectors, the Valley’s face is largely the same when it emerges – in the form of “hurry up and wait”, in the form of rapidly shifting priorities and requirements, presenting as bloat, as time delays, as wasted effort and make-work and squandered resources. It is these inefficiencies that are tolerated due to the ever-pending ‘next stage’, and once platform X accessible or software Y is installed, it’s often posited that the problems will dissipate, and all will be solved.

But it is this hook, and promise of pending maturity, that makes the Data Maturity Valley treacherous for organizations. While it’s easily exemplified with the dual implementation of data storage and data visualization platforms, the Valley of inefficiency can arise in many scenarios that share the same dynamic, including:

• updates from an existent software/platform to a new one
• replacement from an old software/platform to a new one
• migration from a legacy platform to a new one without proper steps put in place

From the perspective of technical debt, this can also occur when:

• replacement or updated platform does not cover the same ground or address the same requirements as the previous version
• replacement or updated platform creates technical debt and requires some learning curve/training to achieve the same means as the legacy platform

From the epochal viewpoint again, it ultimately occurs when there is a seam between the first stage of some transformation and any other phase (n+1), which is not a phenomenon that is unique to solely dual-platform implementation.

Causes

The Data Maturity Valley is carved out when a certain set of factors become present; much like a tornado, it requires a number of pre-existing conditions to align perfectly (as they often do) for such an impactful deficit to arise. One thing that is important to underscore is the seeming inescapability of it.

These sorts of gullies should be viewed as systemic inevitabilities, instead of a burden that falls on the planning or project or execution teams. While they are the front lines to mitigate the issues from these Valleys, the buck does not stop there. It is the factors of reality that create these issues instead, the systoles and diastoles that happen when the ‘human element’, in all its glory, encounters even the most meticulous crafted plans and strategies.

The roots of a Data Maturity Valley are indicative of those same human elements, revealing issues like: IT bandwidth restrictions, installation delays, technology outages, time bloat on things like training and documentation, personnel absences, lack of integrated systems planning, download latency, security requirements and concerns, filesharing delays, etc.

Outside of the immediate sphere of technology, you have holidays, catastrophic weather, electrical issues, infrastructure failure, war, plague, famine, pestilence… any factor that comes in between a supposed plan and it’s supposed reality can be a core cause, to the extent that the avoidance of all possible causes becomes a snipe hunt.

To prescribe “how to prevent Data Maturity Valleys” should be viewed in the same vein as “how to avoid ever leaving your house” or “how to prevent all human contact”– its inevitability comes from the larger pursuit towards something better, a stridence towards improvement that should be fostered and rewarded. It’s a growing pain – firms looking to maintain a competitive edge are nudged towards enterprise platform integration, like a bird is pushed from the nest. To stay, to be locked in a status quo, is to languish—and the leap forward, presenting its own risks, is the only rational path towards survival.

The point is that the costs of doing business present the conditions to create these Valleys – the severity and impact of the Valleys can of course be mitigated, and some of these causes are existent in a way where you’d rather have them versus not have them (i.e. security protocols, vacation time, etc). The core effect here being the presence of multiple epochs, and even if the thought is that they can overlap (if not occur simultaneously), it is the fact that there is a ‘multi’ element that fosters the perfect conditions for the Valley to arise, like fungus inoculating on a tree stump.

Effects

The medium of exchange for the Data Maturity Valley is an erosion in efficiency. While these can have smaller, lagging effects on business components, they can also affect entire teams, departments, and firms. Systemic inefficiencies warp the face of the firm at the cost of resource consumption and begin to drag down the company culture at large.

While it may not be something that can be pinned on solely on leadership, it is proven that commitments to lost causes and ‘one-size-fits-all’ strategies hinder a firm’s efficacy, often at the cost of wasted labor or increased attrition.

Ultimately, there is natural interruption to business as usual. Once a Valley metabolizes, it is incumbent upon project teams and managers to mitigate the various effects that can ripple across the firm. The effects can be felt with a particular intensity in the face of a large-scale platform transformation, or by data science and engineering teams that seem to have one foot in the world of IT and one foot in the world of more qualitative management, with not enough leverage to affect the decisions of either. Some of these notable effects can appear as:

Communications Lapses

• When different teams are using different platforms (or versions), communication between these teams often breaks down. The lacking alignment creates silos, leading to miscommunication and inefficiencies. This communication lapse around certain workstreams not only increases waste, but it can affect the viability of the project as a whole. Which can lead to—

Paused, Delayed, or Held-up Work

• As platforms are not fully integrated, timelines become delayed, and teams may have to pause work or develop temporary solutions to continue their operations. These temporary “make-work” fixes are often inefficient and consume resources that could be better used elsewhere. Which can lead to—

Risk of Temporary Solution Permanence

• One of the more impactful effects is the risk that ad-hoc and patchwork solutions may become baked into business-as-usual practices. What was meant to be a short-term workaround can become an ingrained part of the operational process. This can then lead to long-term inefficiencies and time waste as employees are forced to pivot more than necessary, even when a Valley is traversed and a final roll-out is usable. Which can lead to—

Resource Waste and Stress

• Teams dealing with the Valley often experience increased workloads and stress as they attempt to balance existing work while managing inefficiencies introduced by platform delays. Training, communications, and support efforts for temporary solutions can exacerbate the drain on time and resources.
• This waste can often be discrete too; additional searching, outreach, and assistance can increase time delays as workers have to operate around the established inefficiencies for make-work solutions that are not worth the waste. Additionally, this can all be happening while the latent platform in question is sitting on the shelf, laying in wait while business hobbles along without it. Which can lead to—

Decreased Value

• The potential value from the new platform remains unrealized during the Valley. For example, a transformation project aiming to boost productivity may experience diminished returns as teams are unable to fully utilize the new system. This gap delays the positive impact that the technology was supposed to bring, and firms may continue to face inefficiencies in the meantime.

These can also be system- or industry-specific effects that may be harder to root out and ameliorate. For example, a Data Maturity Valley emerging during an implementation of automated scheduling software in automobile production can create pauses in the assembly line, and the loss of productivity can have ripple effects throughout the company even after the initial Valley is addressed. It is imperative to not only take stock in the high-level symptoms that a Data Maturity Valley can cause. For proper risk analysis, one must apply this rubric to a specific use case or industry to see what externalities can arise that may be unique to a given firm.

Mitigation Tactics

The tactics for mitigating the scope and severity of a Maturity Valley are essential to ensure business as usual is up and running as soon as possible, and that the inefficiencies of the Valley do not reverberate beyond what’s necessary.

Firms need to be adaptable, to be able to expand, contract, and get physical. If we’re using the words of Patrick Lencioni, both employee and firms need to ‘get naked’ – they need to recognize their inherent shortcomings and be up front about the inevitable hurdles ahead. This, unfortunately, requires humility — to carve a company identity out of some image of infallible perfection is to be destined for failure. It is building a foundation upon what’s merely a visage.

The awareness and expectation of these Valleys should be incorporated into any firm’s enterprise risk management plan. While a ‘batten down the hatches’ type of approach may oversimplify the roles required for mitigation, ultimately, it’s an all-hands and all-team solution. Any vertical that views themselves as an island in this scenario, and does not communicate, collaborate, or contribute to this exercise of adaptability, is setting themselves to get caught in the gears of the digital transformation. With that, there are several best practices for reducing the severity of operational inefficiencies:

Appoint A Specific Project Team

Communication across teams is essential during epochs of implementation – Teams working on different platforms or at different stages must be aware of each other’s progress, timelines, and challenges. This can reduce the likelihood of misaligned expectations and ensure smoother transitions. This allows people to rally around a point-person or a specific group that can field any questions or concerns without creating additional time-waste in the search for answers

An ideal group would have representation from all major stakeholder groups. This would streamline outbound communications and make sure all perspectives are heard from affected parties. While this team would have some loss in their own productivity (if their role is not for this explicitly), temporarily concentrating this to a small group creates a value-add to the rest of the teams and is overall beneficial to a firms’ efficiency.

Treat the Implementation Phase Like a Product

This product-first viewpoint would persist across multiple epochs and can be handled similarly to product development cycles that use Agile or scrum. Standup meetings, artifacts, short-term sprints, and clearly defined roles all allow for issues and inconsistencies to be addressed in one fell swoop, instead of falling into the delay of communications.

Build a Master Timeline

Have schedulers, managers, and all relevant team members aware to ‘report’ or otherwise document new work that comes up during the entire phase across all epochs. It is essential to create all-encompassing project timelines that include input from all relevant teams—procurement, implementation, operations, and support. Pinpoint specific areas that would be affected most if a serious maturity Valley were to open up

Mark definitive start dates and end dates for all implementation epochs, and ideally pad out the end date by some margin. This additional time can be removed if the implementation is complete within the padded schedule [weak maturity Valley] or be used to mitigate inefficiencies and change or otherwise update the implementation timelines [strong maturity Valley].

Pad timelines, as part of general communications make all teams that issue work aware of the implementation, specific risks, and the general timeline. Project timelines should include buffers to account for potential delays, and teams should plan for a period of temporary inefficiencies during integration. For example, firms could add extra time to the project timeline to allow for the full integration of platforms before beginning critical work. Projects can essentially lead the receiver by building out a larger initial timeframe in the face of a multi-platform rollout and can incorporate both platforms without worrying about any follow-on risk from a Valley emerging

Hold Forums as a Means of Disseminating Information

It is critical to go over the implementation status, schedule trainings, and maintain awareness. A seamless rollout is contingent on contractors, consultants, and end-users being informed of the roll-out timeline will help manage expectations and reduce disruptions. By building larger initial timelines and keeping all stakeholders updated, projects can better accommodate any delays caused by platform integration issues.

Manage Expectations as a Means of Prioritizing the Long-Term Vision

While temporary inefficiencies are inevitable, firms need to remain focused on the long-term benefits of technological transformation rather than short-term disruptions. Again – the pursuit of this change (and its inevitable pitfalls) represents the entropy a firm needs to remain viable in a technology-driven business environment. Being aware of the means of identification for such pitfalls– as well as their causes, effects, and mitigation strategies, allows for the affected firm to unify under the banner of progress. When expectations are properly communicated and managed amongst stakeholders, it allows for that little bit of breathing room in the face of temporarily increased inefficiency. With this, the effects can be managed, timelines can be shifted, waste can be mitigated, and the potential damage from a Data Maturity Valley will fail to be totally realized.

Final Remarks

It is worth emphasizing that keeping with the above tactics does not snuff out any appearance of these Valleys; however, they are effective at assuaging it’s effects and preventing harmful reverberations from echoing through an organization. This is especially critical during such a transformational phase. It is a serious undertaking to migrate from an existent, established platform to a brand-new unknown. The inertia of technological change bucks against our innate status quo bias, and it is this conflict that pushes firms to uncharted territory. Effective navigation requires taking the above considerations into mind and having total cognizance of the parties, pipelines, and workflows that lie in the path ahead. The more apt a firm is at identifying the causes and symptoms of a Data Maturity Valley, the better equipped they are to weather its effects. Once this path is successfully traversed, a firm will find that it is established in its maturity and literacy in the ever-present language of business today.