WHAT IS OBSERVABILITY AND WHY DO CUSTOMERS NEED IT
If someone hears Enterprise Observability they may understand it, support it, and even grasp its complexity. But will they realize it is the Magnum Opus of IT
—- AVP Enterprise Data, Financial Services Customer
Observability is an emergent property of complex systems. Without this discipline, as systems mature in complexity so too will entropy in the environment.
The discipline of Observability is fundamentally about improving software delivery and operations. This is a socio-technical domain requiring cross-functional commitment from multiple teams.
When implemented correctly Observability represents an amplifying function for forward-thinking Product Teams and is necessary to meet the needs of modern architecture.
WHEN DO CUSTOMERS NEED OBSERVABILITY
THE VALUE CONTINUUM
Observability
Value
Monitoring
Value
Monolith
-
Single stateful data store
-
System metrics are relevant, and emphasis is on post-hoc analysis
-
Fairly static and unchanging set of resources with known thresholds of expectation
-
Focus is uptime and failure prevention; post-incident analysis is adequate to meet goals
-
Correlation occurs across a limited number of dimensions
Cloud-Native
-
Polyglot persistence
-
Telemetry insufficient for ad-hoc understanding
-
Infrastructure is dynamic and elastic with loosely coupled services across Product Teams
-
Focus on reliability and resiliency with constructs such as error budgets, SLOs, SLIs
-
Correlation occurs across an unlimited number of dimensions & cardinality
COMMON CHALLENGES DRIVING OBSERVABILITY
Technological revolutions require existing businesses to master a new means for production. For the digital revolution that we firmly sit in now, that new means of production is software.
~Carlota Perez,
-
Honorary Professor at University College London
-
Honorary Professor at University of Sussex
-
Adjunct Professor at TalTech, Estonia
-
Software architectural complexity, composition, operating costs, and time-to-market is ever increasing
-
Managing data ingest costs with existing observability tooling, while preserving value of data, requires constant evaluation
-
Tool sprawl and tool dependence are creating solutions that are often siloed and therefore brittle
-
Resilience engineering is an imperative for teams providing platform services
-
Product Teams often need to introduce their own tooling to meet evolving and novel needs not met by current strategy
-
E2E testing complexity and testing economics is normalizing Testing in Production (TiP) and Chaos Engineering as accepted standards
OBSERVABILITY EVOLVED
From Here
Traditional Observability
-
Applications instrumented with various language-dependent client libraries
-
Metric data is typically stored in a Time-Series Database (TSDB) and helps to answer ‘known unknowns’
-
Tracing data introduced the notion of request context propagation to move context between services and processes.
-
Log data has a long legacy with limited support for tracing and monitoring (log linking, custom attributes)
-
As current deployments often leverage disparate libraries, collection agents/daemons, protocols, data models, and backends - correlation becomes challenging and brittle
To Here
Modern Observability
-
Commoditize telemetry collection and propagate context objects (key:value) at time of collection. In this way, if both the tracing and metrics signals are enabled, recording a metric may automatically create a trace exemplar
-
Develop an observability pipeline to route full-fidelity data to low(er) cost mediums, while passing on propagated context to any backend platform
-
Solution-specific instrumentation not only promotes vendor lock-in, but the surface area of software diversity is too great for a commercial team to keep up with
OBSERVABILITY SOLUTION
A TAXONOMY
Domain: Observability Operating Model (OOM) Establishing an Observability Operating Model (OOM) is easily the most fundamental and critical element to success. As Observability is a journey, establishing clear operational motives and common goals will allow many teams to self-identify and co-create value.
Domain: Telemetry Collection & Context Propagation Establish architectural tenets that commoditize telemetry collection and propagate context to backend systems. This can be a long pole to attack, and you will need to think carefully about the developer experience as you focus on signal emission capture.
Domain: Observability Pipeline An Observability Pipeline establishes guardrails (compliant facility that aligns with IT control narratives) and advanced data controls (e.g., obfuscation, removal of superfluous data, enrichment), while promoting vendor neutrality for observability platforms.
Domain: Platform(s) & Data Interpretation Aggregation of disparate observability tools (COTS/custom) creates a common interpretation layer for all teams to operate from, and correlation improves overall central tendency metrics (e.g., MTTD, MTTR) while simplifying tools rationalization efforts.
Domain: Incident Analysis & Feedback Maturing an observability discipline promotes value extraction beyond baseline central tendency metrics (e.g., human factors, agile metrics, regression/performance management, resilience engineering, chaos engineering, telemetry verification automation, etc.)
MARKET TRENDS
DEFINING A STRATEGY
When done correctly, Observability represents an amplifying function that elevates many different domains in an organization (rising tide phenomenon).
Though strategy is highly dependent on characteristics that are unique to each individual organization, there are common starting points we typically advocate for.
Click on each strategic element below for more information:
FROM FIELD PERSPECTIVES
TO FIELD SUCCESSES
When considering Observability as a large-depth-of-field domain we ‘think globally but act locally’.
We consider the whole when devising strategy but act pragmatically to deliver value with every engagement on our customer’s overall journey!
Our research perspectives and strategic areas of focus (short-term & long-term lens), are calibrated from field experience implementing Observability journeys.
Click here for our Observability Pipeline paper
OBSERVABILITY
PODCAST EPISODES
CO-CREATING VALUE VIA
TARGETED PILOTS
Overlaying gap analysis from Platform Engineering Team & Product Team roadmaps will help craft viable pilots
Product Teams/Developer community may be evaluated against an Observability Maturity Model (OMM) to help prioritize value streams
HOW WE CAN HELP!
As with many domains that have broad applicability to an organization the option always exists to allow Product Teams to address individual activities on their own. We always caution that without a cultural shift to proactively think about observability in every business and architectural decision, organizations tend to lose efficiencies and accretive benefits.
Observability truly fits the aphorism, "a rising tide raises all ships". Many areas are direct or indirect benefactors of getting observability right, and we specialize in activating a company around a common set of goals.
Please contact us to learn more about, or exchange ideas on, this domain!
