app-performance-monitoring

Overview

APM answers "why is this slow?" with data: which transactions are slow, which spans dominate, which database queries drag, and how performance trends across releases. This skill covers instrumenting applications for performance visibility and using that data to find and fix real bottlenecks — independent of any specific APM vendor.

When to use

• Finding why an endpoint or page load is slow
• Setting up transaction tracing and span instrumentation
• Identifying N+1 queries, slow serialization, or chatty service calls
• Tracking performance across releases (did v2.1 get slower?)
• Deciding what to instrument with custom spans vs automatic

Core concepts

Transactions and spans. A transaction = one unit of work (an HTTP request, a background job, a page load). Spans = its decomposed steps (DB query, cache call, external API, template render). Automatic instrumentation covers frameworks and drivers; custom spans mark your business logic ("compute recommendations"). The trace waterfall shows exactly where time goes — optimize the longest bars first.

Measure percentiles, not averages. Averages hide the pain: p50 tells you the typical experience, p95/p99 the bad one users actually complain about. Track latency as a distribution; alert on p99, optimize the spans that dominate p99 traces.

Apdex / satisfaction framing. Translate latency into user experience: what fraction of requests are "satisfying" (< T), "tolerable", "frustrating"? It focuses optimization on what users feel and gives non-engineers an intuitive number.

Sampling strategy. Tracing every request at high traffic is expensive (overhead + ingest cost). Sample: 100% of errors and slow outliers, a percentage of normal traffic (enough for representative p99s), and dynamic sampling that keeps interesting traces. Head-based sampling is simple; tail-based keeps the important ones — know which you have.

Correlate with releases. Every deploy should mark the timeline; a p99 jump starting exactly at release X is a diagnosis, not a mystery. Tag transactions with release version and compare distributions across releases, not just point values.

Practical workflow

1. Instrument: install the APM SDK/tracer, enable auto-instrumentation for your framework/DB/cache/HTTP clients, and add custom spans around key business operations.
2. Establish baselines: per-endpoint p50/p95/p99, throughput, and error rate over a representative week — you can't detect regressions without "normal".
3. Find the bottleneck: sort transactions by total time impact (latency × volume — a slightly slow hot endpoint beats a very slow rare one), open representative slow traces, and read the waterfall.
4. Fix the usual suspects: N+1 queries (batch/eager-load), missing indexes, over-fetching (select only needed fields), synchronous calls that could parallelize, uncached repeated computation, slow serialization of huge payloads.
5. Verify with data: compare p99 before/after the fix on comparable traffic; watch for regressions in the release comparison view.
6. Guard with alerts/SLOs: latency SLOs per critical endpoint with burn-rate alerting; performance budgets in CI for key pages/endpoints to catch regressions before deploy.

Common pitfalls

• Optimizing the average — p50 looks fine while p99 users suffer; always work from the tail.
• No custom spans — auto-instrumentation shows "DB: 800ms" but not which business operation caused 40 queries; span your domain logic.
• Tracing 100% of traffic — overhead and cost explode; sample deliberately.
• One slow trace = the truth — outliers mislead; aggregate across many traces before concluding.
• Ignoring the frontend — backend p99 is great but users wait on 4MB of JavaScript; measure real user experience (RUM) alongside.
• Performance work without baselines — "we made it faster" with no before/after numbers; measure first, then optimize, then verify.