We kept blaming the data. Turns out, the data was innocent. 🔍
For weeks, our SLTs were being missed. Query timeouts. Stale dashboards. Frustrated users. Every indicator pointed at the data layer — so that's where we dug.
What we actually found:
🔴 The root cause wasn't bad data — it was data CONTENTION. Hundreds of concurrent queries fighting over the same shared Fabric capacity, queuing, blocking, and cascading into failure.
🌍 And then there was the geography problem nobody had mapped — our Fabric capacity tenant sitting across the Atlantic, adding 80–150ms of round-trip latency to EVERY query. Not slow computation. Slow continents.
The result? Gateway timeouts that looked like data failures. Connection abortions mid-flight. SLT compliance sitting at 54%.
Once we tackled concurrency limits, workload management, and migrated our tenant region closer to our users?
✅ Response times dropped 68%
✅ SLT compliance hit 91%
✅ Gateway timeouts: near zero
The data was fine the whole time. We were just asking too much of it, from too far away.
Full write-up in the article below 👇 — especially relevant if you're running analytics on Microsoft Fabric or dealing with cross-region BI deployments.
#DataEngineering #MicrosoftFabric #PowerBI #DataPlatform #CloudArchitecture #Analytics #PerformanceEngineering #ServiceLevels
A deep-dive into how high-concurrency query contention and cross-Atlantic infrastructure latency quietly erode service levels — and what to do about it.
The Symptom Everyone Sees, The Cause Nobody Suspects
In most performance post-mortems I've been part of, the conversation starts the same way: "The data is slow."
Dashboards are lagging. Reports are timing out. Business users are frustrated. SLTs (Service Level Targets) are being breached, and fingers are pointing squarely at the data layer.
And to be fair — the data is where the pain surfaces. But here's the uncomfortable truth that took us time to fully unpack:
> The data wasn't broken. The data was overwhelmed — and it had a geography problem.
The Investigation: Peeling Back the Layers
Layer 1 — SLT Misses and the Data Blame Game
Our Service Level Targets were consistently missed during peak business hours. Query response times that should have landed under 3 seconds were stretching to 30, 60, sometimes over 120 seconds. Dashboards were either stale or outright failing to load.
The immediate assumption? Data quality issues. Bad indexes. Poorly written queries. An overloaded dataset.
We tuned queries. We rebuilt indexes. We optimised data models.
The problem persisted.
Layer 2 — The Real Root Cause: Data Contention Under Concurrent Load
What our monitoring eventually revealed was not a quality problem but a concurrency problem.
During peak hours, hundreds of users were simultaneously firing queries against the same datasets. The underlying engine — running on a shared Microsoft Fabric capacity — was not failing because the data was wrong. It was failing because too many queries were competing for the same computational resources at the same time.
This is data contention: a condition where high volumes of concurrent queries queue up, block each other, and create a cascading slowdown that ripples all the way to the end user experience.
Key symptoms we identified:
- Query queue depth spiking dramatically during 9–11 AM and 2–4 PM windows
- Throttling events logged at the Fabric capacity level
- Spill-to-disk operations increasing as memory pressure mounted
- Individual query execution time remaining acceptable in isolation, but degrading 10–40x under concurrent load
The data was never the culprit. Contention was.
Layer 3 — The Atlantic Problem: Latency Nobody Mapped
Here is where the investigation took an unexpected turn.
Our Microsoft Fabric capacity tenant was provisioned in a region across the Atlantic — physically and network-topologically distant from the majority of our user base. What looked like slow query responses was, in many cases, not slow computation at all. It was network round-trip time compounding every single interaction.
The effects were insidious:
- Gateway timeouts — connections from local gateways to the remote Fabric tenant were breaching timeout thresholds before query results could be returned, not because queries were slow, but because the network handshake and data transfer time pushed the total wall-clock time over the edge.
- Connection abortions — under load, TCP connections to the distant tenant were being dropped mid-flight. The client saw a failure. The capacity was actually working — the work was simply never delivered.
- Compounding effect — contention-induced delay + cross-Atlantic latency + gateway timeout threshold = a perfect storm that made every SLT miss look far worse than the underlying compute performance warranted.
A query that took 8 seconds to execute looked like a 45-second failure from the user's perspective. The network ate the difference.
The Architecture of the Problem
[User / BI Client]
|
| (Local network)
|
[On-Premises / Regional Gateway]
|
| ← Cross-Atlantic hop (~80–150ms RTT)
|
[Microsoft Fabric Capacity Tenant - Remote Region]
|
[Shared Capacity Pool] ← Contention point: concurrent queries competing here
|
[Dataset / Lakehouse / Warehouse]
Every query traversed this path — twice (request and response). Under contention, queries that lingered in the execution queue long enough would cause the gateway to give up before the answer arrived.
What We Did About It
1\. Addressed Contention at the Capacity Layer
- Scaled up Fabric capacity during peak windows using autoscale policies
- Implemented query concurrency limits and workload management rules to prioritise critical service-level reports
- Introduced incremental refresh and aggregation tables to reduce raw query payload sizes
- Shifted non-urgent batch workloads to off-peak hours to reduce simultaneous demand
2\. Tackled the Geography Problem
- Engaged with our Fabric tenant configuration to migrate capacity to a region closer to our primary user base — a process that required planning but delivered the most dramatic latency improvement
- Increased gateway timeout thresholds as an interim measure to prevent premature connection drops while longer queries completed legitimately
- Implemented connection pooling and keep-alive configurations on the gateway layer to reduce connection setup overhead on every request
3\. Improved Observability
- Instrumented query telemetry to distinguish network latency from compute latency — critical for correctly diagnosing future incidents
- Built a capacity utilisation dashboard to give early warning of contention events before they breached SLTs
- Established a baseline for expected cross-region RTT so anomalies could be detected faster
The Results
Once the contention management and regional migration work completed:
- Average query response time dropped by ~68% during peak windows
- SLT compliance improved from ~54% to over 91% within 6 weeks
- Gateway timeouts effectively dropped to near-zero
- User satisfaction scores for the analytics platform improved significantly
The data, as it turned out, had been perfectly fine the whole time.
Key Takeaways for Data & Platform Engineers
1. Don't confuse the symptom with the cause. Slow data experiences are often infrastructure and concurrency problems wearing a data costume. 2. Concurrent query volume is a first-class concern. Design your capacity with peak concurrency in mind, not just peak data volume. 3. Geography matters more than people think. Cross-region tenancy is often an afterthought in platform provisioning. It shouldn't be. Measure your RTT early and often. 4. Gateway timeouts are a network story, not a query story. If timeouts correlate with specific time windows and user geographies, suspect latency before suspecting code. 5. Observability must separate layers. If you can't distinguish network time from compute time in your telemetry, you will misdiagnose performance problems — repeatedly.
Final Thought
The most dangerous performance problems are the ones that look obvious but aren't. SLT misses that appear to be data problems can spend months in the wrong queue — being "fixed" by data engineers while the real causes (concurrency limits, tenant geography, gateway configuration) sit untouched.
Invest the time to instrument your full stack. The answer is rarely where the pain is loudest.
Have you encountered similar patterns in your data platforms? I'd love to hear how your teams approached contention and latency challenges — drop a comment below.
#DataEngineering #MicrosoftFabric #PowerBI #ServiceLevels #DataPlatform #CloudArchitecture #Latency #PerformanceEngineering #Analytics #DataContention #TechLeadership