Servers Get Stressed Too

Subtle Ways We Overload Our Systems Without Realizing It

Abdulmateen Tairu (@taycode)
F1 and Football Fanatic

PyCon Africa 2025

Why This Talk?

• I've spent years building and scaling backend systems.
• I noticed something painful: we stress our servers more than our users do.
• Sometimes it's not traffic that kills performance — it's code.
• This talk is about those subtle anti-patterns that creep into our systems and quietly cause pain.

What Happens When Servers Are Stressed?

• High p95/p99 latency
• Increased timeouts and failures
• Spikes in CPU/memory usage
• Infrastructure cost creep
• Cascading failures across services

"It's not just scale that hurts systems — it's poor design under scale."

7 Subtle Anti-Patterns That Stress Your Servers

We'll go through each with examples and fixes:

1. Long request cycles
2. OFFSET pagination
3. N+1 queries
4. Hot-path aggregations
5. Fat payloads
6. Multiple row locking
7. Using database for search

1. Long Request Cycles

Problem: Doing heavy work like image/PDF generation, DB writes, or S3 uploads inside HTTP requests.

1. Long Request Cycles - The Fix

Before:

@app.post("/generate-report")
def generate_report(user_id: int):
    data = fetch_data(user_id)  # 2 seconds
    pdf = create_pdf(data)      # 5 seconds
    url = upload_to_s3(pdf)     # 3 seconds
    return {"url": url}         # Total: 10 seconds!

After: Use Celery, RQ, or any task queue

@app.post("/generate-report")
def generate_report(user_id: int):
    job = queue.enqueue(create_report_task, user_id)
    return {"job_id": job.id, "status": "processing"}, 202

# User polls /jobs/{job_id} or gets webhook when done

⚡ Response time: 10s → 50ms

2. OFFSET Pagination

Problem: Using OFFSET for deep pagination

Django:

# View handling page 200 (50 items per page)
def order_list(request):
    page = int(request.GET.get('page', 1))  # page = 200
    per_page = 50
    offset = (page - 1) * per_page  # offset = 9950
    
    orders = Order.objects.order_by('-created_at')[offset:offset+per_page]
    # Django generates: LIMIT 50 OFFSET 9950
    # DB must scan 10,000 rows! 😱

SQL Generated:

SELECT * FROM orders ORDER BY created_at DESC 
LIMIT 50 OFFSET 9950;

2. OFFSET Pagination - The Problem

Typical e-commerce pagination UI:

2. OFFSET Pagination - The Fix (Cursor-Based)

Use keyset/cursor pagination - filter by last seen value instead of OFFSET

Before (Django): OFFSET-based

# Slow for deep pages
orders = Order.objects.order_by('-created_at')[offset:offset+50]
# LIMIT 50 OFFSET 9950 - scans 10,000 rows!

After (Django): Cursor-based

# Fast for any page depth
last_created_at = request.GET.get('cursor')  # From previous page
if last_created_at:
    orders = Order.objects.filter(
        created_at__lt=last_created_at
    ).order_by('-created_at')[:50]
else:
    orders = Order.objects.order_by('-created_at')[:50]

# Return cursor for next page
return JsonResponse({
    'orders': [...],
    'next_cursor': orders.last().created_at.isoformat()
})

⚡ Uses index | 🚀 Constant performance at any depth

2. Cursor Pagination - How It Works

SQL Generated:

-- Instead of OFFSET
SELECT * FROM orders 
WHERE created_at < '2024-01-15 10:30:00'  -- Last seen timestamp
ORDER BY created_at DESC 
LIMIT 50;  -- Only scans 50 rows!

📊 Performance: 2.5s → 15ms (consistent!)

3. N+1 Queries

Problem: Fetching related data in loops:

posts = Post.objects.all()  # 1 query
for post in posts:
    print(post.author.name)  # 1 query each = 100 queries!

100 posts = 101 DB roundtrips! 🔥

3. N+1 Queries - The Fix

Before: N+1 queries (Django example)

posts = Post.objects.all()
for post in posts:
    print(post.author.name)  # Hits DB every time
# Result: 1 + 100 = 101 queries

After: Eager loading with select_related

posts = Post.objects.select_related('author').all()
for post in posts:
    print(post.author.name)  # Already loaded!
# Result: 1 query with JOIN

⚡ 101 queries → 1 query

Use select_related() for ForeignKey & OneToOne (uses SQL JOIN)

3. N+1 Queries - prefetch_related

For ManyToMany and reverse ForeignKey relationships

Problem: N+1 with tags (ManyToMany)

posts = Post.objects.all()
for post in posts:
    for tag in post.tags.all():  # Hits DB every time!
        print(tag.name)
# Result: 1 + 100 = 101 queries

Fix: Use prefetch_related

posts = Post.objects.prefetch_related('tags').all()
for post in posts:
    for tag in post.tags.all():  # Already loaded!
        print(tag.name)
# Result: 2 queries (1 for posts, 1 for all tags)

⚡ 101 queries → 2 queries

Use prefetch_related() for ManyToMany & reverse FK (separate queries)

3. select_related vs prefetch_related

Post.objects.select_related(
    'author', 'category'
)

Post.objects.prefetch_related(
    'tags', 'comments'
)

💡 Pro tip: Combine them! Post.objects.select_related('author').prefetch_related('tags')

4. Hot-Path Aggregations

Problem: Running expensive aggregations on every request

# Django view - runs on EVERY dashboard load
def dashboard(request):
    total_users = User.objects.count()  # COUNT(*) on 10M rows
    total_revenue = Order.objects.aggregate(
        total=Sum('amount')
    )['total']  # SUM() on millions of rows
    avg_order = Order.objects.aggregate(
        avg=Avg('amount')
    )['avg']  # AVG() calculation
    
    return render(request, 'dashboard.html', {
        'total_users': total_users,      # 800ms
        'total_revenue': total_revenue,  # 1.2s
        'avg_order': avg_order,          # 950ms
    })  # Total: 3 seconds PER page load! 😱

💀 100 users viewing dashboard = 100 full table scans!

4. Hot-Path Aggregations - Fix #1: Caching

Store computed results in Redis/Memcached, refresh periodically in background

Before: Count on every request

# Django view
def dashboard(request):
    total_users = User.objects.count()  # 800ms on 10M rows
    return JsonResponse({"total_users": total_users})

After: Cache + background refresh (Django + Celery)

# View - reads from cache
def dashboard(request):
    total_users = cache.get("total_users", 0)  # 2ms!
    return JsonResponse({"total_users": total_users})

# Celery task - runs every 5 minutes
@shared_task
def update_user_stats():
    count = User.objects.count()
    cache.set("total_users", count, timeout=300)  # 5 min TTL

⚡ 800ms → 2ms

✅ Simple | ⚡ Fastest | ⚠️ Lost on restart

4. Hot-Path Aggregations - Fix #2: Stats Table

Persist stats in DB table, update via signals or background jobs

Django Model:

# models.py
class SystemStats(models.Model):
    metric_name = models.CharField(max_length=100, primary_key=True)
    metric_value = models.BigIntegerField(default=0)
    updated_at = models.DateTimeField(auto_now=True)
    
    class Meta:
        verbose_name_plural = "System Stats"
        
    def __str__(self):
        return f"{self.metric_name}: {self.metric_value}"

View - Fast read from stats table:

def dashboard(request):
    stats = SystemStats.objects.get(metric_name='total_users')
    return JsonResponse({"total_users": stats.metric_value})
    # Fast indexed lookup! No COUNT() needed

✅ Persistent | ⚡ Sub-millisecond reads | 💾 Survives restarts

4. Stats Table - Update Options

Two ways to keep stats up-to-date:

from django.db.models.signals import post_save
from django.dispatch import receiver

@receiver(post_save, sender=User)
def update_user_count(sender, **kwargs):
    stats, _ = SystemStats.objects\
        .get_or_create(
            metric_name='total_users'
        )
    stats.metric_value = User.objects.count()
    stats.save()

from celery import shared_task

@shared_task
def refresh_stats():
    SystemStats.objects.update_or_create(
        metric_name='total_users',
        defaults={
            'metric_value': 
                User.objects.count()
        }
    )
    
# Schedule in celery beat
# '*/10 * * * *'  # Every 10 min

4. Hot-Path Aggregations - Fix #3: Materialized Views

DB-native precomputed views (PostgreSQL). Great for complex multi-table aggregations

Create Materialized View (PostgreSQL):

-- SQL migration
CREATE MATERIALIZED VIEW user_stats_mv AS
SELECT 
    COUNT(*) as total_users,
    COUNT(*) FILTER (WHERE created_at > NOW() - INTERVAL '30 days') 
        as new_users,
    COUNT(*) FILTER (WHERE is_active = true) as active_users,
    AVG(EXTRACT(YEAR FROM age(date_of_birth))) as avg_age
FROM auth_user;

-- Add index for faster reads
CREATE INDEX ON user_stats_mv (total_users);

💡 The view is precomputed and stored like a table

✅ Handles complex aggregations with JOINs, FILTERs, etc.

⚡ Reads are as fast as regular table queries

4. Materialized Views - Django Model

Create an unmanaged Django model to query the view

Django Model (unmanaged):

# models.py - represents the materialized view
class UserStatsMV(models.Model):
    total_users = models.BigIntegerField()
    new_users = models.BigIntegerField()
    active_users = models.BigIntegerField()
    avg_age = models.FloatField()
    
    class Meta:
        managed = False  # Django won't create/migrate this
        db_table = 'user_stats_mv'  # Points to our MV
        
    def __str__(self):
        return f"User Stats: {self.total_users} total"

💡 managed = False tells Django this table exists but not to touch it

✅ No migrations needed | 🔍 Use Django ORM normally

4. Materialized Views - Query & Refresh

Query the view like any other Django model

View - Query like any model:

def dashboard(request):
    stats = UserStatsMV.objects.first()
    return JsonResponse({
        'total': stats.total_users,
        'new': stats.new_users,
        'active': stats.active_users,
        'avg_age': stats.avg_age
    })  # Fast! No aggregation at query time

Refresh with Celery:

from django.db import connection

@shared_task
def refresh_user_stats():
    with connection.cursor() as cursor:
        cursor.execute("REFRESH MATERIALIZED VIEW user_stats_mv;")
        
# Schedule with Celery Beat - runs every hour

💾 Persistent | 📊 Complex queries | ⏰ Hourly/daily refresh

4. Hot-Path Aggregations - Choose Your Fix

💡 Pro tip: Combine strategies! Cache stats table reads for maximum speed.

🐍 Django specific: Use Celery Beat for scheduling all refresh tasks

5. Fat Payloads

Problem: Sending or receiving unnecessarily large data

💀 Both stress your server and hurt performance!

5. Fat Payloads - Problem #1: Heavy Output

Sending too much data in API responses

Before (Django): Returning everything

from rest_framework import serializers, viewsets

class UserSerializer(serializers.ModelSerializer):
    class Meta:
        model = User
        fields = '__all__'  # Returns ALL fields! 😱

class UserViewSet(viewsets.ModelViewSet):
    queryset = User.objects.all()
    serializer_class = UserSerializer
    
# Response for 100 users: 2.5MB
# Includes: profile_pic (base64), bio, preferences, metadata, etc.

5. Fat Payloads - Fix #1: Slim Output

Return only what's needed

After (Django): Field selection

class UserListSerializer(serializers.ModelSerializer):
    class Meta:
        model = User
        fields = ['id', 'name', 'email']  # Only essentials

class UserViewSet(viewsets.ModelViewSet):
    queryset = User.objects.only('id', 'name', 'email')  # DB level
    
    def get_serializer_class(self):
        if self.action == 'list':
            return UserListSerializer  # Slim for lists
        return UserDetailSerializer  # Full for detail view

For images:

# ❌ Don't do this
return {'image': base64.b64encode(image_data)}

# ✅ Do this
return {'image_url': f'https://cdn.example.com/imgs/{user.id}.jpg'}

📦 Payload: 2.5MB → 15KB (gzipped)

5. Fat Payloads - Problem #2: Heavy Input

Receiving large data in requests

Before (Django): Large JSON uploads

class BulkUploadView(APIView):
    def post(self, request):
        # Client sends 50MB JSON with 10,000 records + images
        data = request.data  # Loads entire payload into memory!
        
        for item in data['items']:  # 10,000 items
            User.objects.create(
                name=item['name'],
                profile_pic=item['image_base64']  # Huge!
            )
        return Response({'status': 'ok'})

5. Fat Payloads - Fix #2: Handle Input Properly

Use proper file handling + background processing

Solution: File upload + background processing

class BulkUploadView(APIView):
    def post(self, request):
        file = request.FILES.get('csv_file')  # Not JSON!
        
        # Stream file to S3/storage (doesn't load into memory)
        upload_path = default_storage.save(f'uploads/{file.name}', file)
        
        # Process in background worker
        process_bulk_upload.delay(upload_path)
        
        return Response({
            'job_id': '...',
            'status': 'processing'
        }, status=202)

# Background task processes file line by line
@shared_task
def process_bulk_upload(file_path):
    with default_storage.open(file_path, 'r') as f:
        for line in f:  # Stream processing
            data = json.loads(line)
            User.objects.create(name=data['name'])

✅ File streaming (no memory spike) | 🔄 Background processing

⚙️ Set request size limits: DATA_UPLOAD_MAX_MEMORY_SIZE = 10 * 1024 * 1024 (10MB)

6. Multiple Row Locking

Problem: Too many concurrent locks stress the database

Scenario: Processing 100 payments concurrently

# Cron job queues 100 individual payment tasks
@periodic_task(run_every=crontab(minute='*'))
def queue_pending_payments():
    payments = Payment.objects.filter(status='pending')
    for payment in payments:  # 100 pending payments
        process_single_payment.delay(payment.id)  # 100 tasks!

# Workers process payments concurrently
@shared_task
def process_single_payment(payment_id):
    # Each locks ONE row
    payment = Payment.objects.select_for_update().get(id=payment_id)
    charge_customer(payment)  # Takes 2-3 seconds
    payment.status = 'completed'
    payment.save()
    
# Problem: 100 concurrent locks stress the database! 😱

🔥 100 concurrent row locks = high DB contention and CPU usage!

6. Multiple Row Locking - Database Stress

6. Multiple Row Locking - The Fix

Limit concurrent processing or use optimistic locking

Current Setup: Cron queues 100 tasks, all run concurrently

@periodic_task(run_every=crontab(minute='*'))
def queue_pending_payments():
    payments = Payment.objects.filter(status='pending')
    for payment in payments:  # 100 pending payments
        process_single_payment.delay(payment.id)  # 100 tasks!
        
# All 100 run concurrently = 100 locks! 😱

Fix Option 1: Limit Celery worker concurrency

# Same code, but configure Celery workers
# celery -A myapp worker --concurrency=10

# Now only 10 payments processed at once
# Remaining 90 wait in queue
# Result: Max 10 concurrent locks instead of 100!

⚙️ Set CELERYD_CONCURRENCY=10 or use --concurrency flag

6. How Limiting Concurrency Works

With limited worker concurrency:

6. Fix #2: Optimistic Locking

No locks! Use version field to detect conflicts

Add version field to model:

class Payment(models.Model):
    amount = models.DecimalField(max_digits=10, decimal_places=2)
    status = models.CharField(max_length=20)
    version = models.IntegerField(default=0)  # Version field!

Process without locks:

@shared_task
def process_single_payment(payment_id):
    # Read WITHOUT locking
    payment = Payment.objects.get(id=payment_id)
    old_version = payment.version
    
    # Do expensive processing (no lock held!)
    charge_customer(payment)
    
    # Update only if version hasn't changed
    updated = Payment.objects.filter(
        id=payment_id,
        version=old_version  # Check version hasn't changed
    ).update(
        status='completed',
        version=old_version + 1  # Increment version
    )
    
    if not updated:
        # Someone else updated it, retry or skip
        raise Retry()

6. Optimistic vs Pessimistic Locking

💡 Tip: Use optimistic locking for 100 concurrent payments to eliminate DB lock stress!

7. Using Database for Search

Problem: Using SQL LIKE or full-text search for complex search needs

Django Example:

# Searching products with ILIKE/CONTAINS
def search_products(request):
    query = request.GET.get('q')
    products = Product.objects.filter(
        Q(name__icontains=query) | 
        Q(description__icontains=query) |
        Q(category__name__icontains=query)
    )
    return JsonResponse({'results': list(products.values())})

SQL Generated:

SELECT * FROM products 
WHERE name ILIKE '%laptop%' 
   OR description ILIKE '%laptop%'
   OR category ILIKE '%laptop%';
-- Full table scan on every search! 😱

7. Using DB for Search - Why It's Bad

7. Using DB for Search - The Fix

Use dedicated search engines: Elasticsearch, Algolia, Meilisearch, etc.

Example with Elasticsearch + Django:

from elasticsearch_dsl import Document, Text, Keyword
from elasticsearch_dsl.connections import connections

# Define search document
class ProductDocument(Document):
    name = Text(analyzer='standard')
    description = Text(analyzer='standard')
    category = Keyword()
    
    class Index:
        name = 'products'

# Search with Elasticsearch
def search_products(request):
    query = request.GET.get('q')
    search = ProductDocument.search().query(
        "multi_match", 
        query=query, 
        fields=['name^2', 'description', 'category'],  # Boost name
        fuzziness='AUTO'  # Handle typos!
    )
    results = search.execute()
    return JsonResponse({
        'results': [hit.to_dict() for hit in results]
    })

⚡ Elasticsearch handles millions of docs with sub-100ms search times

7. Search Engines - The Benefits

📊 Search performance: 3-5s → <100ms

Server Health Checklist ✅

• ☐ Offload long work to background queues
• ☐ Avoid OFFSET – use keyset pagination
• ☐ Kill N+1s with eager loading
• ☐ Cache or precompute heavy stats
• ☐ Trim API payloads
• ☐ Avoid locking many rows at once
• ☐ Use dedicated search engines, not DB LIKE queries

Final Thoughts

• Performance is systemic, not just code-deep.
• Your server is doing its best — help it breathe.
• Empathy isn't just for users… it's for infrastructure too.

I'm Looking for My Next Role! 🚀

If your team needs someone who:

• Builds scalable backend systems that don't stress out
• Optimizes for performance AND maintainability
• Actually cares about server health and infrastructure empathy
• Writes Python and Typescript like it's second nature

Let's talk!

💼 linkedin.com/in/abdulmateen-tairu

Scan for all my links
linktr.ee/tairuabdulmateen

Q&A

Let's talk backend pain.