TTFB: Why Time To First Byte is important for both SEO and LLMs

Time To First Byte (TTFB) is one of the most critical, yet overlooked, performance metrics for websites. While most focus on page load time or visual rendering, TTFB is the first indicator of how responsive your site is — and that matters both for search engines and AI crawlers. Slow TTFB can mean that AI systems crawl fewer pages, prioritize your content lower, or give up crawling your site entirely. This guide explains what TTFB is, why it matters, and how to optimize it.

What is TTFB?

Time To First Byte (TTFB) is the time from when a browser/crawler sends an HTTP request until it receives the first byte of data from the server.

Components of TTFB:

1. DNS lookup — Time to translate domain name to IP address

2. Connection time — Time to establish TCP connection

3. SSL/TLS handshake — Time to establish HTTPS connection (if applicable)

4. Server processing time — Time the server takes to generate response

5. Network latency — Time it takes for the first byte to travel back

Formula:

TTFB vs. other metrics

TTFB measures server responsiveness before any content is sent.

FCP (First Contentful Paint) measures when the first visual content renders.

LCP (Largest Contentful Paint) measures when the main element is loaded.

TTFB happens first — it is the foundation for all other load metrics.

Why TTFB matters for AI crawlers

1. Crawler budgets

AI crawlers (and search engines) have limited time per site. If your TTFB is slow, they crawl fewer pages before their time budget runs out.

Example:

• Site A: TTFB = 200ms → Crawler gets to crawl 500 pages in 5 minutes

• Site B: TTFB = 2000ms → Crawler only gets to crawl 150 pages in 5 minutes

Site A gets 3x more crawl coverage.

2. Crawl prioritization

Crawlers prioritize fast sites. If your TTFB is consistently slow, crawlers will:

• Return less frequently

• Crawl fewer pages per visit

• Prioritize competitors' sites

3. Data quality

Long TTFB can result in timeouts. AI gets incomplete data or gives up entirely.

Google's guidance:

• Good TTFB: < 600ms

• Okay TTFB: 600ms - 1200ms

• Bad TTFB: > 1200ms

The same applies to AI crawlers.

Why TTFB also matters for SEO

Google has confirmed that TTFB is a ranking factor via Core Web Vitals indirectly (it affects LCP).

Impact:

• Slow TTFB → Slow LCP → Worse ranking

• Fast TTFB → Fast LCP → Better ranking

Measure your current TTFB

Browser DevTools (Chrome)

1. Open Chrome DevTools (F12)

2. Go to Network tab

3. Refresh the page

4. Click on the first request (typically your HTML document)

5. See Timing tab

You will see:

PageSpeed Insights

1. Go to https://pagespeed.web.dev/

2. Enter your URL

3. See "Server response time" under diagnostics

WebPageTest

1. Go to https://www.webpagetest.org/

2. Enter your URL and select test location

3. See "Time to First Byte" in results

WebPageTest also shows:

• TTFB per request (not just first HTML)

• Breakdown of DNS, Connect, SSL, Wait

Command-line test with curl

# Measure TTFB
curl -o /dev/null -s -w "DNS: %{time_namelookup}s\nConnect: %{time_connect}s\nSSL: %{time_appconnect}s\nTTFB: %{time_starttransfer}s\nTotal: %{time_total}s\n" https://ditwebsite.dk/

# Output:
# DNS: 0.015s
# Connect: 0.045s
# SSL: 0.132s
# TTFB: 0.456s
# Total: 0.612s

Python script for bulk testing

import requests
import time

def measure_ttfb(url):
    start = time.time()
    r = requests.get(url, stream=True)
    ttfb = time.time() - start
    return ttfb * 1000  # Convert to milliseconds

urls = [
    "https://ditwebsite.dk/",
    "https://ditwebsite.dk/products/",
    "https://ditwebsite.dk/blog/"
]

for url in urls:
    ttfb = measure_ttfb(url)
    print(f"{url}: {ttfb:.2f}ms")

Identify what makes your TTFB slow

TTFB is broken down into several components. Identify bottlenecks:

1. DNS Lookup (should be < 20ms)

Problem: Slow DNS resolver.

Test:

Solution:

• Use fast DNS provider (Cloudflare DNS, Google DNS)

• Enable DNS prefetching:

<link rel="dns-prefetch" href="//ditwebsite.dk">

2. Connection time (should be < 50ms)

Problem: Geographic distance to server or slow network.

Solution:

• Use Content Delivery Network (CDN) to be closer to users/crawlers

• Reduce number of connections (HTTP/2 helps)

3. SSL/TLS handshake (should be < 100ms)

Problem: Slow SSL negotiation.

Test:

openssl s_time -connect ditwebsite.dk:443 -www

Solution:

• Enable TLS 1.3 (faster handshake)

• Use OCSP stapling

• Enable HTTP/2 or HTTP/3

4. Server processing time (should be < 200ms)

Problem: Slow backend or database queries.

This is typically the biggest bottleneck.

Identify:

• Check server logs for slow requests

• Use APM tools (New Relic, Datadog)

• Profile database queries

Optimize server processing time

This is where you get the biggest impact.

1. Implement caching

Server-side caching:

Use Redis or Memcached to cache database results.

Example (PHP with Redis):

<?php
$redis = new Redis();
$redis->connect('127.0.0.1', 6379);

$cache_key = 'homepage_data';
$data = $redis->get($cache_key);

if (!$data) {
    // Data not in cache, fetch from database
    $data = fetch_from_database();
    $redis->setex($cache_key, 3600, serialize($data));  // Cache for 1 hour
} else {
    $data = unserialize($data);
}

echo json_encode($data);

Full-page caching:

Cache entire HTML responses.

Nginx example:

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=my_cache:10m max_size=1g inactive=60m;

server {
    location / {
        proxy_cache my_cache;
        proxy_cache_valid 200 60m;
        proxy_cache_key "$scheme$request_method$host$request_uri";
        proxy_pass http

2. Optimize database queries

Problem: N+1 queries or unindexed queries.

Identify slow queries (MySQL):

-- Enable slow query log
SET GLOBAL slow_query_log = 'ON';
SET GLOBAL long_query_time = 1;  -- Queries over 1 second

-- View slow queries
SELECT * FROM

Solutions:

• Add indexes:

-- Before: Full table scan
SELECT * FROM products WHERE category = 'electronics';

-- After: Create index
CREATE INDEX idx_category ON products(category)

• Use query caching:

-- Enable query cache (MySQL 5.7)
SET GLOBAL query_cache_size = 67108864;  -- 64MB
SET GLOBAL query_cache_type = ON

• Optimize JOINs:

-- Bad: Multiple joins without proper indexes
SELECT * FROM orders
JOIN users ON orders.user_id = users.id
JOIN products ON orders.product_id = products.id
WHERE users.country = 'DK';

-- Good: Add composite indexes
CREATE INDEX idx_user_country ON users(country, id);
CREATE INDEX idx_product_id ON products(id)

3. Upgrade server resources

Check server load:

top
# See CPU usage and memory usage

htop
# More user-friendly alternative

If consistently high (>80% CPU):

• Upgrade CPU

• Add more RAM

• Use load balancing

4. Use a CDN

Content Delivery Networks cache content geographically close to users/crawlers.

Popular CDNs:

• Cloudflare — Free tier available, excellent for small-medium sites

• AWS CloudFront — Highly scalable

• Fastly — Great performance, more technical

• BunnyCDN — Budget-friendly

How CDN improves TTFB:

Without CDN:

• User in US requests site hosted in Europe

• TTFB = 500ms (200ms network latency + 300ms server processing)

With CDN:

• User in US requests → CDN edge server in US

• TTFB = 150ms (20ms network latency + 130ms cache hit)

5. Enable Gzip/Brotli compression

Compression reduces response size, speeding up transfer time.

Nginx config:

gzip on;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
gzip_min_length 256;
gzip_comp_level 6;

# Even better: Brotli
brotli on;
brotli_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text

Apache config:

<IfModule mod_deflate.c>
  AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css text/javascript application/javascript application/json
</IfModule>

6. Reduce server-side processing

Minimize:

• Heavy computations on each request

• External API calls

• File I/O operations

Example (Python Flask):

from flask import Flask, render_template
from functools import lru_cache
import requests

app = Flask(__name__)

# Bad: External API call on every request
@app.route('/weather')
def weather_bad():
    response = requests.get('https://api.weather.com/data')
    return render_template('weather.html', data=response.json())

# Good: Cache external API call
@lru_cache(maxsize=128)
def fetch_weather_cached():
    response = requests.get('https://api.weather.com/data')
    return response.json()

@app.route('/weather')
def weather_good():
    data = fetch_weather_cached()
    return render_template('weather.html', data=data)

7. Use HTTP/2 or HTTP/3

HTTP/2 allows multiplexing, reducing connection overhead.

Enable HTTP/2 (Nginx):

server {
    listen 443 ssl http2;
    server_name ditwebsite.dk;
    # ... rest of config

Test if HTTP/2 is enabled:

curl -I --http2 https://ditwebsite.dk/
# Look for "HTTP/2 200" in response

Optimize specifically for AI crawlers

1. Prioritize important pages

Ensure your most important pages (homepage, key products, top articles) have the fastest TTFB.

Nginx: Separate cache for key pages:

location = / {
    proxy_cache_valid 200 120m;  # Cache homepage for 2 hours
    proxy_pass http://backend;
}

location /products/ {
    proxy_cache_valid 200 60m;  # Cache products for 1 hour
    proxy_pass http

2. Detect and optimize for crawlers

Serve cached version to crawlers:

<?php
$user_agent = $_SERVER['HTTP_USER_AGENT'];

$is_crawler = preg_match('/GPTBot|ClaudeBot|CCBot|Googlebot|bingbot/i', $user_agent);

if ($is_crawler) {
    // Serve static cached HTML
    echo file_get_contents('/cache/static_page.html');
} else {
    // Serve dynamic content
    include('dynamic_page.php');
}

3. Pre-warm cache before crawler visits

If you know when crawlers visit (e.g., after sitemap submission), pre-warm the cache:

#!/bin/bash
# Warm cache by requesting all URLs
while read url; do
  curl -s "$url" > /dev/null
done

Monitor TTFB continuously

Setup automated monitoring

Pingdom:

1. Create account at pingdom.com

2. Add URL checks

3. Set alert threshold (e.g., "alert if TTFB > 800ms")

UptimeRobot:

1. Free monitoring every 5 minutes

2. Email alerts

Custom script (cron job):

#!/bin/bash
# Check TTFB and alert if > 1000ms
TTFB=$(curl -o /dev/null -s -w '%{time_starttransfer}' https://ditwebsite.dk/)
TTFB_MS=$(echo "$TTFB * 1000" | bc)

if (( $(echo "$TTFB_MS > 1000" | bc -l) )); then
  echo "TTFB too high: ${TTFB_MS}ms" | mail -s "TTFB Alert" admin@ditwebsite.dk
fi

Run every hour:

crontab -e
# Add:
0

TTFB targets

Google's recommendations

• Good: < 600ms

• Needs improvement: 600ms - 1200ms

• Poor: > 1200ms

For AI crawlers (recommendations)

• Excellent: < 300ms (gets highest crawl priority)

• Good: 300ms - 600ms (normal crawl rate)

• Acceptable: 600ms - 1000ms (lower crawl rate)

• Poor: > 1000ms (risk of incomplete crawls)

Common TTFB issues and fixes

Implementation Checklist

Use this checklist to improve TTFB:

1. Measure baseline TTFB — Use PageSpeed Insights or WebPageTest

2. Identify bottleneck — DNS, connection, SSL, or server processing?

3. Implement server-side caching — Redis/Memcached for database results

4. Enable full-page caching — For static or semi-static pages

5. Optimize database queries — Add indexes, remove N+1 queries

6. Enable Gzip/Brotli — Compress responses

7. Use CDN — Cloudflare, AWS CloudFront, or BunnyCDN

8. Enable HTTP/2 — Reduce connection overhead

9. Monitor TTFB — Setup Pingdom or custom monitoring

10. Test with curl — Verify improvements

Conclusion

TTFB is a critical metric both for SEO and AI visibility. Slow TTFB means fewer pages crawled, lower prioritization, and potentially incomplete data for AI systems. Most sites can improve TTFB with caching (server-side and full-page), database optimization, and use of CDN.

Start by measuring your current TTFB, identify the biggest bottleneck (typically server processing), and implement caching. For most sites, it's possible to get under 600ms with basic optimization, and under 300ms with CDN and aggressive caching.

Remember: TTFB is the foundation — all other performance metrics depend on it. Optimize TTFB first, then FCP and LCP.