Headless Chrome Limitations: Common Production Issues

Headless Chrome is easy to launch but difficult to operate in production. Everything works locally, but in a container, the browser suddenly crashes, tests in CI sometimes pass and sometimes fail, and a service that renders external links becomes a security risk. This article covers common problems and their solutions: from /dev/shm in Docker to protection against SSRF and crash tracing in Playwright. How everything works internally is explained in the article "How Headless Chrome Works".

⚡ In Brief: Where Headless Chrome Breaks Most Often

When working with Headless Chrome, it's important to consider not only the browser settings but also the specifics of the environment in which it is launched. I would first check memory usage, process lifecycle, and container parameters. These are the aspects worth diagnosing first if the browser is unstable or behaves differently in production than locally.

📚 Table of Contents

• What Exactly Limits Headless Mode
• Memory, Processes, and Resources
• Timeouts and Flaky Tests
• Docker and CI: /dev/shm and Sandbox
• Fonts and Rendering Discrepancies
• Complex Pages, Authorization, and Sessions
• When You're Not Allowed In: Legitimate Options
• Security: SSRF When Rendering External URLs
• Tracing and Debugging: Playwright Trace Viewer
• Production Checklist
• Frequently Asked Questions (FAQ)
• Conclusions
• Sources

🧠 What Exactly Limits Headless Mode

Today, the new headless mode is the same Chrome without visible windows, so most "limitations" are related not to the browser itself, but to the environment in which it is launched: container, memory, user rights, network. A separate story is chrome-headless-shell: it is a lightweight version with fewer dependencies, and according to the developers' description, it is less "authentic" than the full Chrome. The difference between the modes is detailed in the article "How Headless Chrome Works". Below we examine what breaks in practice. In each section, the scheme is the same: symptom, cause, solution options.

💾 Memory, Processes, and Resources

Symptom: over time, the service consumes more and more memory, and Chrome processes accumulate in the system.

Causes:

• Zombie processes. Playwright documentation recommends running the container with the --init flag to avoid the special behavior of processes with PID=1, which are a common cause of zombie processes.
• Unclosed browsers. Among the most common causes of failures in Docker are browser leaks: each launch of launch() creates a full Chrome instance that needs to be closed.

What to do:

• Close the browser in a finally block, even if the task failed.
• Keep one long-running browser and create a separate context for each task instead of launching a new browser every time.
• Limit parallelism and periodically restart the browser (e.g., after a certain number of tasks). Choose specific limits by measuring your load.
• Set container memory limits and monitor them in your monitoring system.

⏱️ Timeouts and Flaky Tests

Symptom: tests pass locally and fail in CI, screenshots are taken before content appears.

Cause: the code relies on time rather than page state. CI has fewer resources, and the page loads slower.

🐳 Docker and CI: /dev/shm and Sandbox

The /dev/shm Problem

Symptom: Chrome works stably on your computer, but crashes randomly or reports it's unavailable in the container. One of the analyses describes exactly these "random" crashes only in Docker.

Cause: According to Puppeteer documentation (archived version), Docker by default allocates only 64MB of shared memory /dev/shm to a container, and Chrome actively uses it.

How to check:

docker exec <container> df -h /dev/shm

If the size is 64MB, and it's almost full when it crashes, the diagnosis is confirmed.

Solution Options

Example for Docker Compose:

services:
  renderer:
    image: my-renderer
    init: true
    shm_size: "2gb"
    mem_limit: 4g

Example for Kubernetes:

spec:
  containers:
    - name: renderer
      image: my-renderer
      resources:
        limits:
          memory: "4Gi"
      volumeMounts:
        - name: dshm
          mountPath: /dev/shm
  volumes:
    - name: dshm
      emptyDir:
        medium: Memory
        sizeLimit: 2Gi

Kubernetes supports such memory-backed volumes, and their default size depends on the node's memory, so sizeLimit should be set explicitly. Data in such volumes uses the pod's memory, so the size must fit within the container's limit.

The 2GB values in the examples are approximate. Choose your own based on measurements: it depends on the number of tabs and page complexity.

Sandbox and User

The common advice "add --no-sandbox" solves the launch problem but removes important protection. It's described that the Playwright image runs as root by default, which disables Chromium's sandbox, unavailable with root. For untrusted sites, a separate user along with a seccomp profile is recommended. Official launch example for Java:

docker run -it --rm --init --ipc=host --user pwuser \
  --security-opt seccomp=seccomp_profile.json \
  mcr.microsoft.com/playwright/java:<version>-noble /bin/bash

The documentation also states that the seccomp_profile.json profile is needed to run Chromium with a sandbox, and this image is not recommended for visiting untrusted sites. Substitute the current image version from the documentation.

🔤 Fonts and Rendering Differences

Symptom: Text looks different on the server than locally, "squares" appear instead of characters in PDFs.

Cause: The minimal image lacks the necessary fonts. In the Dockerfile from Puppeteer documentation, fonts for major character sets (Chinese, Japanese, Arabic, Hebrew, Thai) are installed separately.

What to do:

• Install fonts for the languages you need (for a multilingual blog, this is at least Cyrillic, Latin, and, if necessary, CJK).
• Set the window size explicitly (e.g., --window-size in CLI) so the result doesn't depend on defaults.
• Fix the image and Chrome version. Different versions can yield different results.
• For PDFs, check print styles (@media print): they might differ from screen layout.

🧩 Complex Pages, Authorization, and Sessions

• iframes, Shadow DOM, pop-ups, file uploads. Different libraries handle them differently. Before choosing a tool, check on your pages how easy it is to work with such elements in the documentation for Puppeteer, Playwright, or Selenium.
• Authorization. Don't log in anew in each task. Playwright allows saving session state and reusing it. Store such files as secrets: they contain cookies.
• Isolation. A separate context per task prevents cookies and storage from one task from affecting another.
• Secrets in logs and screenshots. Don't print tokens to the console or save screenshots of pages with private data unnecessarily.

🚧 When You're Not Allowed In: Legal Options

Websites have the right to restrict automated access. If your headless client receives rejections, it's a reason to negotiate, not to disguise yourself. What works legally:

• Use the official API if available.
• Ask the website owner to add your client to an allowlist.
• Honestly identify your client: a clear User-Agent with contact information.
• Adhere to request rate limits and website terms of use.
• Test on your own environments (staging), not on someone else's production.

The robots.txt file is not an authorization mechanism, but respect it: more on robots.txt. And to see what such traffic looks like to a website owner, see the article "Suspicious Traffic in GA4".

🛡️ Security: SSRF When Rendering Other People's URLs

If your service opens user-provided addresses in headless Chrome (screenshots, PDFs, link previews), you create a classic SSRF (Server-Side Request Forgery) vector. OWASP describes SSRF as abusing an application to make requests to an internal or external network, or to the machine itself. In our case, your browser acts as the "application," and requests originate from your network.

Why a browser is riskier than a regular HTTP client. This is my conclusion from how Chrome works (OWASP doesn't mention browsers): a browser itself follows redirects, downloads sub-resources, and executes scripts of another page, which can also make requests to addresses within your network. Therefore, checking only the entered URL is not enough.

OWASP distinguishes two cases. In the first, the application should only access known trusted addresses, and an allowlist is suitable here. In the second, it should access any external addresses, and an allowlist is not feasible. A service that renders other people's URLs most often falls into the second category.

Protection Options

Minimum List Blocked in Option B

According to OWASP table:

Example of Address Verification in Java (Skeleton)

static boolean isPublic(InetAddress a) {
    if (a.isAnyLocalAddress() || a.isLoopbackAddress() || a.isLinkLocalAddress()
            || a.isSiteLocalAddress() || a.isMulticastAddress()) {
        return false;
    }
    byte[] b = a.getAddress();
    if (b.length == 16 && (b[0] & 0xFE) == 0xFC) return false;               // fc00::/7
    if (b.length == 4 && (b[0] & 0xFF) == 100 && (b[1] & 0xC0) == 64) return false; // 100.64.0.0/10
    return true;
}

// Resolve all addresses for the host and reject if any are not public
for (InetAddress a : InetAddress.getAllByName(host)) {
    if (!isPublic(a)) throw new SecurityException("Forbidden address: " + a);
}

This is just a start, not complete protection. OWASP warns that checking DNS responses separately does not protect against DNS rebinding: the address can change between the check and the connection. The client must connect only to the verified address, and a repeated, unfinished DNS query between verification and connection can bypass protection. This is why network isolation (option C) is a mandatory second layer: it works regardless of what DNS returns.

Additional Measures for Headless Browsers

• Allow only http and https. OWASP reminds that SSRF is not limited to HTTP and mentions schemes like file:// and gopher://.
• Intercept requests within the browser itself. In Playwright, this is done with page.route() and context.route(): you can reject requests to forbidden hosts, including sub-resources and redirects. However, interception sees the URL, not the actual IP address, so it complements network isolation, not replaces it. (Check the API in your version's documentation.)
• Do not expose the debugging port externally (--remote-debugging-port): it gives full control over the browser.
• Isolate the process. A separate container without secrets and without access to internal databases, an unprivileged user, limited task time and size.
• Do not disable the sandbox for pages you do not control.

🔧 Tracing and Debugging: Playwright Trace Viewer

The hardest thing about flaky failures is that they are impossible to reproduce. Playwright's Trace Viewer solves this by recording what happened. Playwright Trace Viewer is a graphical tool that allows you to inspect recorded traces after the script has finished. It shows a timeline of actions with screenshots, DOM snapshots for each action, source code, action log, errors, console, and network requests.

Trace Recording Options

Option C: Tracing in a Custom Service (Node.js)

const context = await browser.newContext();
await context.tracing.start({ screenshots: true, snapshots: true, sources: true });
const page = await context.newPage();
try {
  await page.goto(url);
  // ...working with the page
  await context.tracing.stop();                       // success: don't save the file
} catch (e) {
  await context.tracing.stop({ path: `trace-${jobId}.zip` });  // failure: save
  throw e;
} finally {
  await context.close();
}

The Same in Java

BrowserContext context = browser.newContext();
context.tracing().start(new Tracing.StartOptions()
    .setScreenshots(true).setSnapshots(true).setSources(true));
Page page = context.newPage();
try {
    page.navigate(url);
    // ...working with the page
    context.tracing().stop();
} catch (RuntimeException e) {
    context.tracing().stop(new Tracing.StopOptions()
        .setPath(Paths.get("trace-" + jobId + ".zip")));
    throw e;
} finally {
    context.close();
}

The examples are simplified and may not compile. Check the behavior of stop() without a path (the trace is discarded) and the exact class names in your Playwright version's documentation.

How to View Traces

• Locally: npx playwright show-trace trace.zip.
• In the browser: trace.playwright.dev — a statically hosted version of the viewer. You can also pass a trace URL uploaded to accessible storage there.
• Typical analysis order: open the trace of a failure, find the action where it failed, compare DOM snapshots before and after, then review the console and network.

Trace Security

Traces contain page snapshots and network activity, so they may contain personal data, tokens, and cookies (this is my warning, it's not explicitly stated in the documentation). Do not publish them publicly, limit their storage duration and access, and for pages with private data, mask sensitive fields or don't record traces at all.

If You Don't Use Playwright

• For Chrome, there's debugging in headless mode: launch the browser with a remote debugging port and connect DevTools. Keep the port closed to the external network.
• Minimum for any tool: a screenshot and console/network logs on error, a task ID in every log entry.
• Puppeteer has its own Chrome tracing, but it's designed for performance profiling, not step-by-step action reproduction like Playwright. Check the documentation for details.

✅ Production Checklist

• ☐ Container starts with --init (or init: true in Compose).
• ☐ /dev/shm is increased (--shm-size, --ipc=host, or memory-backed emptyDir) and aligned with memory limits.
• ☐ Browser runs as a non-root user, sandbox is not disabled.
• ☐ Browser is closed in finally, parallelism is limited, restarts after N tasks are in place.
• ☐ Fonts for required languages are installed, window size is explicitly set.
• ☐ Chrome and library versions are pinned.
• ☐ Waits are based on page state, not sleep.
• ☐ For external URLs: allowlist or denylist, IP check after DNS, only http/https, redirects to internal addresses are forbidden.
• ☐ Browser runs on a network without access to internal services, cloud metadata is protected (IMDSv2).
• ☐ Debugging port is not exposed externally.
• ☐ Trace or screenshot is recorded on failure, storage duration is managed by privacy policy.
• ☐ Metrics are present: rendering time, errors, restarts.

❓ Frequently Asked Questions (FAQ)

Why does Headless Chrome crash in Docker?

The most common reasons: small /dev/shm size (64MB by default), running as root which disables the sandbox, insufficient memory, and zombie processes. Start with df -h /dev/shm inside the container and the --init and --shm-size flags.

Why does a test pass locally but fail in CI?

Usually due to time dependencies and environment differences: fewer resources, different fonts, slower network. Replace sleep with state waits and enable trace: 'on-first-retry' to see exactly what happened.

How much memory is needed?

There's no universal number: it depends on the pages and the number of tabs. Measure consumption with your tasks, add a buffer, and set a container limit. Memory in /dev/shm is included in this limit.

Is it safe to render user-input URLs?

Only with SSRF protection: address validation, network isolation, cloud metadata protection, and scheme restrictions. Without these, a browser in your network becomes an entry point for an attacker.

Can I use --no-sandbox?

Technically yes, but it reduces security, so it's better to run the browser as a non-root user with an appropriate seccomp profile. For pages you don't control, disabling the sandbox is not recommended.

🏁 Conclusions

• 🔹 Most Headless Chrome issues are related to the environment, not the browser itself: container, memory, permissions, network.
• 🔹 For /dev/shm, there are several options: --shm-size, --ipc=host, memory-backed emptyDir, and --disable-dev-shm-usage should be a last resort.
• 🔹 It's better to configure the sandbox (non-root and seccomp) than to disable it.
• 🔹 If you render external URLs, SSRF becomes the main risk, and a single address validation check doesn't cover it: network isolation is required.
• 🔹 Playwright Trace Viewer allows post-mortem failure analysis: record traces on failure and store them as sensitive data.
• 🔹 Pin versions, measure consumption, and automate the checklist, and production with headless browsers becomes predictable.

📖 Read Also

• How Headless Chrome Works
• Suspicious traffic in GA4: how bots with headless browsers disguise themselves as humans and how to detect them
• Robots.txt: A Complete Guide for SEO and Site Optimization
• How to Distinguish a Bot from a Real User: A Complete Guide with Log Examples

📎 Sources

• Chrome for Developers: Headless mode
• Chrome for Developers: Headless Chrome shell
• Chrome for Developers: Debug in Headless mode
• Playwright (Java): Docker
• Playwright: Trace Viewer
• Playwright: Authentication
• Playwright: Auto-waiting
• Playwright: archived description of Docker image (root and sandbox)
• Puppeteer: Troubleshooting (archived version)
• Puppeteer: discussion of --disable-dev-shm-usage
• Browserless: Running Puppeteer in Docker without crashes
• DEV: WebdriverIO with headless Chrome in Docker
• Kubernetes KEP-1967: size of memory-backed volumes
• Huawei Cloud: changing the size of /dev/shm
• OWASP: Server-Side Request Forgery Prevention Cheat Sheet
• RFC 9309: Robots Exclusion Protocol

Important: Chrome versions, Playwright images, and library APIs change. Check the current documentation before use. Material is current as of October 2026.