LLM Chat Memory Loss: How 300 Playwright Tests Caught the Real Culprit

At 2 AM I was jolted awake by a user ticket: “I chatted with the AI for 40 minutes about my requirements, then refreshed the page — and it forgot everything, asking me to introduce myself again.” My gut reaction was that the conversation was definitely in Redis. Our dev environment never showed this bug. But this kind of intermittent amnesia was impossible to reproduce manually — until I wrote a Playwright end-to-end test, ran it 300 times, and finally surfaced the bug that had been hiding for three days.

Breaking Down the Problem

Implementing memory persistence for an LLM chat boils down to storing the conversation history so that when the user returns, we can attach previous messages and avoid context breaks. We used the classic approach: bind a session_id to a conversation list, push every message into a Redis list on the backend, and set a 7-day expiration. It looked bulletproof.

But the user reports had two odd patterns:

Not every refresh lost data — most of the time a refresh was fine, but occasionally the whole conversation vanished, as if someone hit a reset button.
It only happened after long conversations — short exchanges followed by a refresh never failed, but after a dozen or more turns, refreshing the page caused memory loss with high probability.

Conventional unit tests only verified the “write, then read” path. They never covered the real user sequence: send a message → wait for the reply → page renders → user refreshes → reload. Manual testing could never land on the exact critical moments like “the async write hasn’t finished yet” or “the Redis key is just about to expire.”

What I lacked wasn’t another caching strategy; it was an automated test that could replay the exact user chat timing, and freely control time and page lifecycle.

Solution Design

There are plenty of tools that can drive a real browser: Selenium, Cypress, Playwright. Why Playwright?

Multi-browser + multiple contexts: I needed to simulate the same user opening and closing tabs repeatedly. Playwright’s browser_context gives you isolated sessions out of the box—create them, use them, discard them—exactly like real user behavior.
Network control: Intercept requests and verify when writes complete. Cypress can do this too, but Playwright’s page.wait_for_response felt more intuitive.
Time mocking: I didn’t end up using the native clock API this time, but Playwright has built-in clock utilities to simulate idle waits, avoiding brittle sleep calls in tests.

The overall plan: write a parameterized test with pytest-playwright that simulates “multi-turn chat → forced refresh → check if memory survived,” with full-stack assertions on the backend’s async writes and Redis expiry policies. If history gets lost during the test, immediately dump a screenshot and the network logs for diagnosis.

Core Implementation

To reliably reproduce the issue, I first built a minimal chat backend with two endpoints: POST /chat sends a message and returns an AI response, and GET /history?session_id=xxxfetches the full conversation. Memory lives in Redis, and the session_id is maintained via a cookie. Here’s the intentionally-buggy backend (contains the defects):

# app.py - 有问题的后端，只为复现 bug
import asyncio
import uuid
from flask import Flask, request, jsonify, make_response
import redis.asyncio as redis

app = Flask(__name__)
r = redis.Redis(host='localhost', port=6379, decode_responses=True)

async def save_message(session_id, role, content):
    # 模拟大模型生成回复的异步写入
    await r.rpush(f"chat:{session_id}", f"{role}:{content}")
    # 只要写过一次就设过期，但这里没有续期逻辑
    await r.expire(f"chat:{session_id}", 7*24*3600)

@app.route('/chat', methods=['POST'])
async def chat():
    data = request.json
    session_id = request.cookies.get('session_id', str(uuid.uuid4()))
    user_msg = data['message']
    # 存用户消息 —— 注意这里没有 await！
    asyncio.create_task(save_message(session_id, 'user', user_msg))
    # 模拟 AI 回复（简化为固定话术）
    bot_reply = f"你说的是：{user_msg}，我记得之前聊过。" if await r.llen(f"chat:{session_id}") > 1 else "请开始你的问题。"
    # 同样不等待 bot 消息写入
    asyncio.create_task(save_message(session_id, 'bot', bot_reply))
    resp = make_response(jsonify({"reply": bot_reply, "session_id": session_id}))
    resp.set_cookie('session_id', session_id)
    return resp

@app.route('/history', methods=['GET'])
async def history():
    session_id = request.cookies.get('session_id', '')
    msgs = await r.lrange(f"chat:{session_id}", 0, -1)
    return jsonify(msgs)

if __name__ == '__main__':
    app.run(port=5000)

What this code demonstrates: a persistence implementation that “looks like it works” but contains two fatal bugs. asyncio.create_task fires and forgets – the main thread returns a response before the writes complete. And expire is set only once when the key is first created; subsequent messages never refresh the TTL. Any slight network or CPU delay, and the Redis data may not be written when the user refreshes, or the key may silently expire.

Next comes the Playwright test script, designed to reliably reproduce the “refresh wipes memory” scenario:

# test_memory.py
import re
from playwright.sync_api import Page, expect
import pytest

@pytest.mark.parametrize("rounds", [5, 15])   # 短聊、长聊两种场景
def test_chat_memory_survives_refresh(page: P