FuzeFEATUREDLoad test at 75 users: a fail-closed JWT revocation check logged out valid users
ADOPTED
FuzeFEATURED
Load test at 75 users: a fail-closed JWT revocation check logged out valid users
Question
Where does the single-worker gevent deployment break under concurrent authenticated load, and why?
Method
Disposable local rig in Docker: Postgres+pgvector, Redis, the real 'gunicorn --workers 1 --worker-class gevent' command in a Linux container, 10 users x 40 bookmarks seeded, JWTs minted directly to bypass the login rate limiter, Locust (scripts/locustfile.py) hitting bookmarks, dashboard, text and semantic search, unified-orchestrator recommendations and health. Ran 25 and 75 concurrent users on a laptop Docker VM of about 3.8 GB with shared CPU.
Result
At 75 concurrent users about 38% of authenticated requests returned 401 'Token has been revoked' for tokens that were never revoked (approximate figure from gaps.md and the commit message; no raw Locust output is committed). The author's diagnosis: the Redis pool (max_connections 20, shared by caching, rate limiting and the revocation check) was exhausted and .exists() raised. check_if_token_revoked in run_production.py returned True on any exception, so every failure counted as revoked. Fix in 764e2c6: fail open on Redis errors or a missing connection (the check runs only after signature and expiry validate) and raise the pool to 50. A re-run at 75 users reported 0 failures in 595 requests; because both changes landed together and fail-open cannot return this 401, that does not prove the pool is no longer exhausted. At 25 users the author reports median latency under 100 ms with no failures; at 75 users, 2-5 s, which the author calls consistent with the single gunicorn worker (two workers not re-tested).
Numbers
- authenticated 401 rate at 75 users before fixgaps.md section 13; commit 764e2c6 message. Author-reported; no raw output committed.
- about 38% (approximate as written)
- failures at 75 users after fixgaps.md@764e2c6 line 401. Fail-open and pool 50 were applied together.
- 0 failures across 595 requests
- Redis pool sizegit show 764e2c6 -- backend/utils/redis_utils.py
- 20 to 50 connections
- median latency at 25 usersgaps.md section 13 (author-reported)
- under 100 ms, 0% failures
- median latency at 75 usersgaps.md section 13 (author-reported, laptop Docker VM of about 3.8 GB)
- 2-5 s
Evidence
- commit764e2c6 ↗run_production.py: fail-open check with a comment quoting the 38% figure. redis_utils.py: max_connections 20 to 50.
- filelocustfile.py@764e2c6 ↗Load script using pre-minted JWTs.
- docgaps.md@764e2c6 ↗Section 13 with the method and both runs.
- fileseed_loadtest_data.py@764e2c6 ↗NUM_USERS = 10, BOOKMARKS_PER_USER = 40; mints JWTs with create_access_token.
How this was checked: Read the run_production.py and redis_utils.py diffs, the locustfile and gaps.md section 13. Not re-run: needs Docker (daemon not running here) and Postgres+Redis. No regression test pins the fail-open behaviour (no 'revoked_jti' in tests). Numbers are from a laptop VM, so only the shape transfers to real hosting; the fail-open choice trades some revocation enforcement during Redis trouble for availability. Attribution: Commits 764e2c6, a74c00c and 491a221 carry a 'Co-Authored-By: Claude Sonnet 5' trailer, and gaps.md is an AI-written first-person document (it addresses the repository owner as 'you').
Recorded at the time (a document or commit message states it) · reasons are stated in the repository · commit 764e2c6