Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Rust vs Python Keystone comparison

This page compares the Rust Keystone (keystone-rs) with the Python Keystone (keystone-py) under the same synthetic load on the same machine. It answers one question: how much more load does the Rust implementation serve than the Python one on the same hardware, and what does it cost in memory.

The numbers are a snapshot from one development machine. Each load step ran once, so differences of a few percent are noise. Treat the figures as relative, not absolute.

Summary

One pod of each implementation, both with requests = limits = 24 CPU on a single-node k3s host with 32 cores, the same Postgres, and a closed-loop virtual-user (VU) load that is stepped up until the 95th percentile (p95) latency reaches 500 ms.

RustPythonRust advantage
VUs at p95 = 500 ms, default mix (3% logins)~719~1883.8x
Throughput at that point, default mix~1,955 req/s~515 req/s3.8x
VUs at p95 = 500 ms, no password logins~1,667~1878.9x
Throughput at that point, no password logins~3,455 req/s~475 req/s7.3x
VUs per CPU core, no password logins69.57.88.9x
Memory, idle~260 MiB~2.7 GiB~10x less
Memory, 600 VUs (average / max)385 / 429 MiB-
Memory, 2,100 VUs (average / max)440 / 634 MiB-
Memory, 50 to 400 VUs (range of averages)-2.6 to 2.8 GiB

What this shows:

  • Capacity. With the default mix Rust carries about 3.8x the VUs and throughput of Python at the same p95 limit. The no-login figures are the better comparison of the implementations, see Why password hashing is reduced. The password logins are the expensive part for Rust (hashing is CPU-bound and runs in a bounded pool), so removing them moves Rust from ~719 to ~1,667 VUs. Python does not move (~188 to ~187), which means its limit is not the hashing.
  • Memory. Rust runs in one process and shares everything between async tasks. Python pays a full copy of the application for each of its 24 worker processes. The Rust figure is roughly 260 to 640 MiB for the whole range of loads tested.
  • Reliability. Rust had no failed requests at any step of the sweeps. Python returned about 2% HTTP 500 and 502 responses under load.
  • Equal load. At the same 20 and 50 VUs, Rust answers token validation in 13 to 27 ms (p50) and 16 to 34 ms (p95); Python with its shared cache needs 16 to 55 ms and 90 to 159 ms, and 740 to 1,650 ms without the cache. See Equal-load comparison.
  • Python ceiling. Python’s throughput is flat at about 500 req/s regardless of the VU count. It may not be CPU-bound.

Why password hashing is reduced for the comparison

Password hashing is an external load, not a property of either implementation. Verifying a password costs a fixed amount of CPU, set by the hashing algorithm and its cost parameters (the configured work factor), and it deliberately takes tens of milliseconds. Both servers have to do the same work for every login, and the cost does not depend on the language, the web framework, the database access or the caching.

When logins are a noticeable share of the mix, that fixed cost dominates the CPU budget. In the default mix only 3% of the operations are logins, but each one costs far more CPU than any other operation (removing them roughly doubles the VUs Rust carries), so the VU count at p95 = 500 ms partly measures how many hashes the machine can compute. The other operations (token validation, reads, role assignment) are what the implementations differ in, and their differences are hidden behind the hashing.

The comparison therefore reports two views:

  • Default mix (3% logins). Closer to a deployment where users log in with passwords all the time. It shows the result with the hashing cost included.
  • No password logins (LOADTEST_VU_HASH_PCT=0). Isolates the logic and the framework: request handling, authorization, database round trips and revocation checks. This is the better measure of the performance benefit of the implementation, and it is the figure used for the VUs-per-core comparison.

Production deployments also reduce the share of password logins by design: clients keep a token until it expires instead of a password for each request. A mix dominated by hashing is not typical of the steady state, so the reduced-hashing numbers are the more realistic indication of the benefit.

The Rust implementation also bounds the hashing in a separate pool (with the queue set to wait in these runs), so that hashing does not starve the other requests; the Python implementation hashes in the request worker. This is one more reason the default mix mixes two effects: the cost of the hash itself and how each implementation schedules it.

Method

  • Cluster: single-node k3s (32 cores) deployed with skaffold, see Local development. Both implementations use the same Postgres (CloudNativePG).
  • Rust: StatefulSet with one pod, --release build, policies evaluated in the process by regorus ([api_policy] engine = regorus), jemalloc via LD_PRELOAD, telemetry off, [database] max_pool_size as in the skaffold setup, password hashing pool queue full behaviour set to wait (queue instead of answering 503), CPU requests = limits = 24.
  • Python: Deployment with one pod, uwsgi with 24 single-threaded processes, one memcached shared by all workers, CPU requests = limits = 24.
  • Python threads (--enable-threads) were tried earlier and are not recommended: a 4-process x 4-thread variant used less memory per pod but returned thousands of HTTP 502 responses returned by the wsgi server.
  • Load: tests/loadtest (Goose 0.19) with LOADTEST_PROFILE=vu. Every VU is an end user of a tenant of its own, with its own project, role assignments and token, so the caches of the server see realistic variety. Operations are picked per request according to the mix below.
  • Sweep: the VU count is raised step by step (600, 900, 1200, 1500, 1800, 2100 and up to 2700 for Rust; much lower for Python), 2 minutes per step plus warm-up. The sweep stops when p95 exceeds 800 ms. The VU count at p95 = 500 ms is interpolated between the two steps that bracket it.
  • Metric: per-pod working set from kubectl top, sampled during the run.

VU operation mix

The mix is configurable through environment variables. Defaults:

VariableDefaultOperation
LOADTEST_VU_VALIDATE_PCT75Validate a token (GET /v3/auth/tokens)
LOADTEST_VU_ADMIN_READ_PCT13Admin read
LOADTEST_VU_HASH_PCT3Password login (hashing)
LOADTEST_VU_ASSIGN_PCT2Role assignment
LOADTEST_VU_CREATE_PCT0.2Create resources
LOADTEST_VU_REVOKE_PCT0.02Revoke a token

The remainder is spread over other reads. “No password logins” means LOADTEST_VU_HASH_PCT=0.

Rust results

Default mix

p95 = 500 ms is reached at about 719 VUs and about 1,955 req/s. CPU is the limit: the 24 cores are saturated, and a password login costs far more CPU than any other operation. There were no failed requests at any step.

VUsreq/sp50 (ms)p95 (ms)p99 (ms)
3001,999141206397
4501,980219311466
6001,962299409548
7501,952372524663
9001,936450622802
1,0501,909532720966

Throughput is nearly flat from 300 to 1,050 VUs: the server is already saturated at the lowest step, which fits the bcrypt cost of the logins being the limit.

No password logins

Throughput and latency per step. There were no failed requests at any step.

VUsreq/sp50 (ms)p95 (ms)p99 (ms)
6003,599172178194
9003,536260266540
1,2003,526344356851
1,5003,4724344501,163
1,8003,4415205401,490
2,1003,3996066381,753
2,4003,3417047702,097
2,7003,3237868682,490

p95 = 500 ms at about 1,667 VUs (69.5 VUs per core) and about 3,455 req/s. Throughput slowly decreases as VUs grow while the latency rises linearly. That is the shape of a CPU-bound server with a closed-loop client: extra VUs only queue. The tail is wide at high load: p99 grows to 2.1 to 2.5 s at 2,400 and 2,700 VUs, far above p95. It was not investigated.

Where the CPU goes

A request costs about 3.6 ms of CPU without hashing. From a CPU profile under load:

  • Around 12.6% of CPU is in the kernel, mostly socket I/O to Postgres.
  • A validate that uses two tokens (one to authenticate, one to validate) issues 15 database queries with a cold role cache: 4 for the authenticating token and 11 for the validated one. The database spends 0.005 to 0.4 ms per query and about 0.7 ms in total. The queries are round trips, not work.
  • For comparison, Python Keystone validating the same token with a cold cache (memcached flushed, caller and subject being the same token) issues 36 to 39 data SELECT statements (36 with the catalog, 39 with nocatalog, measured once each). They run in about 20 transactions, each with a connection pre-ping, so Postgres sees about 99 statements (BEGIN, ROLLBACK, SELECT 1 and the data queries) and the validation takes 70 to 290 ms. Per resource it loads the project, its options and tags, the role, its options and the implied roles four times each, the assignments three times, the user and its options twice, then groups, federation data, endpoint filters and the revocation events. With a warm cache the same validation issues no queries at all (29 ms); everything comes from memcached. Rust Keystone deliberately has no cross-request cache to guarantee immediate access revocation, so it pays its 15 queries on every request, while Python pays 36 to 39 only on a miss. This is why the cache hit rate (64% in the earlier shared-memcached runs, where the load creates new keys faster than any cache size keeps them) decides how much database work Python does per request.
  • The revocation check is one SELECT EXISTS over a UNION ALL of one branch per token dimension (audit id, user, project, domain, role, trust).

Removing latency alone does not raise the ceiling of a CPU-bound server. Fewer queries per request reduce client and kernel CPU and would, by a few percent.

Python results

Python reached p95 = 500 ms at about 188 VUs (default mix) and about 187 VUs (no logins), at about 7.8 VUs per core. Throughput at that point is about 515 req/s with the default mix and about 475 req/s without password logins; it is the same within noise because Python does not spend its limit on hashing. About 2% of the requests failed with HTTP 500 or 502 under load.

No password logins, per step (one run each):

VUsThroughput (req/s)p95 (ms)p99 (ms)Failed
1005172823232.0%
2004715325811.8%
3004607788682.0%
4004341,0641,3432.0%

Throughput is flat to slowly falling while the latency grows with the VU count, as for Rust. Python’s memory stays at about 2.7 GiB at every step.

A shared cache is the only valid Python topology, see Python cache topology.

Equal-load comparison

The sweeps above load each side up to its own latency limit, so the VU counts differ. This run puts the same number of VUs on both sides: 20 and 50 VUs, 2 minutes each, no password logins, one pod per side with requests = 4 and limits = 24 CPU (the loadtest skaffold profile). All requests, including Python’s, go through the traefik ingress. Rust was run with both policy engines: the OPA sidecar and the in-process regorus engine (the current default of the skaffold deployment). Values are p50 / p95 / p99 in ms.

20 VUsRust (OPA sidecar)Rust (regorus)Python (cache)Python (no cache)
Throughput (req/s)1,6681,75666830
Failed requests001,85313
All requests12/15/1912/15/1916/88/109729/851/868
Token validate13/16/1712/15/1616/90/103741/856/872
Role assignments, effective5.7/8.1/9.45.2/7.4/8.636/71/117362/469/483
Role assignments, scope.project.id5.0/7.3/8.64.5/6.5/7.818/51/60342/453/457
Memory (MiB)150-225257-3332,414-2,6682,306-2,744
50 VUsRust (OPA sidecar)Rust (regorus)Python (cache)Python (no cache)
Throughput (req/s)1,9602,03463433
Failed requests001,63327
All requests25/33/3725/32/3656/157/1961,605/1,769/1,818
Token validate27/34/3826/32/3655/159/1891,647/1,794/1,843
Role assignments, effective15/20/2315/19/2173/136/1671,253/1,335/1,369
Role assignments, scope.project.id14/18/2113/17/1955/118/1481,204/1,319/1,367
Memory (MiB)208-241317-3412,683-2,7502,672-2,698
  • With the shared cache on, Python’s median is close to Rust’s (token validate p50 16 vs 12 ms at 20 VUs). The difference is in the tail, about 5 to 6x at p95 and p99 for token validation, and in throughput, 2.6 to 3.2x (Rust with regorus). Listing role assignments is 4 to 7x slower at p50.
  • Python’s throughput is about 650 req/s at both load levels, so it is already at its ceiling at 20 VUs. Rust still grows from 20 to 50 VUs.
  • With the cache off, Python falls to about 30 req/s, 20x lower, and latency grows with the VU count: the server is saturated. Memory does not change with the cache, so the cache is not what costs Python its memory.
  • Almost all Python failures with the cache on are HTTP 502 (1,806 at 20 VUs, 1,544 at 50 VUs), about 2% of requests. uwsgi --http-socket closes the connection after every response, and a client that reuses a pooled connection fails with IncompleteMessage when it talks to uwsgi directly. This is the likely source of the 502s behind traefik, but it was not confirmed.
  • Regorus against the OPA sidecar: 5% and 4% more throughput at 20 and 50 VUs and about 1 ms lower p95 for role assignments; token validation and the overall latency are within 1 to 2 ms. That is inside the run-to-run noise, so the policy engine does not change the result at this load. Regorus uses about 100 MiB more memory because the compiled policies live in the process.

The Python no-cache column was measured with OS_CACHE__ENABLED=false on the same pod. It is a measurement of the cost of the cache, not a supported topology.

Memory

StateRust (1 pod)Python (1 pod, 24 processes)
Idle~260 MiB~2.7 GiB
600 VUs (average / max)385 / 429 MiB-
2,100 VUs (average / max)440 / 634 MiB-
50 VUs (average / max)-2,635 / 2,727 MiB
200 VUs (average / max)-2,762 / 2,768 MiB
400 VUs (average / max)-2,768 / 2,769 MiB

The Rust rows are from the no-login sweep and include the warm-up of each step. The compiled policies live in the process, which is part of the footprint.

The Python rows are from the default-mix sweep; the no-login sweep gives the same values (2.74 to 2.76 GiB at 200 to 400 VUs). The Python loads are lower because that is where Python reaches its latency limit, so the two sides are not measured at the same VU count.

Rust memory grows with the number of connections and in-flight requests, not with the number of workers. Python memory is dominated by the per-process copy of the application and does not depend on the cache: earlier 15-minute runs with three pods and 10 processes each measured 1.2 GiB per pod with the cache off, with a small cache and with a large one.

To deploy keystone-rs with a small footprint: set a CPU limit (tokio sizes its pool from the cgroup quota), keep telemetry sampled or off, and use jemalloc (tested through LD_PRELOAD of the Debian libjemalloc2; shipping it would mean using tikv-jemallocator as the global allocator).

Python cache topology

Python Keystone must use one cache shared by all pods. Token validation relies on the revocation events being visible to every API instance. A memcached instance per pod gives every pod a private cache: a token revoked through one pod stays valid in the caches of the others until it expires. That breaks token revocation, is an anti-pattern, and must not be used in a real deployment. All Python figures on this page use the shared memcached, except the explicit no-cache column of the equal-load comparison.

Limitations

  • Single run per load step, on one development machine. The load generator runs on the same host as both servers and Postgres.
  • The load is closed-loop, so request counts depend on latency. The VU count at a fixed p95 is the comparison metric, not the raw request count.
  • The two sides do not serve identical APIs: the load is restricted to /v3 operations that both implement.
  • The Python failures (about 2% HTTP 500 and 502) and its flat ~500 req/s ceiling were not investigated deeper other than reproduced in different constellations.
  • Rust and Python run with telemetry off.
  • The database accumulates about 6,000 to 6,800 password rows from the logins of the load test.
  • The sweeps ran with requests = limits = 24 CPU; the equal-load run used requests = 4, limits = 24 so both versions can be deployed at the same time. The limit, which decides the throughput ceiling, is the same.

Reproducing

cd tests/loadtest
# needs the admin credentials in OS_* variables or OS_CLOUD
LOADTEST_PROFILE=vu \
LOADTEST_VU_HASH_PCT=0 \
cargo run --release -- --host <url> --users 1200 --run-time 2m \
  --report-file reports/<name>.md

Raise --users step by step, keep each step in its own report, and read the p95 from the report. Sample memory during the run with tests/loadtest/sample_metrics.sh. Redirect the output to a file and do not pipe the run through tail.