Skip to main content

Telephone Load Testing Guide

Step-by-step instructions for running the telephone (IVR/DTMF) load test: provisioning an election event with many voters, then driving many simulated phone calls against it. The setup script provisions through step-cli; the runner starts independent ivr-cli processes against real Keycloak and Hasura services, without telephony infrastructure.

All commands below assume a terminal opened inside the dev container (VS Code Dev Containers or GitHub Codespaces) — that's where cargo, step-cli's dependencies (Hasura/Keycloak), and the trustee1/trustee2 containers are reachable.

Every command below writes its output under packages/step-cli/scripts/telephone-load-test-output/ (gitignored) rather than /tmp, so a run survives a container restart and stays easy to find between steps.

Configuration​

setup_telephone_load_test.py and run_telephone_load_test.py take no command-line arguments — every setting lives in packages/step-cli/scripts/telephone-load-test-inputs/config/layers.yaml, under the setup: / telephone_run: sections respectively. Online browser and k6 workloads use the voting load CLI. config/ is gitignored (it holds real per-server credentials); copy the tracked layers.yaml.example template there first:

mkdir -p packages/step-cli/scripts/telephone-load-test-inputs/config
cp packages/step-cli/scripts/telephone-load-test-inputs/layers.yaml.example \
  packages/step-cli/scripts/telephone-load-test-inputs/config/layers.yaml

Edit that copy before each run instead of passing flags. A field left as null falls back to the environment variable named in the comment beside it (already exported in this repo's devcontainer).

Prerequisites​

  • The keycloak, graphql-engine (Hasura), trustee1 and trustee2 containers must be running (docker ps; docker start trustee1 trustee2 if they aren't — the keys ceremony hangs without them).
  • Python 3 with PyYAML (already available in the devenv shell).

0. Build the CLIs​

Both scripts look for their binary on PATH first, falling back to the default release build path — so a release build is enough, no need to keep using cargo run:

CARGO_TARGET_DIR=/workspaces/step/packages/step-cli/rust-local-target \
cargo build \
  --manifest-path packages/step-cli/Cargo.toml \
  --release \
  --package step-cli
CARGO_TARGET_DIR=/workspaces/step/beyond/packages/rust-local-target \
cargo build \
  --manifest-path beyond/packages/Cargo.toml \
  --release \
  --package ivr-cli
export PATH="/workspaces/step/packages/step-cli/rust-local-target/release:$PATH"

1. Stage 1 — provision the election event and voters​

layers.yaml's setup: section already points at the tracked example election event with voting_channel: TELEPHONE and 20 voters — run it as-is:

python3 packages/step-cli/scripts/setup_telephone_load_test.py

This imports the tracked example election event, generates 20 voters (all placed in the same area, with unique numeric username/PIN/date-of-birth each), runs the keys ceremony, publishes, and opens TELEPHONE voting. Outputs land in setup.out_dir (telephone-load-test-output/run by default): a top-level tenants.json index, and one tenant-<tenant_id>/ subdirectory per provisioned tenant holding that tenant's own summary.json and voters CSV (with a single tenant, that's just one subdirectory).

By default Stage 1 provisions setup.tenant_id itself. Set setup.new_tenants: N to instead create N brand-new tenants and import the same election event into each — Stage 2 then places calls across every tenant tenants.json lists. Leaving tenant_id unset defaults new_tenants to 1, so omitting it entirely provisions one fresh tenant instead of reusing an existing one (creating a tenant needs setup.keycloak_admin_user/keycloak_admin_password — a Keycloak master-realm admin, distinct from admin_portal_user — to look up each new tenant's freshly generated api-key-client secret).

A brand-new tenant starts blank — no trustees, and Keycloak/roles config from the generic default template rather than tenant_id's own. For each one, the script: exports tenant_id's Keycloak/roles config (export-tenant-config), downloads it and re-uploads it so the new tenant owns its own copy (documents are tenant-owned records — a document can't be imported into a tenant that doesn't own it), imports that copy (import-tenant-config), then copies tenant_id's registered trustees (list-trustees / create-trustee) so the keys ceremony has trustees to work with at all. None of this needs configuring — it's automatic whenever new_tenants > 0.

Set setup.use_existing_tenants to a list of already-existing tenant IDs to also provision this election event into, alongside tenant_id itself / any new_tenants brand-new tenants — typically tenants a previous run's new_tenants created (copy them straight from that run's tenants.json). Unlike new_tenants, these are never created or cloned — the script authenticates into each directly and provisions the election event, so every one must already have its own registered trustees and Keycloak/roles config in place (true for any tenant new_tenants itself created previously). Looking up each one's api-key-client secret needs keycloak_admin_user/keycloak_admin_password, same as new_tenants > 0.

The keys ceremony defaults to setup.ceremony_policy: AUTOMATIC: each trustee's braid service still does its DKG round the same way, but nothing needs to log in as trustee1/trustee2 to confirm it — the ceremony's status flips to done on its own once every trustee's public key is on the board, matching the Admin Portal's "automatic ceremony" option. Set it to MANUAL to instead drive complete-key-ceremony as each configured trustee, as the CLI always did previously.

Each run appends a random 5-character suffix to the election event's alias (e.g. TECUMSEH - DATAFIX Test - K3F9Q) — the admin portal's election event list renders alias, not name, so that's the field that needs the suffix to actually be visible there. The full alias is printed at the end of the run and recorded as election_event_alias in summary.json.

Using your own election event JSON instead of the tracked example works the same way — just point setup.election_event_json at it in layers.yaml. If it has more than one area, set setup.voter_area_name to pick which one voters are generated into (null defaults to the first area).

Importing the election event also creates its own Keycloak realm (tenant-<tenant_id>-event-<election_event_id>, printed as keycloak_realm in summary.json), seeded with its own ivr-service/ivr-voting clients — these are not in the tenant's realm alongside api-key-client. Since every Stage 1 run creates a brand-new election event (and therefore a new realm), fetch telephone_run.keycloak_ivr_service_client_secret/ keycloak_ivr_voting_client_secret from this realm after each run — Keycloak admin console → the realm printed above → Clients → ivr-service/ivr-voting → Credentials tab — rather than reusing a value from a previous run's election event.

Match the IVR authentication and ballot​

Telephone credentials must be numeric and no longer than eight digits. The setup starts usernames at 100 (setup.voter_username_start); use a RandomNumeric password policy with at most eight digits. The realm's /ivr-config endpoint defines the actual login fields, which may be voter ID and PIN or date of birth and PIN.

The setup opens the TELEPHONE voting channel explicitly. Opening only ONLINE does not make a voter eligible for a telephone call. All generated voters use setup.voter_area_name (the first area by default), because one DTMF ballot template must match the contests offered to every voter.

2. Get a DTMF template​

packages/step-cli/scripts/dtmf-template.example.txt — the default for telephone_run.dtmf_template — is already captured for the tracked example election's first area and works as-is — skip to step 3.

If you're testing a different election event JSON or setup.voter_area_name, capture a new template: the ballot portion (candidate numbering, confirm/submit keys) depends on that election's contests. First get a phone_config.json and a running session store by running Stage 2 once with any existing template (the calls themselves may fail if the template doesn't match your election — that's fine, the side effects are what you need):

python3 packages/step-cli/scripts/run_telephone_load_test.py

Then drive one call by hand, noting every prompt and keystroke:

PHONE_CONFIG_PATH=packages/step-cli/scripts/telephone-load-test-output/calls/phone_config.json \
beyond/packages/rust-local-target/release/ivr-cli \
  --bundle dev \
  --system-number +111111111111 \
  --number +15550000000 \
  --show-internal-state

Log in with the first row of the voters CSV. Transcribe the full keystroke sequence into a copy of dtmf-template.example.txt (and point telephone_run.dtmf_template at your copy), replacing the identifier/PIN lines with {{VOTER_ID}}/{{PIN}} or {{DOB}}/{{PIN}} — check which fields this realm's IVR flow expects first:

read -r -p 'Election realm from summary.json: ' IVR_REALM
read -rs -p 'ivr-service client secret: ' IVR_CLIENT_SECRET
TOKEN=$(curl \
  --fail \
  --silent \
  --show-error \
  --request POST \
  --data-urlencode grant_type=client_credentials \
  --data-urlencode client_id=ivr-service \
  --data-urlencode "client_secret=$IVR_CLIENT_SECRET" \
  "http://keycloak:8090/realms/$IVR_REALM/protocol/openid-connect/token" \
  | python3 -c 'import json,sys; print(json.load(sys.stdin)["access_token"])')
curl \
  --fail \
  --silent \
  --show-error \
  --header "Authorization: Bearer $TOKEN" \
  "http://keycloak:8090/realms/$IVR_REALM/ivr-config"
unset TOKEN IVR_CLIENT_SECRET

(IVR_REALM is summary.json's keycloak_realm — the election event's own realm, not the tenant's; see the note in step 1. The client secret goes in telephone_run.keycloak_ivr_service_client_secret in layers.yaml — or .devcontainer/.env.development for the local devcontainer stack, which always targets the same seeded test election event.)

3. Stage 2 — fan out the simulated calls​

python3 packages/step-cli/scripts/run_telephone_load_test.py

This generates phone_config.json, renders one DTMF input file per voter from the template, and fans out telephone_run.concurrency parallel ivr-cli calls. A local valkey container is started automatically as the session store if none is reachable (reused across runs; disable with telephone_run.start_valkey: false). Results land in results.csv and per-call logs under telephone_run.out_dir (telephone-load-test-output/calls by default).

4. Stage 3 — clean up: delete the election event(s) and tenant(s)​

cleanup_telephone_load_test.py automates this stage: it reads Stage 1's tenants.json (every tenant it provisioned into — one, unless setup.new_tenants/setup.use_existing_tenants was set) and each tenant's summary.json, then for each (tenant_id, election_event_id) pair re-authenticates step-cli against that tenant (a session is scoped to one tenant at a time) and calls delete-election-event. Once every election event is gone, it also deletes every non-bootstrap tenant (never the bootstrap tenant itself — see below):

python3 packages/step-cli/scripts/cleanup_telephone_load_test.py

Unlike the other load-test scripts, this one takes command-line flags — they scope how destructive a run is, which is a per-invocation choice, not something that belongs in layers.yaml:

  • --events-only — delete election events only; leave every tenant realm in place, including ones setup.new_tenants created this run.
  • --new-tenants-only — delete election events as usual, but only delete tenants tenants.json marks "source": "new" — tenants Stage 1 reused via setup.use_existing_tenants ("source": "existing"), and any entry with no source at all (e.g. a hand-written tenants.json), are left in place. This is the field setup_telephone_load_test.py writes per tenant to record whether it created that tenant this run or just provisioned an election event into an existing one — see the note on setup.use_existing_tenants above.

With neither flag: delete every election event and every non-bootstrap tenant, as it's always done.

It reads the same setup: section of layers.yaml Stage 1 used. For the bootstrap tenant (setup.tenant_id) it reuses the already-known setup.keycloak_client_secret; for every other tenant, it looks up that tenant's own api-key-client secret via setup.keycloak_admin_user/keycloak_admin_password first — the same Keycloak master-realm admin lookup setup_telephone_load_test.py did, whether that tenant was created fresh or reused via setup.use_existing_tenants.

delete-election-event calls the delete_election_event GraphQL mutation, which queues an async task tearing down the election event's Postgres/Hasura rows, its Keycloak realm, and its ImmuDB and document-store data — the command blocks and polls until that task finishes (or fails/times out after 5 minutes), so a Success! means cleanup is actually done, not just queued.

To delete an event by hand, authenticate the CLI against its tenant, then run:

read -r -p 'Synthetic election event ID from summary.json: ' ELECTION_EVENT_ID
step-cli step delete-election-event \
  --election-event-id "$ELECTION_EVENT_ID"

delete-tenant deletes a tenant outright — its Postgres/Hasura rows (trustees, templates, election types, any tenant-level documents), its Keycloak realm, and its remaining S3 documents — via the delete_tenant GraphQL mutation, polled the same way as delete-election-event. It refuses to run while the tenant still has any election events, and it's a super-admin-only action: the caller must be authenticated as the bootstrap tenant (setup.tenant_id), not as the tenant being deleted — matching how create-tenant itself is authorized.

The tenant-delete role only needs to exist in the bootstrap tenant's own Keycloak realm (never in the tenants being deleted), and is already assigned to the admin/admin-light roles there in the dev container's default realm import. Deployed environments provisioned via beyond's client-setup chart don't seed this role yet (kept out deliberately, to avoid changing that chart's realm export for every deployment), so before running cleanup against such an environment, add the tenant-delete role to the bootstrap tenant's realm by hand — Keycloak admin console → bootstrap tenant realm → Realm roles → create tenant-delete, then assign it to $ADMIN_PORTAL_USER's role (or the role it inherits it from) — otherwise delete-tenant fails with an authorization error. After authenticating the CLI against the bootstrap tenant, delete a disposable tenant with:

read -r -p 'Disposable tenant ID to delete: ' TENANT_ID
step-cli step delete-tenant \
  --tenant-id "$TENANT_ID"

The bootstrap tenant itself is never deleted, even once its election event is gone — it's the persistent identity these scripts authenticate as, meant to be reused across runs, not a disposable one Stage 1 created.

Notes​

  • Each voter casts exactly one vote. Re-running Stage 2 against the same telephone_run.run_dir re-uses the same (already-voted) voters — every call logs in successfully but reports "voting is now complete" without casting a ballot, since there's nothing left for that voter to vote on. This is the system correctly rejecting a duplicate vote, not a failure. To place a fresh batch of calls, re-run Stage 1 to provision a new election event and voter set.
  • Re-running after a dev container restart: the auto-started valkey container is reused if it's already there (even if stopped), so you don't need to remove it manually between runs.
  • IVR client secrets are per-election-event, not per-tenant. Since Stage 1 provisions a new election event realm every run, a keycloak_ivr_service_client_secret/keycloak_ivr_voting_client_secret that worked for a previous run's election event will not work for a new one — re-fetch them from the new realm each time (see the note in step 1). With more than one tenant (setup.new_tenants > 1), the flat telephone_run.keycloak_ivr_service_client_secret/ keycloak_ivr_voting_client_secret can cover at most one of them — set telephone_run.tenant_ivr_secrets (keyed by tenant_id) for the rest.