Before Buying an AI Gateway, Ask the Vendor to Revoke a Key

A useful question for your next AI security demo: can we revoke this test credential and try it again?

At aikey, we work on AI access and governance. We recommend that question because it turns a feature claim into an observable result. A dashboard can show that a policy was saved while an execution point is still using an older version.

Use these tests in an agreed, isolated environment with synthetic data and test credentials. Collect measurements from your own setup and compare them with the behavior you agreed to buy.

---

1. Measure what happens after revocation

Make one successful request, revoke the credential or its permission, and reuse it. Record the change time and decisions at each relevant entry point. Confirm that a rejection is caused by revocation, not a timeout or rate limit, and keep sampling for inconsistent enforcement across replicas.

Include the sampling interval. Checking every few seconds will not tell you the exact instant enforcement changed. One test also cannot establish worst-case behavior under every load condition.

Keep new requests, queued jobs, and running tasks separate. Rejecting the next call does not mean an upstream job has stopped. Disabling an account login may not revoke its API credentials either.

For a small team, the practical question is simple: when someone should lose access, which routes actually stop accepting their requests?

---

2. Find out what “budget control” means

If project budgets are in scope, use a small test threshold and controlled traffic. Note the configured threshold, the action triggered, and the spend recorded after outstanding requests finish. Keep the measurement window and pricing basis explicit.

Test another project at the same time. A limit intended for one workload should have the isolation behavior you agreed to buy.

Alerts, request rate limits, token quotas, and spending caps are different features. Usage delays and concurrent requests can produce overshoot. Ask whether an overshoot bound is guaranteed, under which conditions, and how access is restored. An observed amount is not a contractual limit.

Also watch the client after rejection. A retry loop can keep creating load even when requests no longer reach the model provider.

---

3. Check where sensitive content goes

Use synthetic content intended to trigger the agreed rules, plus ordinary text that should pass. Compare the expected decision with the observed decision for each sample.

If the requirement is redaction before external transmission, look for evidence at that boundary. A warning in the dashboard is not sufficient evidence of pre-transmission handling.

Then inspect the logs. Raw content may still be stored there, so review who can access it and why it is retained. Record false positives in the sample set, but do not turn a handful of examples into a general detection-rate claim.

---

4. Test a policy update failure

Where policy is distributed to execution points, test an interruption of that update path. For a managed service, request a vendor-supported test; if none is available, mark the behavior unverified. Does the execution point reject calls, use cached rules, or allow traffic under a documented fallback?

Record the behavior, cache validity, and alert. If cached policy remains active, clarify what happens to newly issued restrictions that cannot reach it.

After recovery, check both the policy version and request behavior. Restoring a network connection is not the same as restoring the intended controls.

There is a tradeoff between continuity and enforcement here. Decide it for the workload before an incident makes the decision urgent.

---

5. Ask someone else to explain one request

Pick an anomalous request from the tests and ask a teammate to reconstruct its authenticated subject, application, target identifier, decision, and outcome from the available records.

Do not assume the credential identifies a person. Shared credentials without trustworthy user context leave an attribution gap. A recorded model name also cannot independently prove the upstream model's identity.

Retries may create several related records. Check whether your teammate can distinguish them without copying raw prompts into another shared document.

Audit access and retention deserve their own checks. More logged content is not automatically better evidence.

---

Keep an honest result sheet

Use a short record for each case: conditions, expected behavior, observed behavior, evidence, and an owner for follow-up. Mark results as passed, failed, unverified, or not applicable.

Separate deployed features from integrations still required and roadmap promises. These five checks cover part of the access and governance layer; they do not certify model behavior or tool execution safety.

The point is to discover operational boundaries while you still have time to change the setup. A narrowly scoped product with clear limits may fit your workload better than a broad feature list you have not tested.

This is also the standard we want aikey to be evaluated against. Our focus is AI access, authorization, governance, and auditing alongside existing identity and gateway infrastructure; actual coverage needs verification for the deployment.

You can learn more about aikey if you are working through these questions. For enterprise deployment conversations, email aikeyfounder@gmail.com for business inquiries.

Which of these checks would expose the biggest unknown in your current setup?