End-to-End Testing SMS Verification in CI with Playwright and OTP Bolt

Most teams skip phone verification in end-to-end tests because "SMS can't be automated". It can — and once it runs in CI you'll catch broken signup flows the same day they ship rather than when support tickets roll in. This walkthrough adds a real SMS step to a Playwright suite using the OTP Bolt API.

The approach

1. A Playwright fixture rents a temporary number through the API.
2. The test types the number into your signup form and clicks Send code.
3. The fixture polls the activation until the code arrives.
4. The test types the code and asserts the account was created.
5. On teardown, the fixture finishes or cancels the activation so nothing is wasted.

Prerequisites

• An OTP Bolt account with some wallet credit.
• An API key stored as a CI secret, for example OTPBOLT_KEY.
• Your staging app configured to send real SMS (mocks off for this suite).

The fixture

// tests/fixtures/otp.ts
import { test as base } from "@playwright/test";

const API = "https://otpbolt.com/api/v1";
const H = { Authorization: `Bearer ${process.env.OTPBOLT_KEY}`, "Content-Type": "application/json" };

type Rental = { id: string; phone: string; waitForCode: (ms?: number) => Promise<string> };

export const test = base.extend<{ otpNumber: Rental }>({
  otpNumber: async ({}, use, testInfo) => {
    const service = (testInfo.project.metadata?.otpService as string) ?? "telegram";
    const country = (testInfo.project.metadata?.otpCountry as string) ?? "us";
    const res = await fetch(`${API}/activations`, { method: "POST", headers: H, body: JSON.stringify({ service, country }) });
    if (!res.ok) throw new Error(`rent failed: ${res.status} ${await res.text()}`);
    const act = await res.json();
    let done = false;

    const waitForCode = async (ms = 180_000) => {
      const until = Date.now() + ms;
      while (Date.now() < until) {
        const a = await (await fetch(`${API}/activations/${act.id}`, { headers: H })).json();
        if (a.status === "received" && a.code) { done = true; return a.code as string; }
        if (["cancelled", "expired"].includes(a.status)) throw new Error("no SMS — refunded");
        await new Promise(r => setTimeout(r, 4000));
      }
      throw new Error("timed out waiting for SMS");
    };

    await use({ id: act.id, phone: act.phone, waitForCode });

    // teardown: close successful rentals, cancel the rest (cancelled = refunded)
    const path = done ? "finish" : "cancel";
    await fetch(`${API}/activations/${act.id}/${path}`, { method: "POST", headers: H }).catch(() => {});
  },
});

The test

// tests/signup.spec.ts
import { expect } from "@playwright/test";
import { test } from "./fixtures/otp";

test("phone signup end-to-end", async ({ page, otpNumber }) => {
  await page.goto(process.env.STAGING_URL + "/signup");
  await page.getByLabel("Phone number").fill(otpNumber.phone);
  await page.getByRole("button", { name: "Send code" }).click();

  const code = await otpNumber.waitForCode();
  await page.getByLabel("Verification code").fill(code);
  await page.getByRole("button", { name: "Verify" }).click();

  await expect(page.getByText("Welcome")).toBeVisible();
});

Keeping CI fast and cheap

• Run the SMS suite on a schedule or on release branches, not on every commit. A nightly job across your core markets is usually enough.
• Parallelise across countries with Playwright projects — each project sets metadata.otpCountry.
• Fail fast on 409 out-of-stock by trying a second country from a short fallback list.
• Set a hard timeout (three minutes is generous) so a stuck route can't hold a CI runner hostage.
• Use a developer plan if you run more than a few dozen verifications a week; it lowers the per-OTP price and raises your API rate limit.

Making flaky tests deterministic

SMS is a real network, so occasional slowness happens. Treat it explicitly:

• Retry the whole test once on no SMS or timed out — with a new number — rather than retrying individual steps.
• Record delivery time as a test attachment; trends tell you when a route is degrading.
• Keep a separate, clearly labelled job for SMS tests so their occasional flake doesn't block unrelated merges.

Security notes

• Store OTPBOLT_KEY as a masked CI secret and scope it to the jobs that need it.
• Rotate keys from the dashboard if a runner or log ever leaks.
• Never log full SMS bodies in public CI logs if they contain links or tokens.

With this fixture in place, a broken SMS template, a mis-configured provider or a phone-input regression fails your nightly build — long before it fails a customer.