Usually fine
A plain email-and-password form on a test account. Put the credentials in the prompt or an env var and the agent will fill and submit it like any other form.
Setup guide
Claude Code + Playwright MCP, pointed at your pull request.
A complete walkthrough for wiring an agent-driven browser check into your PR review — what to install, what to feed it, and the prompt that makes it verify rather than summarise.
The loop, end to end
By the end of this you will be able to hand Claude Code a pull request number and get back a browser-verified answer to "does this change actually do what it says". Four inputs make it work, and skipping any one of them is why most attempts return a confident summary of the diff instead of a real check.
- A browser the agent can drive — Playwright MCP.
- The diff, so it knows what changed.
- The stated intent — the PR title and body — so it knows what to hold the change against.
- A running instance to click through, ideally that PR's own preview deployment.
Install Playwright MCP
Once per machine. This gives Claude Code a real browser rather than a fetch tool.
claude mcp add playwright npx @playwright/mcp@latest
# confirm it registered
claude mcp list
# confirm it registered
claude mcp list
Start a session from inside your project directory so the agent has the code alongside the browser. Reading the implementation is half of how it works out what a change was meant to do.
Collect the context the agent needs
The GitHub CLI gives you the diff and the stated intent. The preview URL usually comes from your host's bot comment on the PR.
# what changed
gh pr diff 123 > /tmp/pr-123.diff
# what it claims to do
gh pr view 123 --json title,body,headRefName
# where it is deployed
gh pr view 123 --json comments --jq '.comments[].body' | grep -o 'https://[^ ]*vercel.app'
gh pr diff 123 > /tmp/pr-123.diff
# what it claims to do
gh pr view 123 --json title,body,headRefName
# where it is deployed
gh pr view 123 --json comments --jq '.comments[].body' | grep -o 'https://[^ ]*vercel.app'
The prompt that makes it verify
This is where most setups go wrong. Asked to "test the PR", an agent will read the diff, decide it looks correct, open one page, and agree with itself. You have to force the order of operations: derive the claim first, then go and try to break it.
> Read /tmp/pr-123.diff and the PR description below.
1. Write down, in one sentence, what a user should now be able to do
that they could not before. Do not look at the browser yet.
2. Open <preview-url> and attempt exactly that, as a user would.
3. Report PASS only if you observed it working end to end.
Report FAIL with the step that broke and what you saw instead.
4. If you could not reach the flow at all, say BLOCKED — do not guess.
1. Write down, in one sentence, what a user should now be able to do
that they could not before. Do not look at the browser yet.
2. Open <preview-url> and attempt exactly that, as a user would.
3. Report PASS only if you observed it working end to end.
Report FAIL with the step that broke and what you saw instead.
4. If you could not reach the flow at all, say BLOCKED — do not guess.
Why step 1 matters
Writing the claim down before opening the browser stops the agent from working backwards from what it observes. Without it, whatever the page happens to do becomes the expected behaviour, and the check passes by construction.
The explicit BLOCKED verdict matters just as much. An agent with no way to say "I could not get there" will report a pass on the login screen.
Auth, which is where this gets hard
If your product has a login, this step decides whether the whole setup is useful or decorative. Roughly in order of how well it holds up:
Fragile
Hosted sign-in from Clerk, Supabase, Auth0 or Firebase. The cross-domain redirect and the bot-detection heuristics on those pages make the run inconsistent between attempts.
Generally blocked
Magic links and 2FA codes — the agent has no inbox and no authenticator. Preview protection stops it before your app is even reached.
A common workaround is a bypass token or a seeded session cookie for non-production environments. That is real work to set up and maintain, and it is worth knowing that going in rather than discovering it at step 4.
Running it without a human in the loop
Claude Code runs non-interactively with -p, so you can put the whole thing on a
runner. Install the CLI, provide an API key, and pass the prompt in.
npm install -g @anthropic-ai/claude-code
gh pr diff "$PR" > /tmp/pr.diff
claude -p "$(cat ./verify-prompt.txt)" > result.txt
gh pr diff "$PR" > /tmp/pr.diff
claude -p "$(cat ./verify-prompt.txt)" > result.txt
Check claude --help for the current tool-permission flags before you wire this
into CI — headless runs need tool use pre-authorised, and the exact flag names move between
releases.
Before you gate a merge on this
Decide what a FAIL means when the same commit passes on a rerun. Non-determinism is the normal case here, not an edge case, and a required check that goes red at random gets switched off within a fortnight.
What this setup will not give you
Everything above is worth doing, and for a solo project it may be all you need. Four things stay out of reach, and they are all structural rather than fixable with a better prompt.
- It runs when asked. The pull requests that break production are the ones nobody thought to check.
- The evidence disappears when the session closes. There is no recording to link in review.
- Context fills up. Page snapshots are verbose, and long flows crowd out the code the agent was reasoning about.
- Nothing gates the merge. A terminal verdict is not a status check.
The full comparison goes through those in detail, including the cases where the DIY loop is genuinely the better choice. If you would rather this just happened on every pull request, with video and a verdict on the PR itself, that is what Kery for Claude Code does.
FAQ
Setup questions.
How do I set up Playwright MCP with Claude Code?
Add the server once with `claude mcp add playwright npx @playwright/mcp@latest`, then start a session in your project. Claude Code can navigate, click, fill forms, and read the accessibility tree of any URL you give it, including a pull request's preview deployment.
Can Claude Code review a pull request in a browser?
Yes, if you give it three things: the diff, the preview URL, and a prompt that tells it to verify rather than summarise. The walkthrough on this page covers all three. What it will not do is run unprompted on every pull request.
How do I give Claude Code the diff for a PR?
Pipe it in from the GitHub CLI. `gh pr diff 123` gives the patch and `gh pr view 123 --json title,body` gives the stated intent, which is what tells the agent what the change is supposed to do.
Can I run this in CI instead of locally?
You can run Claude Code headlessly with `claude -p` on a runner that has the CLI and an API key. Budget for it: browser transcripts are token-heavy, runs are non-deterministic, and you will need to decide what a failure even means before you gate a merge on it.
How do I handle login?
This is the step that breaks most setups. You can script a form login in the prompt, but hosted providers, magic links, 2FA, and preview protection are where an ad hoc browser session usually stalls. See the auth section below for what works and what does not.
When should I stop doing this manually?
When you want the check to happen whether or not somebody remembers to ask, and you want a durable artifact on the pull request afterwards. That is the point where a purpose-built runner earns its place over a session prompt.
Put proof in every pull request.
Connect a repo and Kery starts checking pull requests against their preview deploys. No test scripts, no CI config.