Defending a Proxy VPS Against Active Probing

Censorship systems use two layers. The first is passive detection: deep packet inspection watches every flow and scores it for proxy-like properties, such as a TLS fingerprint that does not match any real browser, a connection that carries far more upload than download, fixed packet sizes, or a long-lived encrypted session to a foreign IP that hosts nothing else. Passive detection alone is error-prone, so it rarely blocks on its own. Instead it produces a suspect list.

The second layer is active probing. A separate prober, often from an IP range unrelated to the censorship infrastructure, connects to the suspected IP and port and behaves like a client of various proxy protocols. It might replay a captured handshake, send a deliberately malformed request, open a TLS connection and then sit idle, or speak the Shadowsocks, VMess or Trojan handshake and watch what comes back. The prober is not trying to use your proxy; it is trying to distinguish it from a real service. If the server reacts in a way only a proxy would, for example by closing the connection in a protocol-specific way, returning a recognizable error, or accepting garbage that a real webserver would reject, the IP and port are confirmed and blocked.

This technique is documented and real. The Great Firewall of China was shown years ago to probe Tor bridges and obfuscated proxies within minutes of first use. Russia's RKN escalated to similar follow-up probing during its 2022 to 2024 throttling of Tor, common VPN protocols and Shadowsocks. The defensive question is therefore not 'can my traffic be flagged' but 'when a prober connects, does my server give a non-website answer'. Everything below is about removing that tell.

Probing strategies cluster into a few recognizable patterns, and understanding them tells you exactly what your server must never do:

• Replay probing. The censor records the first bytes a real client sent to your server, then later replays those exact bytes from a different IP. Early Shadowsocks accepted a replayed handshake and started decrypting, which is a dead giveaway because a webserver would simply reject unknown input. Modern transports defeat this with strict replay protection.
• Protocol-guess probing. The prober speaks each known proxy protocol in turn (Shadowsocks AEAD, VMess, Trojan, plain SOCKS) and looks for a server that completes any of them. A real HTTPS server never completes a Shadowsocks handshake.
• Malformed-input probing. The prober sends random bytes, a truncated TLS ClientHello, or an HTTP request with a bad method. A real webserver responds with a clean TLS alert or an HTTP 400 page; many naive proxies hang, reset oddly, or echo a distinctive error.
• TLS-anomaly probing. The prober opens a real TLS connection and inspects the certificate, supported versions and ALPN. A self-signed cert, an oddly fresh Let's Encrypt cert on a residential-looking IP, missing HTTP/2, or a certificate whose hostname does not match the SNI all stand out.
• Timing and behavior probing. The prober connects and stays silent, or sends a valid TLS handshake then no data. A proxy that times out differently from a webserver, or that only ever responds when fed valid proxy traffic, reveals itself by what it does not do.

The unifying rule the censor relies on is simple: a real webserver responds consistently and harmlessly to anything thrown at it, including garbage, while a proxy only behaves correctly when spoken to in its own protocol. Defeating active probing means your server must pass that test for every possible probe, not just the happy path.

Reality (implemented in Xray-core and sing-box) is currently the strongest answer to active probing because it removes the artifact a prober looks for entirely. When an unauthenticated connection arrives on port 443, your server does not present its own certificate or run its own TLS termination. Instead it transparently forwards the TLS handshake to a real, unrelated website (the dest) and relays that site's genuine certificate chain back to the connecting party.

Authenticated clients are recognized cryptographically: the server holds an X25519 private key, and only a client carrying the matching public key plus a valid short ID completes a real tunneled session. Every other connection, including the censor's probe, is handed off to the borrowed site and sees precisely what it would see if it had connected to that site directly. There is no self-signed certificate, no fresh ACME cert, no domain registered in your name, and no protocol-specific reply. A replay probe, a malformed-input probe and a TLS-anomaly probe all receive the same answer a real website gives.

The configuration discipline is where Reality succeeds or fails against probing. The borrowed dest must support TLS 1.3 and HTTP/2, serve real content directly without redirecting, be foreign and unlikely to be blocked in your users' country, and ideally be hosted near your VPS region so the apparent origin is geographically consistent with your server's IP. The serverNames your clients send must be exact hostnames that the dest serves a valid certificate for, because a dest/SNI mismatch produces a certificate inconsistency a prober can catch. Use the XTLS Vision flow (flow: "xtls-rprx-vision") so the inner tunneled TLS session does not leave a recognizable 'TLS in TLS' packet-shape pattern. A correctly configured Reality endpoint on a clean offshore IP gives a prober nothing static to match.

Reality is not the only design built around the same principle. Two related approaches give the same anti-probing property and are worth knowing because they fit different stacks.

• ShadowTLS (v3). ShadowTLS puts a real TLS handshake to a genuine external site in front of your actual proxy (commonly Shadowsocks-2022). To a prober, the server completes a real TLS 1.3 handshake with the certificate of, say, a major foreign vendor site, because the handshake is genuinely proxied to that site. Only a client that knows the pre-shared key can switch the established connection over to the inner proxy after the handshake; an unauthenticated prober just sees a real TLS session to a real site and gets nothing proxy-shaped. ShadowTLS v3 also adds protections against the relay and replay tricks that broke v1. It pairs well if you already run Shadowsocks-2022 and want to wrap it.
• Fallback-to-real-site (Trojan and VLESS+TLS). If you do run your own domain and certificate, the classic anti-probing measure is a fallback: any connection whose payload is not valid proxy traffic (wrong password, a bare HTTP request, garbage, or a probe) is silently handed to a real backend web service, typically a local nginx serving an actual website, or reverse-proxied to a genuine site. The probe receives a normal web page or a normal TLS error, never a proxy error. In Xray this is the fallbacks block; in Trojan-Go it is the fallback_addr. The key is that the fallback must serve something believable: a real, non-empty site, with a certificate whose hostname matches, not a default 'Welcome to nginx' page or an empty 444.

All three designs, Reality, ShadowTLS and TLS fallback, implement one rule: an unauthenticated visitor must receive exactly what a real website would return. They differ only in whether you borrow a remote site (Reality, ShadowTLS) or host your own believable site as the fallback (Trojan, VLESS+TLS). Choose based on whether you want zero domain footprint (Reality) or already have a domain and cert you are comfortable using (fallback).

A perfect transport is undone by a sloppy server. A prober does not only hit your proxy port; it scans the whole IP. The defensive goal is that the only thing any external scan ever sees is one clean, real-looking TLS 1.3 endpoint and nothing that contradicts the cover story.

• One service on 443, nothing else exposed. Never run a self-signed admin panel, a monitoring dashboard, or a control port alongside your transport on a public interface. A fresh self-signed cert on port 8443 next to your Reality endpoint hands the censor exactly the anomaly the dest was meant to hide.
• Move and key-lock SSH. Key-only authentication, a non-standard port, and fail2ban. An open password-auth SSH on 22 is itself a fingerprint and an attack surface.
• Default-deny inbound firewall. Allow only the transport port and your management port. Everything else returns nothing, which is consistent with a single-purpose web host.
• Keep the cover convincing over time. If your Reality or ShadowTLS dest starts redirecting, drops TLS 1.3, or gets blocked locally, your cover is gone; re-verify the dest periodically and keep a spare candidate. If you use a fallback site, make sure it keeps serving real content.
• Use a clean, unflagged IP. Active probing confirms a suspicion that usually starts from the IP's reputation or a shared-range block. A fresh IP with no prior proxy history and full root control removes the two weakest links: a pre-flagged address and a provider that might disclose or reset your server under pressure.

This is where the hosting choice matters as much as the config. Running these transports on a clean offshore IP with full root access lets you control every byte a prober can observe. For a single private endpoint an offshore VPS is the right fit; for many concurrent users or higher throughput an offshore dedicated server gives the headroom and a dedicated, unshared IP. Either way, the principle is the same: leave the prober nothing to find but a normal website.