Three layers: SIP over TLS (port 5061) to protect signalling metadata in transit, SRTP for media encryption, and optionally ZRTP at the endpoint pair for key agreement that bypasses any infrastructure key escrow. Asterisk supports the first two natively. ZRTP requires endpoint support (Jitsi Desktop, Linphone, some Snom and Yealink phones).
What this protects against: passive network observers, carrier-side metadata harvesting (for the leg covered by TLS), opportunistic media interception. What it does not protect against: the PBX operator themselves having access to media if SRTP terminates at the PBX, lawful access compelled at the PBX.