XMPP: Admin-in-the-middle

Some people promote XMPP-based instant messengers as the “privacy-friendly alternative” to other messengers. In our opinion, you can’t refer to XMPP-based messaging as “privacy-friendly” as long as you don’t control all XMPP servers.

In this article, we show the perspective of an XMPP server administrator. Unsurprisingly, an XMPP administrator (or any other server-side party) can inject arbitrary messages, modify address books, and log passwords in cleartext.

Always stay in the loop!
Subscribe to our RSS/Atom feed.

TL;DR

  • Server-side parties (e.g., administrators, attackers, law enforcement) can transparently modify, log, and monitor nearly everything when communicating via XMPP—independent of end-to-end encryption. “Transparently” means your XMPP client doesn’t learn about these server-side actions; showing no warnings in most cases.
  • Contrary to claims, law enforcement can easily detect and block XMPP traffic. Furthermore, many XMPP servers are physically centralized, hosted by a small number of hosting companies.
  • Federation, decentralization, encryption, and “use Tor” don’t solve these issues as XMPP processes data in cleartext and produces tons of metadata.

What is the problem?

“XMPP servers are federated and decentralized, so your data is private and secure” is a claim we frequently read on the internet.

An image showing a meme about XMPP.
Some people praise XMPP as the ultimate solution when it comes to instant messaging. Found on reddit.com, https://r.nf/pmtucm (🔍 Zoom in)

However, when you install XMPP server software and look at how it processes your data, you quickly realize that server-side parties are in full control. “Full control” means servers-side parties can transparently modify, log, and monitor your data since XMPP manages user accounts and groups on the server. Managing this data happens in cleartext, even if all clients enable end-to-end encryption.

The situation looks worse if you also consider that privacy policies don’t meet basic requirements of the European GDPR and XMPP servers are physically centralized. In 2019 and 2020, we checked more than 1,000 XMPP servers, showing only seven hosting companies hosted 50% of the XMPP servers. On the one hand, people suggest XMPP as the “best way against nation-state espionage”; on the other hand, law enforcement can quickly identify XMPP servers and gets tons of data and metadata in cleartext when looking at the server.

We recommend hosting your own XMPP server instead of trusting unknown parties on the internet. Self-hosting requires much more knowledge and time than just using an arbitrary public XMPP server.

Demonstrating the power of an XMPP admin

We set up the ejabberd Community Server and XMPP clients to demonstrate the power of an XMPP admin. We stayed with the default configuration to make everything as verifiable as possible. Keep in mind even a “hardened configuration” focused on security doesn’t deprive the server admin of their power.

Our server-side setup for the demo

We tested the ejabberd Community Server four times:

  • ejabberd 18.06 on Debian 9 (original test in 2018)
  • ejabberd 19.08 on Ubuntu 18.04 LTS (retest in 2019)
  • ejabberd 20.12 on Ubuntu 20.04 LTS (retests in 2020 and 2021)

We added our domain name, a valid TLS certificate, and the IP address of our server to “ejabberd.yml.” We stayed with the defaults (e.g., TCP port 5222 for client-to-server connections, starttls_required: true; TCP port 5269 for server-to-server connections).

Our client-side setup for the demo

We checked the following XMPP clients over time:

  • Conversations (Android; via F-Droid; OMEMO enabled)
  • Gajim Nightly (Arch; OMEMO enabled)
  • Gajim Stable (Arch)
  • Psi (Windows 11)
  • Psi+ (Windows 11; OMEMO enabled)

We added our own testing accounts to the XMPP clients, and enabled OMEMO if available.

Test 1: Monitor all XMPP connections

Contrary to the claim, “law enforcement can’t detect and block XMPP traffic,” XMPP traffic is detectable. This might be the most unsurprising finding as XMPP is just another communication protocol, no magic. We can simulate the external perspective (e.g., law enforcement) with tcpdump and Wireshark by capturing the network traffic passing through a router:

#! /bin/bash
mkfifo /tmp/pcap_file
ssh [router] "tcpdump -n -w - -i [interface] '(port 5222) or (port 5269)'" > /tmp/pcap_file
wireshark -k -i /tmp/pcap_file

We monitor all client-to-server and server-to-server connections of XMPP as an external party since we look at the network traffic. This trivial first test already shows unsurprising issues:

  • We see the public IP addresses of clients connecting to the XMPP server. While clients can obfuscate their IP addresses by using Tor, the remaining data and metadata stays the same.
  • We see the IP addresses of other XMPP servers communicating with our test server, indicating relations between XMPP servers and clients. Of course, XMPP servers and clients could hide their actual IP address by using Tor; however, user data is still exposed to the server (see the following tests).

Disclosed metadata contains:

  • The IP addresses of senders and receivers of each packet.
  • The approximate locations based on IP addresses.
  • Timestamps of each packet.
  • Bytes transmitted, allowing us to distinguish between XMPP status messages and user-written messages.
  • Number of packets sent and received.

External parties (e.g., state actors, law enforcement, server hosting companies, and internet service providers) can see when clients communicate and how long they communicate. External parties can identify XMPP traffic based on dedicated ports and cleartext XMPP packets, even if all parties use end-to-end encryption between their clients and the servers enforce transport encryption. So this simple test may already reveal huge parts of your social graph in the XMPP world.

Additionally to detecting and blocking XMPP traffic, you likely pick an XMPP server in Germany when you choose a random one. In July 2019, we identified the hosters of more than 1,000 XMPP servers. Only seven companies hosted 50% of the XMPP servers. These companies were located in three different countries, mainly in Germany (again, 50% of the XMPP servers). Read “The German Problem with Tor” on World of Matthew as he describes this issue from another perspective.

Test 2: Use ejabberdctl for information gathering

We switch from the perspective of external parties to the server-side view. This is the view of an administrator and parties with access to ejabberdctl on the XMPP server.

“ejabberdctl” is a powerful administration script for ejabberd. Server-side parties can use it to gather information about all users of the XMPP server. For instance:

  • See all users connected to the server (connected_users_info), including JID, type of connection, IP address, port number, resource priority, session duration.
  • See all devices of a connected user (user_resources).
  • See all contacts of a user (get_roster).
  • See all MUCs (muc_online_rooms).
  • See all MUC options (get_room_options).
  • See all MUC affiliations (get_room_affiliations) like owner, admin, member, outcast, none.
  • See all MUCs where a user is an occupant (get_user_rooms).
  • Read the content of vCards (get_vcard).

Server-side parties can access most private data of users, including their contact lists, group memberships, and other personal data like IP addresses, names, and devices. Accessing the database or monitoring network traffic isn’t necessary.

State-of-the-art messengers either implement client-side account management or enforce end-to-end encryption so that the server always sees encrypted data, rendering this approach useless.

Test 3: Analyze the Mnesia database

Ejabberd uses Mnesia to store all data by default. Server-side parties can enter sudo ejabberdctl dump dump.txt to dump the whole Mnesia database to “dump.txt” in the Mnesia folder.

This Mnesia dump includes:

  • User-specific authentication data for SCRAM-SHA-1 (hashed password, salt, number of iterations).
  • User-specific key material (e.g., for OMEMO PreKeys).
  • PEP (Personal Eventing Protocol) information.
  • Last activity of the user (timestamp).
  • Private storage of users (bookmarks, like saved MUCs).
  • MUC-related data (members, options).
  • Everything listed in the section of test 2.
  • and more

Either the clients or the server need to store this information, of course. In the case of XMPP, the server stores this information in cleartext. State-of-the-art messengers either implement client-side account management or enforce end-to-end encryption so that the server always sees encrypted data as mentioned before.

Test 4: See the content of messages and passwords in cleartext

You can change the log level to DEBUG by configuring loglevel: 5 (or in newer versions loglevel: debug) in your “ejabberd.yml.”

Ejabberd logs everything in debug mode, for example:

  • Full message metadata (message ID, from JID+client to JID, timestamp).
  • Full message content, which may include images, files, location data, and audio files.
  • Full message status (received by a client, displayed (= read by) to a user).
  • Full client status (offline, online, typing).
  • Non-public user actions (user changed avatar, status message, vCard information or created a private MUC).
  • and more

The content of messages

If the clients didn’t configure end-to-end encryption for message content, server-side parties can read everything as shown in the following screenshot:

An image showing metadata of XMPP.
This side-by-side comparison shows the perspective of a user (Gajim, on the right-hand side) and the same message from the server-side perspective (log file, on the left-hand side). (🔍 Zoom in)

Enforcing end-to-end encryption (in the following “OMEMO”) doesn’t change much as you can see in the following screenshot:

An image showing overrated protection by OMEMO.
This side-by-side comparison shows a cleartext message (left side) and an OMEMO-protected message (right side), as seen by the server. You can clearly see that most data remains in cleartext. (🔍 Zoom in)

You can see the message content in the “body” part on the left side:

body = [#text{lang = <<>>, data = <<"This is a cleartext message from lenka to jiri ">>}],

This part doesn’t appear on the right side as enabling OMEMO protects the message content. Instead of seeing the cleartext message, server-side parties only learn that OMEMO is enabled and a message was sent. However, the remaining data is still in cleartext. So end-to-end encryption adds minimal protection against server-side parties.

Our tests show that end-to-end encryption (e.g., OMEMO) adds minimal protection against server-side parties.

Passwords in cleartext

loglevel: 5 (or “debug”) logs every single message and all activities mentioned above plus passwords in cleartext.

Some XMPP proponents insisted that this isn’t true as XMPP servers use SCRAM-SHA-1 for password hashing. However, it doesn’t matter whether you enable or disable server-side password hashing as clients send new/changed passwords to XMPP servers in cleartext. The server logs the cleartext password before hashing it.

ejabberd logs every password in cleartext when somebody registers a new account or changes their password, as shown in the following picture:

An image showing logged passwords in cleartext.
Cleartext passwords in ejabberd.log. Whenever a user registers an account or changes his password, his password is logged (in-band registration is used in this example). (🔍 Zoom in)

A server-side party can monitor all actions of users connected to this server. The party can even monitor users connected to other XMPP servers as long as they are members of MUCs managed on the local server or chat with users on the local server.

Info
Client-side encryption such as OMEMO, OpenPGP, or OTR hides the content of messages. Everything else is still in cleartext, including metadata and passwords. Using Tor may obfuscate your real IP address; everything else is still visible. Using Tor + client-side encryption hides a small subset of the information visible to server-side parties. The vast majority of data remains in cleartext.

Test 5: Manipulate user data

We remained passive so far (tests 1 to 4), meaning we did not modify any user data. However, ejabberdctl allows server-side parties to modify user data:

  • We can inject spoofed messages from arbitrary JIDs (send_message chat), even if these users aren’t on our server.
  • We can add arbitrary JIDs to a user’s contact list (add_rosteritem) or delete JIDs (delete_rosteritem).
  • We can arbitrarily change the vCards of our local users (set_vcard).
  • We can invite users to MUCs (send_direct_invitation).
  • We can transfer the ownership of MUCs to another user (set_room_affiliation) and kick the original owner (muc_unregister_nick).
  • We can change the MUC configuration (change_room_option).
  • We can delete messages older than x days if not delivered (delete_old_messages).
  • and more

Our tests show that XMPP clients process and display spoofed XMPP messages differently. Some clients display the spoofed message without any warnings; others only do this under certain conditions.

The following screenshot shows a spoofed message coming from “snowden[at]nsa.gov.” The Conversations client showed these messages during our original test. Gajim ignored them until the sender JID of the message was in the user’s roster.

An image showing a spoofed message allegedly sent by Edward Snowden.
Conversations shows this spoofed message from snowden@nsa.gov. Gajim shows such messages after an admin added the sender's JID to the user's contact list. (🔍 Zoom in)

Another screenshot shows a spoofed message coming from “user1” to “user2” in Psi+, an XMPP client. Psi+ displays the spoofed message without any warnings. When OMEMO is enabled, Psi+ shows OMEMO as disabled for each spoofed message. Fortunately, OMEMO stays enabled afterwards, so the server-side party can’t read the answer of user2 in this example.

An image showing a spoofed message in Psi+.
Psi+ displays the spoofed message at 16:54:17. (🔍 Zoom in)

Lessons learned and recommendations

XMPP relies on server-side account management, resulting in the issues described above. Using client-side encryption such as OMEMO, OpenPGP, or OTR plus Tor solves a small number of these issues only. For instance, even if you always enforce OMEMO and use Tor, the server-side party can still log your password, access your contact card, see your group memberships, and knows when your client goes online and offline.

While other communication protocols work the same way, we rarely see XMPP proponents openly talking about these issues. In our opinion, one can hardly recommend XMPP as the “most private messenger” when considering the amount of cleartext data exposed on the server.

If you want to self-host an XMPP server, we recommend the following steps:

  • Set up a firewall with a strict ruleset.
  • Regularly update all packages on your server.
  • Install only software that you need on your server. Remove everything else.
  • Harden your SSH configuration: Use modern cryptography, 2FA, key-based authentication, non-root login, and an IP allowlist.
  • Use TLS certificates signed by a public certificate authority (e.g., signed by Let’s Encrypt).
  • Harden your TLS configuration: Try TLS 1.3 only (or set up TLS 1.2 + forward secrecy + AEAD, strong parameters for DHE and ECC).
  • Disable XMPP federation if you don’t need it.
  • Disable unused XMPP mods (XEPs), esp. “in-band registration.”
  • Disable server-side logging if you don’t need it.
  • Enforce client-side OMEMO if they support it.
Warning
Securely hosting your own server isn't done by implementing a "Host your own XMPP server in 5 minutes" guide. You must keep your server secure continuously.
An image showing a complaint about unreachable server admins.
This post in the Fediverse summarizes what we mean by 'hosting your own server isn't done by implementing a "Host your own XMPP server in 5 minutes" guide.' (🔍 Zoom in)

Summary

Our findings show server-side parties can read, manipulate, and delete nearly all users' data. XMPP clients can’t detect most of the server-side actions. Considering many (European) XMPP servers still don’t come with a privacy policy meeting the basic requirements of the European GDPR or many XMPP servers are physically centralized, the overall situation doesn’t look as “privacy-friendly” as claimed by some people.

We encourage tech-savvy people (especially XMPP developers) to further improve XMPP servers and clients to render certain misuse cases impossible (e.g., XMPP clients displaying spoofed XMPP messages without any warnings).

We strongly recommend hosting your own XMPP server in a controlled environment or switching to instant messengers that either implement client-side account management or enforce end-to-end encryption for account management.

We republished this article in November 2021.

Changelog

  • Nov 7, 2021: Rephrased “Test 1” to highlight the disproven claim; added a meme and a screenshot.
  • Nov 6, 2021: Added information about Psi and Psi+; added another screenshot showing cleartext messages vs. OMEMO messages.

Read also