This week, matrix.org identified that their server infrastructure got compromised. Matrix.org is the reference server of the open Matrix protocol used for real-time communication. People reacted differently: some tried to defend matrix.org since it offers an open and decentralized communication platform; others stated that matrix.org is a security mess and that they will stay with XMPP or their favorite instant messaging protocol. While matrix.org still recovers from the data breach, it seems clear that this breach was possible due to human error and organizational shortcomings.
In this article, we discuss lessons learned from the matrix.org data breach that are relevant for everyone.
Always stay in the loop!
Subscribe to our RSS/Atom feed.
Lesson 1: Purely focusing on technical security causes insecurity
Many security blogs solely focus on technical aspects of information security: They tell you about their powerful TLS transport encryption, their extremely hardened operating system, and about the endless number of HTTP security headers they set. However, information security is also about humans and processes/organization. If you only focus on technical aspects, you only look at 33% of all aspects of information security at a max.
Needless to say that humans play a vital part in information security. Attackers can potentially exploit hundreds of human characteristics. Professional social engineers can easily trick people into weakening technical security measures in place. Other humans misunderstand technology and turn off security measures. Then, some humans don’t know about technical security measures and don’t use them. Finally, there is human error.
On the other hand, state-of-the-art technical measures and security-aware people are useless if people don’t know what to do in case of incidents, changes, or emergencies. Look at some questions:
- How do you react if you realize that your server is offline?
- How do you respond if somebody reports vulnerabilities?
- How do you react if someone illegally accessed your server and changed content?
- How do you respond if someone copied the database of all people who get newsletters from you?
- Who is responsible for keeping the server’s software up-to-date?
- Did you document the configuration of every application running on your server?
- Do you regularly test if technical security measures are configured securely and work as expected?
- Where do you store backups?
- Do you regularly check all security-related log entries to spot indicators of compromise in advance?
- Do you regularly check if all applications running on your server are still maintained?
- and so on
You hopefully see that information security needs well-defined processes and organization. Processes and organization must also be tested and improved. Everything requires structured documentation.
Besides, even carefully-configured technical security measures aren’t “secure” forever. Over time, people might find security vulnerabilities in your software. You need to evaluate the vulnerabilities, patch them, or implement workarounds. Operating systems need to be updated or upgraded. Some applications may break, others may be left without maintenance by developers. Newly-discovered attacks weaken crypto algorithms, and you must replace the algorithms.
So, don’t solely focus on the technical aspects of information security. Even 100% of technical security only results in less than 33% of overall security.
Lesson 2: The cause of the breach isn’t limited to matrix.org at all
Another valuable lesson is that attackers can breach any server. Some people reacting to the matrix.org breach immediately stated that they would stick with XMPP or their favorite messaging protocol. However, matrix.org was initially breached due to vulnerabilities in an outdated Jenkins application running on a server. Jenkins isn’t related to the Matrix protocol in any way. Every server runs software, and software contains security vulnerabilities. If you don’t update your software, you will likely be the next target of an automated attack. Outdated software can affect any server type: Web servers, mail servers, XMPP servers, database servers, or proxy servers.
The breach emphasizes the significance of security-aware humans and well-defined processes and organization in information security. You must clearly define processes to keep your software up-to-date and monitor server-side activity. If you don’t do this, your server could be the next victim.
Lesson 3: Think twice about using any service on the internet
Let’s talk about your and our role as a user on the internet. Every week, another data breach gets disclosed. Then, some people start to tell everybody that they always knew this. Of course, this service got hacked. They told you so. Other people tell you that you only need to use their services or recommendations to be secure. In the meantime, most people ignore every breach notification since this IT stuff isn’t their daily business, and it is all too technical. Ultimately, these reactions don’t help you in any way if you are affected by a data breach.
In reality, most server operators never become aware that their server is already compromised. While every server operator needs to securely implement various security measures, an attacker only needs to exploit one vulnerability. There can’t be 100% security anywhere. You, as a user, need to be proactive by implication. Information security is a shared responsibility, and no party can assure it completely. For example, if you use outdated client software or outdated operating systems, your data is also at risk. If you use weak credentials or reuse credentials, your data is also at risk.
Being proactive means that you should be aware of which data you share with third parties. This data includes personal data as well as metadata. Using any service on the internet results in loss of control of your data. So, think twice before registering for a new service. Write down which data you actively share with any service. Read InfoSec news regularly. Learn about the basics of information security, and keep in mind that technical aspects cover not more than 33% of information security. Being proactive allows you to quickly get an idea of which data could be leaked in case of a data breach.
Lesson 4: Think twice about running your own server on the internet
Let’s talk about the responsibilities of you as a server operator/administrator. We sometimes read forum posts written by people who just became an admin. Some of them never touched a Linux terminal before and tried to set up their server using random shell scripts from the internet. Others blindly implement guides that weren’t updated for years and contain configuration examples that are either invalid or insecure now. Then, some tool-driven admins only focus on getting good grades by assessment tools, again forgetting the whole human/organizational part of information security. And many of these novice admins believe that they don’t get hacked since they are only running such a small blog/website/messaging server.
So, if you run a server, be aware of this. Keep your software up-to-date. Define the processes mentioned above. Publish your contact information for security purposes. Define another channel to inform affected users in case of any incidents. And, most importantly, only process the least amount of personal data possible. Monitor your log files and look for indicators of compromise.
Never collect data that isn’t necessary. Data that isn’t on your server can’t be leaked.
Don’t set up and run a server in the following cases:
- You don’t understand the basic concepts of its operating system.
- You don’t understand the basics of information security.
- You don’t monitor log files and system processes.
- You can’t ensure regular security updates.
The internet doesn’t need another vulnerable server joining a botnet soon to attack even more people. The internet doesn’t need just another leak of personal data.
Lesson 5: React to any security-related messages
Finally, we would like to point out that some server operators have a dangerous habit: They don’t react to security-related messages. We tried to contact dozens of administrators before. Only a small number of operators replied and thanked us for the information about vulnerabilities in their products or services. An even smaller number of operators fixed the vulnerabilities afterward. Others, like the UltraVNC team, tried to hide vulnerabilities without fixing them, which is another unfortunate habit.
So, at least reply if you ever receive security-related messages. Carefully consider the findings. Set up various possibilities to contact you quickly. Set up a website that tells people how to reach you securely. Upload a security.txt file providing structured information. If you run a server, you are responsible for its security and any data processed by it.
The matrix.org team identified their data breach after one month, and seemingly only due to a third party that pointed out the vulnerabilities in Jenkins. The attacker could access the production database and likely copied everything, including account data, cleartext chats, and passwords. The attacker also copied secret OpenPGP keys used for signing packages for Debian and Riot/Web (AD0592FE47F0DF61 and E019645248E8F4A1), Cloudflare API keys used for DNS, and probably other secrets. The initial reaction of the matrix.org team didn’t result in changing all secret keys. Due to this, the attacker was able to deface the matrix.org website afterward.
If you are affected by the matrix.org data breach, change your password, and tell other affected users about it. Keep in mind that you always lose control of some of your data if you use the internet.
If you operate a server, implement technical measures, define processes and organization, plus regularly inform yourself and your team about social engineering attempts. Finally, share your knowledge and experience with others for free instead of complaining about “irresponsible” companies, groups, and people. The internet doesn’t need the next list of “5 tips for more security” or “everybody can run a server” statements.