This week, matrix.org identified that their server infrastructure got compromised. Matrix.org is the reference server of the open Matrix protocol used for real-time communication. People reacted differently: some tried to defend matrix.org since it is offering an open and decentralized communication platform, others stated that matrix.org is a security mess, and that they will stay with XMPP or their favorite instant messaging protocol. While matrix.org still recovers from the data breach, it seems to be already clear that this breach was possible due to human error and organizational shortcomings.
In this article, we discuss lessons learned from the matrix.org data breach that are important for everyone.
Always stay in the loop!
Subscribe to our RSS/Atom feeds.
Lesson 1: Purely focusing on technical security causes insecurity
We often see information security blogs that are solely focused on technical aspects of information security: They tell you about their extremely strong TLS transport encryption, their extremely hardened operating system, and about the endless number of HTTP security headers they set. However, information security is also about humans and processes/organization. If you are solely focused on technical aspects, you only look at 33% of all aspects of information security at a max.
Needless to say that humans play a vital part in information security. There are hundreds of human characteristics that can be exploited. Professional social engineers can easily trick people into weakening technical security measures in place. Other humans misunderstand technology and turn off security measures. Then, there are humans who don't know about technical security measures, and don't use them at all. Finally, there is human error.
On the other hand, state-of-the-art technical measures and security-aware people are useless if people don't know what to do in case of incidents, changes, or emergencies. Look at some questions:
- How do you react if you realize that your server is offline?
- How do you react if somebody reports vulnerabilities?
- How do you react if someone illegally accessed your server and changed content?
- How do you react if someone copied the database of all people who get newsletters from you?
- Who is responsible for keeping the server's software up-to-date?
- Did you document the configuration of every application running on your server?
- Do you regularly test if technical security measures are configured in a secure way and work as expected?
- Where do you store backups?
- Do you regularly check all security-related log entries to spot indicators of compromise in advance?
- Do you regularly check if all applications running on your server are still maintained?
- and so on
You hopefully see that information security needs well-defined processes and organization. Processes and organization must also be tested and improved. Everything requires structured documentation.
Besides, even carefully-configured technical security measures aren't “secure” forever. Software contains security vulnerabilities and needs to be patched. Operating systems need to be updated or upgraded. Some applications may break, others may be left without maintenance by developers. Crypto algorithms are weakened by newly-discovered attacks and need to be replaced.
So, don't solely focus on technical aspects of information security. Even 100% technical security only results in less than 33% of overall security.
Lesson 2: The cause of the breach isn't limited to matrix.org at all
Another important lesson is that ANY server can be breached. Some people reacting to the matrix.org breach immediately stated that they will stick with XMPP or their favorite messaging protocol. However, matrix.org was initially breached due to vulnerabilities in an outdated Jenkins application running on a server. This isn't related to the Matrix protocol in any way. Every server runs software and software contains security vulnerabilities. If you don't update your software, you will be likely the next target of an automated attack or bad people. This is true for web servers, for mail servers, for XMPP servers, for database servers etc.
Again, the breach clearly shows the importance of security-aware humans and well-defined processes/organization in information security. You must clearly define processes to keep your software up-to-date, and monitor server-side activity. If you don't do this, your server could be the next target.
Lesson 3: Think twice about using any service on the internet
Let's talk about your and our role as a user on the internet. Every week, another data breach gets disclosed. Then, some people start to tell everybody that they always knew this. Of course, this service got hacked. They told you so. Other people tell you that you only need to use their services or recommendations to be secure. In the meantime, the majority of people simply ignores any breach notification since this IT stuff isn't their daily business and it is all too technical. In the end, all of the mentioned reactions don't help you in any way if you are affected by a data breach.
In reality, most server operators never become aware that their server is already compromised. While every server operator needs to securely implement many different security measures, an attacker only needs to exploit one single vulnerability. It is obvious that there can't be 100% security anywhere. You as a user need to be proactive by implication. Information security is a shared responsibility and can never be assured by only one party. For example, if you use outdated client software or outdated operating systems, your data is also at risk. If you use weak credentials or reuse credentials, your data is also at risk.
Being proactive means that you should be aware of which data you share with third parties. This includes personal data as well as metadata. Using any service on the internet results in loss of control of your data. So, think twice before registering for a new service. Write down which data you actively share with any service. Read InfoSec news on a regular basis. Learn about the basics of information security, and keep in mind that technical aspects cover not more than 33% of information security. Being proactive allows you to quickly get an idea which data could be leaked in case of a data breach.
Lesson 4: Think twice about running your own server on the internet
Let's talk about the responsibilities of you as a server operator/administrator. We sometimes read forum posts written by people who just became an admin. Some of them never touched a Linux terminal before, and try to set up their server by using random shell scripts from the internet. Others blindly implement guides that weren't updated for years and contain configuration examples that are either invalid or insecure now. Then, there are tool-driven admins who only focus on getting good grades by assessment tools, again forgetting the whole human/organizational part of information security. And many of these novice admins believe that they don't get hacked since they are only running such a small blog/website/messaging server.
So, if you run a server, be aware of this. Keep your software up-to-date. Define the above-mentioned processes. Publish contact information for security purposes. Define another channel to inform affected users in case of any incidents. And, most importantly, only process the least amount of personal data possible. Never collect data that isn't necessary. Data that isn't on your server can't be leaked. Furthermore, monitor your log files and look for indicators of compromise.
If you can't ensure regular updates, don't set up and run a server. If you don't monitor your log files, don't set up and run a server. If you don't understand basic concepts of servers, don't set up and run a server. If you don't understand basic concepts of information security, don't set up and run a server. You get the idea. The internet doesn't need just another vulnerable server that joins a botnet soon to attack even more people. The internet doesn't need just another leak of personal data.
Lesson 5: React to any security-related messages
Finally, we would like to point out that there is a bad habit of some server operators: They don't react to security-related messages. We tried to contact dozens of administrators before. Only a small amount of operators replied and thanked for being informed about vulnerabilities in their products/services. Actually, an even smaller amount of operators fixed the vulnerabilities afterwards. Others, like the UltraVNC team, tried to hide vulnerabilities without fixing them which is another really bad habit.
So, at least send a reply if you ever receive security-related messages. Carefully consider the findings. Set up different possibilities to quickly contact you. Set up a website that tells people how to securely contact you. Use upcoming standards like security.txt to provide structured information. If you run a server, you are responsible for its security and for the security of any data processed by it.
Follow us on Mastodon:
The matrix.org team identified their data breach after one month, and seemingly only due to a third party that pointed out the vulnerabilities in Jenkins. The attacker was able to access the production database and likely copied everything, including account data, cleartext chats, and passwords. The attacker also copied secret OpenPGP keys used for signing packages for Debian and Riot/Web (AD0592FE47F0DF61 and E019645248E8F4A1), CloudFlare API keys used for DNS, and probably other secrets. The initial reaction of the matrix.org team didn't result in changing all secret keys. Due to this, the attacker was able to deface the matrix.org website afterwards.
If you are affected by the matrix.org data breach, change your password, and tell other affected users about it. Keep in mind that you always lose control of some of your data if you use the internet—always. If you operate a server, implement not only technical measures but also processes/organization and regularly inform yourself and your team about social engineering attempts. Finally, share your knowledge and experience with others for free instead of complaining about “irresponsible” companies, teams, and people. The internet doesn't need the next list of “5 tips for more security” or “everybody can run a server” statements.