Web server security – Part 8: Basic log file analysis

Web server security – Part 8: Basic log file analysis

In this part of the Web server security series, we show basic log file analysis using lnav (“The Log File Navigator”).

Contents

  1. Basics
  2. Requirements
  3. lnav – The Log File Navigator
  4. How to review log files
  5. Summary
  6. Links

Always stay in the loop!
Subscribe to our RSS/Atom feeds.

Basics

Many services running on your server (and client) produce log files. A well-known log file created by most web servers is “access.log” that lists every request of clients, and additional information. We will look at this file in a few moments. Additionally, most web servers produce “error.log” files that list errors.

Other log files are produced by tools like AIDE, ClamAV, ufw, your packaging tool, rkhunter, and your operating system. A typical location for log files is the folder “/var/log/” on Linux-based systems. However,—and this is important—there are many different log formats. Several well-known log formats are the Common Log Format (CLF), the Extended Log Format (ELF), and syslog. (Actually, there are different syslog standards.)

Apache 2.4 uses CLF for its access.log file but log formats can be easily customized.

Requirements

The following guide is heavily based on lnav 0.8.5 (“The Log File Navigator”). However, most processes and technologies are independent from this tool. So you can implement the same ideas using your favorite log file analyzer.

We assume that you downloaded log files from your server to analyze them on your local client. Alternatively, you can install lnav directly on your server and access it via SSH.

So we need:

  • some log files retrieved from your server (we will use a fake “access.log” file)
  • lnav (“The Log File Navigator”) installed on your client
  • basic knowledge about the structure of log files
  • optionally: basic knowledge about SQL queries

The Log File Navigator is an open-source, advanced log file viewer for the command line (via ncurses). It is available for Linux and macOS. The big advantage for private users is that there is no need to install it on the server. You can install and run it on your local machine. lnav supports well-known log formats like CLF, syslog, sudo, and more.

You can not only look at log files but also directly search for information using SQL queries. lnav uses SQLite virtual tables. So there is no real SQL database, but lnav allows you to use SQL statements in some kind of virtual environment. We will show SQL queries in a bit. Moreover, lnav supports syntax highlighting. For instance, errors (HTTP status code 4xx) are marked red by default.

Step 1: Load an access.log file

Loading a log file is easy. Just enter lnav [log-file].log. This opens your log file in your terminal. At the same time, lnav automatically creates an SQLite virtual table, so we can use SQL queries later. lnav also understands log files produced by logrotate with “….log.1” or “….log.gz” endings. There is no need to convert such files. Directly open them.

The following screenshot shows the basic layout of lnav.

The basic layout of lnav 0.8.5 after loading an access.log file. The line containing the HTTP status code 404 is automatically marked red.
The basic layout of lnav 0.8.5 after loading an access.log file. The line containing the HTTP status code 404 is automatically marked red. (🔍 Zoom in)

Moreover, lnav supports loading all log files of a folder into a single view. Just enter lnav [folder-that-contains-log-files]. Lines are automatically combined and sorted by date and time. Keep in mind that this requires synchronized and correct timestamps.

Step 2: Try out some commands

We introduce some basic commands to use lnav. A good starting point is ?. This command shows a comprehensive help page. You can return to the standard view using q. You close lnav if you press q again.

Navigation

For navigation, you can use your mouse wheel or your keyboard. Use the arrow keys (PAGE DOWN, PAGE UP) to navigate line by line, or g to move to the top of the file or G to move to the bottom of the file.

Furthermore, you can directly jump to errors using e (HTTP status code 4xx or 5xx). Moreover, you can use 0 or SHIFT + 0 to jump to the next/previous day.

Bookmarks

Bookmarking can be important if you investigate requests. To bookmark the current line, press m. To jump from bookmark to bookmark, use u or U.

C deletes all bookmarks.

Views

There are different views available. One of them is shown after pressing T. This adds a new column that shows the time elapsed since the first line of the file. Another view is a histogram (i).

Use q to switch to the “normal” view again.

Step 3: Use SQL queries

A powerful feature of lnav is support for SQL queries. Some examples are shown if you press ? (“SQL QUERIES (experimental)”).

In our case, the name of the table is “access_log”. This is shown at the top of the page (see screenshot above: “Current SQL table”). To enter an SQL query, press ; and enter it. lnav shows errors in your query in real-time.

It also shows accessible columns that you can use in your query. For instance, lnav recognizes the following fields in our access.log file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
Known message fields for table access_log:
├ c_ip             = 220.79.251.56
├ cs_username      = -
├ timestamp        = 14/Aug/2019:11:37:35 +0200
├ cs_method        = GET
├ cs_uri_stem      = /amet.csv
├ cs_uri_query     = null
├ cs_version       = HTTP/2.0
├ sc_status        = 200
├ sc_bytes         = 4957
├ cs_referer       = -
├ cs_user_agent    = Opera/8.43.(X11; Linux i686; cs-CZ) Presto/2.9.189 Version/12.00
├ body

If you know SQL, just enter queries. One example is the following query:

SELECT c_ip, sc_status, cs_uri_stem
FROM access_log
WHERE sc_status LIKE "4%"

The output shows the IP address, HTTP status code, and requested file of all requests that resulted in a 4xx error. You can use v to switch between the SQL result page and the “normal” view.

You can also list all different user agents:

SELECT DISTINCT cs_user_agent
FROM access_log

Or you can look at all requests that didn’t use HTTP GET:

SELECT c_ip, cs_method, cs_uri_stem
FROM access_log
WHERE cs_method<>"GET"

This may be interesting if you disallowed everything but GET.

There are endless possibilities of querying. Since only you know your configuration, you have to define what should be logged and what you want to know. For instance, you provide a blog using static files (HTML etc.) only. So there is no need to request dynamic files like PHP. In this case, you can look for requests for PHP files (… cs_uri_stem LIKE "%php%"). This may discover bots and attackers.

How to review log files

Log files can become really valuable in case of security incidents. They can help you detecting ongoing or previously-undiscovered security incidents. However, you must know how to review them. In the following, we provide a short guide for private users.

  1. Identify log sources: In the best case, you already wrote down the location of all log files of your server (see part 0 of this series). In case of security incidents, you must sort out irrelevant files.
  2. Copy relevant log files to a single location.
  3. Load all copied log files (e.g., use lnav), and check if their timestamps are synchronized and correct.
  4. Use tools like lnav to look for errors (e), warnings (w), or other strange requests (e.g., requests for “wp-admin.php”). Look for keywords like “session”/“user”, “password”/“authentication”, “denied”/“granted”/“failed”/“failure”, or “sudo”/“su”. Other common attack patterns are excessive attempts to access non-existing files, code in the URL, or requests that resulted in HTTP codes 4xx or 5xx.
  5. Write down any suspicious findings and combine them.
  6. Repeat steps 4 and 5 to confirm any theories.

Of course, you can customize this basic process, and use more tools for it. See also part 6 of this series.

This article is part of the "Web server security" series.
Read other articles of this series.

Summary

Tools like lnav (“The Log File Navigator”) allow quicker analysis of log files. Instead of manually searching for attack-like behavior, you can use SQL queries, load and combine multiple files at once, and switch between different views.

However, keep in mind that not only tools but also underlying processes and organization are important. You must know where log files are stored, how they are created and how long information is available. This requires a basic security concept. Understand the structure of your log files, and use customization of logging rules if available.

We may show more advanced tools in upcoming parts of this series that permanently monitor your server and send alerts in case of suspicious events. In part 4, we already showed Fail2ban that automatically blocks IP addresses based on log files.

See also