Журнал ошибок nginx

Capture detailed information about errors and request processing in log files, either locally or via syslog.

This article describes how to configure logging of errors and processed requests in NGINX Open Source and NGINX Plus.

Setting Up the Error Log

NGINX writes information about encountered issues of different severity levels to the error log. The error_log directive sets up logging to a particular file, stderr, or syslog and specifies the minimal severity level of messages to log. By default, the error log is located at logs/error.log (the absolute path depends on the operating system and installation), and messages from all severity levels above the one specified are logged.

The configuration below changes the minimal severity level of error messages to log from error to warn:

error_log logs/error.log warn;

In this case, messages of warn, error crit, alert, and emerg levels are logged.

The default setting of the error log works globally. To override it, place the error_log directive in the main (top-level) configuration context. Settings in the main context are always inherited by other configuration levels (http, server, location). The error_log directive can be also specified at the http, stream, server and location levels and overrides the setting inherited from the higher levels. In case of an error, the message is written to only one error log, the one closest to the level where the error has occurred. However, if several error_log directives are specified on the same level, the message are written to all specified logs.

Note: The ability to specify multiple error_log directives on the same configuration level was added in NGINX Open Source version 1.5.2.

Setting Up the Access Log

NGINX writes information about client requests in the access log right after the request is processed. By default, the access log is located at logs/access.log, and the information is written to the log in the predefined combined format. To override the default setting, use the log_format directive to change the format of logged messages, as well as the access_log directive to specify the location of the log and its format. The log format is defined using variables.

The following examples define the log format that extends the predefined combined format with the value indicating the ratio of gzip compression of the response. The format is then applied to a virtual server that enables compression.

http {
    log_format compression '$remote_addr - $remote_user [$time_local] '
                           '"$request" $status $body_bytes_sent '
                           '"$http_referer" "$http_user_agent" "$gzip_ratio"';

    server {
        gzip on;
        access_log /spool/logs/nginx-access.log compression;
        ...
    }
}

Another example of the log format enables tracking different time values between NGINX and an upstream server that may help to diagnose a problem if your website experience slowdowns. You can use the following variables to log the indicated time values:

  • $upstream_connect_time – The time spent on establishing a connection with an upstream server
  • $upstream_header_time – The time between establishing a connection and receiving the first byte of the response header from the upstream server
  • $upstream_response_time – The time between establishing a connection and receiving the last byte of the response body from the upstream server
  • $request_time – The total time spent processing a request

All time values are measured in seconds with millisecond resolution.

http {
    log_format upstream_time '$remote_addr - $remote_user [$time_local] '
                             '"$request" $status $body_bytes_sent '
                             '"$http_referer" "$http_user_agent"'
                             'rt=$request_time uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"';

    server {
        access_log /spool/logs/nginx-access.log upstream_time;
        ...
    }
}

When reading the resulting time values, keep the following in mind:

  • When a request is processed through several servers, the variable contains several values separated by commas
  • When there is an internal redirect from one upstream group to another, the values are separated by semicolons
  • When a request is unable to reach an upstream server or a full header cannot be received, the variable contains 0 (zero)
  • In case of internal error while connecting to an upstream or when a reply is taken from the cache, the variable contains - (hyphen)

Logging can be optimized by enabling the buffer for log messages and the cache of descriptors of frequently used log files whose names contain variables. To enable buffering use the buffer parameter of the access_log directive to specify the size of the buffer. The buffered messages are then written to the log file when the next log message does not fit into the buffer as well as in some other cases.

To enable caching of log file descriptors, use the open_log_file_cache directive.

Similar to the error_log directive, the access_log directive defined on a particular configuration level overrides the settings from the previous levels. When processing of a request is completed, the message is written to the log that is configured on the current level, or inherited from the previous levels. If one level defines multiple access logs, the message is written to all of them.

Enabling Conditional Logging

Conditional logging allows excluding trivial or unimportant log entries from the access log. In NGINX, conditional logging is enabled by the if parameter to the access_log directive.

This example excludes requests with HTTP status codes 2xx (Success) and 3xx (Redirection):

map $status $loggable {
    ~^[23]  0;
    default 1;
}

access_log /path/to/access.log combined if=$loggable;

Usecase: Sampling TLS Parameters

Many clients use TLS versions older than TLS 1.3. Though many ciphers are declared insecure, older implementations still use them; ECC certificates offer greater performance than RSA, but not all clients can accept ECC. Many TLS attacks rely on a “man in the middle” who intercepts the cipher negotiation handshake and forces the client and server to select a less secure cipher. Therefore, it’s important to configure NGINX Plus to not support weak or legacy ciphers, but doing so may exclude legacy clients.

You can evaluate the SSL data obtained from the client and determine what proportion of clients get excluded if support for older SSL protocols and ciphers is removed.

The following configuration example logs the SSL protocol, cipher, and User-Agent header of any connected TLS client, assuming that each client selects the most recent protocol and most secure ciphers it supports.

In this example, each client is identified by its unique combination of IP address and User-Agent.

  1. Define the custom log format sslparams that includes the version of the SSL protocol ($ssl_protocol), ciphers used in the connection ($ssl_cipher), the client IP address ($remote_addr), and the value of standard User Agent HTTP request field ($http_user_agent):

    log_format sslparams '$ssl_protocol $ssl_cipher '
                      '$remote_addr "$http_user_agent"';
    
  2. Define a key-value storage that will keep the IP address of the client and its User Agent, for example, clients:

    keyval_zone zone=clients:80m timeout=3600s;
    
  3. Create a variable, for example, $seen for each unique combination of $remote_addr and User-Agent header:

    keyval $remote_addr:$http_user_agent $seen zone=clients;
    
    server {
        listen 443 ssl;
    
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers   HIGH:!aNULL:!MD5;
    
        if ($seen = "") {
            set $seen  1;
            set $logme 1;
        }
        access_log  /tmp/sslparams.log sslparams if=$logme;
    
        # ...
    }
    
  4. View the log file generated with this configuration:

    TLSv1.2 AES128-SHA 1.1.1.1 "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
    TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 2.2.2.2 "Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1"
    TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 3.3.3.3 "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:58.0) Gecko/20100101 Firefox/58.0"
    TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 4.4.4.4 "Mozilla/5.0 (Android 4.4.2; Tablet; rv:65.0) Gecko/65.0 Firefox/65.0"
    TLSv1 AES128-SHA 5.5.5.5 "Mozilla/5.0 (Android 4.4.2; Tablet; rv:65.0) Gecko/65.0 Firefox/65.0"
    TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 6.6.6.6 "Mozilla/5.0 (Linux; U; Android 5.0.2; en-US; XT1068 Build/LXB22.46-28) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.108 UCBrowser/12.10.2.1164 Mobile Safari/537.36"
    
  5. Process the log file to determine the spread of data:

    cat /tmp/sslparams.log | cut -d ' ' -f 2,2 | sort | uniq -c | sort -rn | perl -ane 'printf "%30s %s\n", $F[1], "="x$F[0];'
    

    In this output, low‑volume, less secure ciphers are identified:

    ECDHE-RSA-AES128-GCM-SHA256 =========================
    ECDHE-RSA-AES256-GCM-SHA384 ========
                     AES128-SHA ====
    ECDHE-RSA-CHACHA20-POLY1305 ==
        ECDHE-RSA-AES256-SHA384 ==
    

    Then you can check the logs to determine which clients are using these ciphers and then make a decision about removing these ciphers from the NGINX Plus configuration.

    For more information about sampling requests with NGINX conditional logging see the blog post.

Logging to Syslog

The syslog utility is a standard for computer message logging and allows collecting log messages from different devices on a single syslog server. In NGINX, logging to syslog is configured with the syslog: prefix in error_log and access_log directives.

Syslog messages can be sent to a server= which can be a domain name, an IP address, or a UNIX-domain socket path. A domain name or IP address can be specified with a port to override the default port, 514. A UNIX-domain socket path can be specified after the unix: prefix:

error_log  syslog:server=unix:/var/log/nginx.sock debug;
access_log syslog:server=[2001:db8::1]:1234,facility=local7,tag=nginx,severity=info;

In the example, NGINX error log messages are written to a UNIX domain socket at the debug logging level, and the access log is written to a syslog server with an IPv6 address and port 1234.

The facility= parameter specifies the type of program that is logging the message. The default value is local7. Other possible values are: auth, authpriv, daemon, cron, ftp, lpr, kern, mail, news, syslog, user, uucp, local0 ... local7.

The tag= parameter applies a custom tag to syslog messages (nginx in our example).

The severity= parameter sets the severity level of syslog messages for access log. Possible values in order of increasing severity are: debug, info, notice, warn, error (default), crit, alert, and emerg. Messages are logged at the specified level and all more severe levels. In our example, the severity level error also enables crit, alert, and emerg levels to be logged.

Live Activity Monitoring

NGINX Plus provides a real-time live activity monitoring interface that shows key load and performance metrics of your HTTP and TCP upstream servers. See the Live Activity Monitoring article for more information.

To learn more about NGINX Plus, please visit the Products page.


Nginx — это высокопроизводительный HTTP- сервер с открытым исходным кодом и обратный прокси-сервер, отвечающий за обработку нагрузки некоторых из крупнейших сайтов в Интернете. При управлении веб-серверами NGINX одной из наиболее частых задач, которые вы будете выполнять, является проверка файлов журналов.

Знание того, как настраивать и читать журналы, очень полезно при устранении неполадок сервера или приложений, поскольку они предоставляют подробную информацию об отладке.

Nginx записывает свои события в журналы двух типов: журналы доступа и журналы ошибок. Журналы доступа записывают информацию о клиентских запросах, а журналы ошибок записывают информацию о проблемах сервера и приложений.

В этой статье рассказывается, как настроить и прочитать журналы доступа и ошибок Nginx.

Настройка журнала доступа

Каждый раз, когда клиентский запрос обрабатывается, Nginx генерирует новое событие в журнале доступа. Каждая запись события содержит отметку времени и включает различную информацию о клиенте и запрошенном ресурсе. Журналы доступа могут показать вам местоположение посетителей, страницу, которую они посещают, сколько времени они проводят на странице и многое другое.

Директива log_format позволяет вам определять формат регистрируемых сообщений. Директива access_log включает и устанавливает расположение файла журнала и используемый формат.

Самый простой синтаксис директивы access_log следующий:

access_log log_file log_format;

Где log_file — это полный путь к файлу журнала, а log_format — формат, используемый файлом журнала.

Журнал доступа можно включить в блоке http , server или location .

По умолчанию журнал доступа глобально включен в директиве http в основном файле конфигурации Nginx.

/etc/nginx/nginx.conf

http {
  ...
  access_log  /var/log/nginx/access.log;
  ...
}

Для удобства чтения рекомендуется создавать отдельный файл журнала доступа для каждого серверного блока. Директива access_log установленная в директиве server access_log директиву, установленную в директиве http (более высокого уровня).

/etc/nginx/conf.d/domain.com.conf

http {
  ...
  access_log  /var/log/nginx/access.log;
  ...

  server {
    server_name domain.com
    access_log  /var/log/nginx/domain.access.log;
    ...
  }
}

Если формат журнала не указан, Nginx использует предопределенный комбинированный формат, который выглядит следующим образом:

log_format combined '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

Чтобы изменить формат ведения журнала, отмените настройку по умолчанию или определите новую. Например, чтобы определить новый формат ведения журнала с именем custom, который расширит комбинированный формат значением, показывающим заголовок X-Forwarded-For добавьте следующее определение в директиву http или server :

log_format  custom  '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';

Чтобы использовать новый формат, укажите его имя после файла журнала, как показано ниже:

access_log  /var/log/nginx/access.log custom;

Хотя журнал доступа предоставляет очень полезную информацию, он занимает дисковое пространство и может повлиять на производительность сервера. Если на вашем сервере мало ресурсов и у вас загруженный веб-сайт, вы можете отключить журнал доступа. Чтобы сделать это, установите значение access_log директиву off :

Настройка журнала ошибок

Nginx записывает сообщения об ошибках приложения и общих ошибках сервера в файл журнала ошибок. Если вы испытываете ошибки в своем веб-приложении, журнал ошибок — это первое место, с которого можно начать поиск и устранение неисправностей.

Директива error_log включает и устанавливает расположение и уровень серьезности журнала ошибок. Он имеет следующую форму и может быть установлен в блоке http , server или location :

error_log log_file log_level

Параметр log_level устанавливает уровень ведения журнала. Ниже перечислены уровни в порядке их серьезности (от низкого до высокого):

  • debugdebug сообщения.
  • info — Информационные сообщения.
  • notice — Уведомления.
  • warn — Предупреждения.
  • error — Ошибки при обработке запроса.
  • crit — Критические проблемы. Требуется быстрое действие.
  • alert — Оповещения. Действия должны быть предприняты немедленно.
  • emerg — Чрезвычайная ситуация. Система находится в непригодном для использования состоянии.

Каждый уровень журнала включает в себя более высокие уровни. Например, если вы установите уровень журнала , чтобы warn , Nginx будет также регистрировать error , crit , alert и emerg сообщения.

Если параметр log_level не указан, по умолчанию используется error .

По умолчанию директива error_log определена в директиве http внутри основного файла nginx.conf:

/etc/nginx/nginx.conf

http {
  ...
  error_log  /var/log/nginx/error.log;
  ...
}

Как и в случае с журналами доступа, рекомендуется создать отдельный файл журнала ошибок для каждого блока сервера, который переопределяет настройку, унаследованную от более высоких уровней.

Например, чтобы настроить журнал ошибок domain.com на warn вы должны использовать:

http {
  ...
  error_log  /var/log/nginx/error.log;
  ...

  server {
    server_name domain.com
    error_log  /var/log/nginx/domain.error.log warn;
    ...
  }
}

Каждый раз, когда вы изменяете файл конфигурации, вам необходимо перезапустить службу Nginx, чтобы изменения вступили в силу.

Расположение файлов журнала

По умолчанию в большинстве дистрибутивов Linux, таких как Ubuntu , CentOS и Debian , журналы доступа и ошибок расположены в каталоге /var/log/nginx .

Чтение и понимание файлов журнала Nginx

Вы можете открывать и анализировать файлы журнала, используя стандартные команды, такие как cat , less , grep , cut , awk и т. Д.

Вот пример записи из файла журнала доступа, в котором используется стандартный формат журнала Nginx для объединения:

192.168.33.1 - - [15/Oct/2019:19:41:46 +0000] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"

Давайте разберемся, что означает каждое поле записи:

  • $remote_addr192.168.33.1 — IP-адрес клиента, выполняющего запрос.
  • $remote_user- — Пользователь, $remote_user аутентификацию по HTTP. Если имя пользователя не задано, в этом поле отображается - .
  • [$time_local][15/Oct/2019:19:41:46 +0000] — Время на локальном сервере.
  • "$request""GET / HTTP/1.1" — тип запроса, путь и протокол.
  • $status200 — Код ответа сервера.
  • $body_bytes_sent396 — Размер ответа сервера в байтах.
  • "$http_referer""-" — URL перехода.
  • "$http_user_agent"Mozilla/5.0 ... — Пользовательский агент клиента (веб-браузер).

Используйте команду tail для просмотра файла журнала в режиме реального времени:

tail -f  access.log 

Выводы

Файлы журналов содержат полезную информацию о проблемах с сервером и о том, как посетители взаимодействуют с вашим сайтом.

Nginx позволяет настроить журналы доступа и ошибок в соответствии с вашими потребностями.

Если у вас есть какие-либо вопросы или отзывы, не стесняйтесь оставлять комментарии.

In this tutorial, you will learn everything you need to know about logging in
NGINX and how it can help you troubleshoot and quickly resolve any problem you
may encounter on your web server. We will discuss where the logs are stored and
how to access them, how to customize their format, and how to centralize them in
one place with Syslog or a log management service.

Here’s an outline of what you will learn by following through with this tutorial:

  • Where NGINX logs are stored and how to access them.
  • How to customize the NGINX log format and storage location to fit your needs.
  • How to utilize a structured format (such as JSON) for your NGINX logs.
  • How to centralize NGINX logs through Syslog or a managed cloud-based service.

Prerequisites

To follow through with this tutorial, you need the following:

  • A Linux server that includes a non-root user with sudo privileges. We tested
    the commands shown in this guide on an Ubuntu 20.04 server.
  • The
    NGINX web server installed
    and enabled on your server.

🔭 Want to centralize and monitor your NGINX logs?

Head over to Better Stack and start ingesting
your logs in 5 minutes.

Step 1 — Locating the NGINX log files

NGINX writes logs of all its events in two different log files:

  • Access log: this file contains information about incoming requests and
    user visits.
  • Error log: this file contains information about errors encountered while
    processing requests, or other diagnostic messages about the web server.

The location of both log files is dependent on the host operating system of the
NGINX web server and the mode of installation. On most Linux distributions, both
files will be found in the /var/log/nginx/ directory as access.log and
error.log, respectively.

A typical access log entry might look like the one shown below. It describes an
HTTP GET request to the server for a favicon.ico file.

217.138.222.101 - - [11/Feb/2022:13:22:11 +0000] "GET /favicon.ico HTTP/1.1" 404 3650 "http://135.181.110.245/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36" "-"

Similarly, an error log entry might look like the one below, which was generated
due to the inability of the server to locate the favicon.ico file that was
requested above.

2022/02/11 13:12:24 [error] 37839#37839: *7 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 113.31.102.176, server: _, request: "GET /favicon.ico HTTP/1.1", host: "192.168.110.245:80"

In the next section, you’ll see how to view both NGINX log files from the
command line.

Step 2 — Viewing the NGINX log files

Examining the NGINX logs can be done in a variety of ways. One of the most
common methods involves using the tail command to view logs entries in
real-time:

sudo tail -f /var/log/nginx/access.log

You will observe the following output:

107.189.10.196 - - [14/Feb/2022:03:48:55 +0000] "POST /HNAP1/ HTTP/1.1" 404 134 "-" "Mozila/5.0"
35.162.122.225 - - [14/Feb/2022:04:11:57 +0000] "GET /.env HTTP/1.1" 404 162 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0"
45.61.172.7 - - [14/Feb/2022:04:16:54 +0000] "GET /.env HTTP/1.1" 404 197 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36"
45.61.172.7 - - [14/Feb/2022:04:16:55 +0000] "POST / HTTP/1.1" 405 568 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36"
45.137.21.134 - - [14/Feb/2022:04:18:57 +0000] "GET /dispatch.asp HTTP/1.1" 404 134 "-" "Mozilla/5.0 (iPad; CPU OS 7_1_2 like Mac OS X; en-US) AppleWebKit/531.5.2 (KHTML, like Gecko) Version/4.0.5 Mobile/8B116 Safari/6531.5.2"
23.95.100.141 - - [14/Feb/2022:04:42:23 +0000] "HEAD / HTTP/1.0" 200 0 "-" "-"
217.138.222.101 - - [14/Feb/2022:07:38:40 +0000] "GET /icons/ubuntu-logo.png HTTP/1.1" 404 197 "http://168.119.119.25/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36"
217.138.222.101 - - [14/Feb/2022:07:38:42 +0000] "GET /favicon.ico HTTP/1.1" 404 197 "http://168.119.119.25/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36"
217.138.222.101 - - [14/Feb/2022:07:44:02 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36"
217.138.222.101 - - [14/Feb/2022:07:44:02 +0000] "GET /icons/ubuntu-logo.png HTTP/1.1" 404 197 "http://168.119.119.25/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36"

The tail command prints the last 10 lines from the selected file. The -f
option causes it to continue displaying subsequent lines that are added to the
file in real-time.

To examine the entire contents of an NGINX log file, you can use the cat
command or open it in your text editor:

sudo cat /var/log/nginx/error.log

If you want to filter the lines that contain a specific term, you can use the
grep command as shown below:

sudo grep "GET /favicon.ico" /var/log/nginx/access.log

The command above will print all the lines that contain GET /favicon.ico so we
can see how many requests were made for that resource.

Step 3 — Configuring NGINX access logs

The NGINX access log stores data about incoming client requests to the server
which is beneficial when deciphering what users are doing in the application,
and what resources are being requested. In this section, you will learn how to
configure what data is stored in the access log.

One thing to keep in mind while following through with the instructions below is
that you’ll need to restart the nginx service after modifying the config file
so that the changes can take effect.

sudo systemctl restart nginx

Enabling the access log

The NGINX access Log should be enabled by default. However, if this is not the
case, you can enable it manually in the Nginx configuration file
(/etc/nginx/nginx.conf) using the access_log directive within the http
block.

/etc/nginx/nginx.conf

Copied!

http {
  access_log /var/log/nginx/access.log;
}

This directive is also applicable in the server and location configuration
blocks for a specific website:

/etc/nginx/nginx.conf

Copied!

server {
   access_log /var/log/nginx/app1.access.log;

  location /app2 {
    access_log /var/log/nginx/app2.access.log;
  }
}

Disabling the access log

In cases where you’d like to disable the NGINX access log, you can use the
special off value:

You can also disable the access log on a virtual server or specific URIs by
editing its server or location block configuration in the
/etc/nginx/sites-available/ directory:

server {
  listen 80;

  access_log off;

  location ~* \.(woff|jpg|jpeg|png|gif|ico|css|js)$ {
    access_log off;
  }
}

Logging to multiple access log files

If you’d like to duplicate the access log entries in separate files, you can do
so by repeating the access_log directive in the main config file or in a
server block as shown below:

access_log /var/log/nginx/access.log;
access_log /var/log/nginx/combined.log;

Don’t forget to restart the nginx service afterward:

sudo systemctl restart nginx

Explanation of the default access log format

The access log entries produced using the default configuration will look like
this:

127.0.0.1 alice Alice [07/May/2021:10:44:53 +0200] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4531.93 Safari/537.36"

Here’s a breakdown of the log message above:

  • 127.0.0.1: the IP address of the client that made the request.
  • alice: remote log name (name used to log in a user).
  • Alice: remote username (username of logged-in user).
  • [07/May/2021:10:44:53 +0200] : date and time of the request.
  • "GET / HTTP/1.1" : request method, path and protocol.
  • 200: the HTTP response code.
  • 396: the size of the response in bytes.
  • "-": the IP address of the referrer (- is used when the it is not
    available).
  • "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4531.93 Safari/537.36"
    detailed user agent information.

Step 4 — Creating a custom log format

Customizing the format of the entries in the access log can be done using the
log_format directive, and it can be placed in the http, server or
location blocks as needed. Here’s an example of what it could look like:

log_format custom '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"';

This yields a log entry in the following format:

217.138.222.109 - - [14/Feb/2022:10:38:35 +0000] "GET /favicon.ico HTTP/1.1" 404 197 "http://192.168.100.1/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36"

The syntax for configuring an access log format is shown below. First, you need
to specify a nickname for the format that will be used as its identifier, and
then the log format string that represents the details and formatting for each
log message.

log_format <nickname> '<formatting_variables>';

Here’s an explanation of each variable used in the custom log format shown
above:

  • $remote_addr: the IP address of the client
  • $remote_user: information about the user making the request
  • $time_local: the server’s date and time.
  • $request: actual request details like path, method, and protocol.
  • $status: the response code.
  • $body_bytes_sent: the size of the response in bytes.
  • $http_referer: the IP address of the HTTP referrer.
  • $http_user_agent: detailed user agent information.

You may also use the following variables in your custom log format
(see here for the complete list):

  • $upstream_connect_time: the time spent establishing a connection with an
    upstream server.
  • $upstream_header_time: the time between establishing a connection and
    receiving the first byte of the response header from the upstream server.
  • $upstream_response_time: the time between establishing a connection and
    receiving the last byte of the response body from the upstream server.
  • $request_time: the total time spent processing a request.
  • $gzip_ratio: ration of gzip compression (if gzip is enabled).

After you create a custom log format, you can apply it to a log file by
providing a second parameter to the access_log directive:

access_log /var/log/nginx/access.log custom;

You can use this feature to log different information in to separate log files.
Create the log formats first:

log_format custom '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer"';
log_format agent "$http_user_agent";

Then, apply them as shown below:

access_log /var/log/nginx/access.log custom;
access_log /var/log/nginx/agent_access.log agent;

This configuration ensures that user agent information for all incoming requests
are logged into a separate access log file.

Step 5 — Formatting your access logs as JSON

A common way to customize NGINX access logs is to format them as JSON. This is
quite straightforward to achieve by combining the log_format directive with
the escape=json parameter introduced in Nginx 1.11.8 to escape characters that
are not valid in JSON:

log_format custom_json escape=json
  '{'
    '"time_local":"$time_local",'
    '"remote_addr":"$remote_addr",'
    '"remote_user":"$remote_user",'
    '"request":"$request",'
    '"status": "$status",'
    '"body_bytes_sent":"$body_bytes_sent",'
    '"request_time":"$request_time",'
    '"http_referrer":"$http_referer",'
    '"http_user_agent":"$http_user_agent"'
  '}';

After applying the custom_json format to a log file and restarting the nginx
service, you will observe log entries in the following format:

{
  "time_local": "14/Feb/2022:11:25:44 +0000",
  "remote_addr": "217.138.222.109",
  "remote_user": "",
  "request": "GET /icons/ubuntu-logo.png HTTP/1.1",
  "status": "404",
  "body_bytes_sent": "197",
  "request_time": "0.000",
  "http_referrer": "http://192.168.100.1/",
  "http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.87 Safari/537.36"
}

Step 6 — Configuring NGINX error logs

Whenever NGINX encounters an error, it stores the event data in the error log so
that it can be referred to later by a system administrator. This section will
describe how to enable and customize the error logs as you see fit.

Enabling the error log

The NGINX error log should be enabled by default. However, if this is not the
case, you can enable it manually in the relevant NGINX configuration file
(either at the http, server, or location levels) using the error_log
directive.

error_log /var/log/nginx/error.log;

The error_log directive can take two parameters. The first one is the location
of the log file (as shown above), while the second one is optional and sets the
severity level of the log. Events with a lower severity level than set one will
not be logged.

error_log /var/log/nginx/error.log info;

These are the possible levels of severity (from lowest to highest) and their
meaning:

  • debug: messages used for debugging.
  • info: informational messages.
  • notice: a notable event occurred.
  • warn: something unexpected happened.
  • error: something failed.
  • crit: critical conditions.
  • alert: errors that require immediate action.
  • emerg: the system is unusable.

Disabling the error log

The NGINX error log can be disabled by setting the error_log directive to
off or by redirecting it to /dev/null:

error_log off;
error_log /dev/null;

Logging errors into multiple files

As is the case with access logs, you can log errors into multiple files, and you
can use different severity levels too:

error_log /var/log/nginx/error.log info;
error_log /var/log/nginx/emerg_error.log emerg;

This configuration will log every event except those at the debug level event
to the error.log file, while emergency events are placed in a separate
emerg_error.log file.

Step 7 — Sending NGINX logs to Syslog

Apart from logging to a file, it’s also possible to set up NGINX to transport
its logs to the syslog service especially if you’re already using it for other
system logs. Logging to syslog is done by specifying the syslog: prefix to
either the access_log or error_log directive:

error_log  syslog:server=unix:/var/log/nginx.sock debug;
access_log syslog:server=[127.0.0.1]:1234,facility=local7,tag=nginx,severity=info;

Log messages are sent to a server which can be specified in terms of a domain
name, IPv4 or IPv6 address or a UNIX-domain socket path.

In the example above, error log messages are sent to a UNIX domain socket at the
debug logging level, while the access log is written to a syslog server with
an IPv4 address and port 1234. The facility= parameter specifies the type of
program that is logging the message, the tag= parameter applies a custom tag
to syslog messages, and the severity= parameter sets the severity level of
the syslog entry for access log messages.

For more information on using Syslog to manage your logs, you can check out our
tutorial on viewing and configuring system logs on
Linux.

Step 8 — Centralizing your NGINX logs

In this section, we’ll describe how you can centralize your NGINX logs in a log
management service through Vector, a
high-performance tool for building observability pipelines. This is a crucial
step when administrating multiple servers so that you can monitor all your logs
in one place (you can also centralize your logs with an Rsyslog
server).

The following instructions assume that you’ve signed up for a free
Logtail account and retrieved your source
token. Go ahead and follow the relevant
installation instructions for Vector
for your operating system. For example, on Ubuntu, you may run the following
commands to install the Vector CLI:

curl -1sLf \ 'https://repositories.timber.io/public/vector/cfg/setup/bash.deb.sh' \ | sudo -E bash

After Vector is installed, confirm that it is up and running through
systemctl:

You should observe that it is active and running:

● vector.service - Vector
     Loaded: loaded (/lib/systemd/system/vector.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-02-08 10:52:59 UTC; 48s ago
       Docs: https://vector.dev
    Process: 18586 ExecStartPre=/usr/bin/vector validate (code=exited, status=0/SUCCESS)
   Main PID: 18599 (vector)
      Tasks: 3 (limit: 2275)
     Memory: 6.8M
     CGroup: /system.slice/vector.service
             └─18599 /usr/bin/vector

Otherwise, go ahead and start it with the command below.

sudo systemctl start vector

Afterward, change into a root shell and append your Logtail vector configuration
for NGINX into the /etc/vector/vector.toml file using the command below. Don’t
forget to replace the <your_logtail_source_token> placeholder below with your
source token.

wget -O ->> /etc/vector/vector.toml \
    https://logtail.com/vector-toml/nginx/<your_logtail_source_token>

Then restart the vector service:

sudo systemctl restart vector

You will observe that your NGINX logs will start coming through in Logtail:

Conclusion

In this tutorial, you learned about the different types of logs that the NGINX
web server keeps, where you can find them, how to understand their formatting.
We also discussed how to create your own custom log formats (including a
structured JSON format), and how to log into multiple files at once. Finally, we
demonstrated the process of sending your logs to Syslog or a log management
service so that you can monitor them all in one place.

Thanks for reading, and happy logging!

Author's avatar

Article by

Ayooluwa Isaiah

Ayo is the Head of Content at Better Stack. His passion is simplifying and communicating complex technical ideas effectively. His work was featured on several esteemed publications including LWN.net, Digital Ocean, and CSS-Tricks. When he’s not writing or coding, he loves to travel, bike, and play tennis.

Centralize all your logs into one place.

Analyze, correlate and filter logs with SQL.

Create actionable

dashboards.

Share and comment with built-in collaboration.

Got an article suggestion?
Let us know

Next article

How to Get Started with Logging in Node.js

Learn how to start logging with Node.js and go from basics to best practices in no time.

Licensed under CC-BY-NC-SA

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

The ngx_http_log_module module writes request logs
in the specified format.

Requests are logged in the context of a location where processing ends.
It may be different from the original location, if an
internal
redirect happens during request processing.

Example Configuration

log_format compression '$remote_addr - $remote_user [$time_local] '
                       '"$request" $status $bytes_sent '
                       '"$http_referer" "$http_user_agent" "$gzip_ratio"';

access_log /spool/logs/nginx-access.log compression buffer=32k;

Directives

Syntax: access_log
path
[format
[buffer=size]
[gzip[=level]]
[flush=time]
[if=condition]];

access_log off;
Default:
access_log logs/access.log combined;
Context: http, server, location, if in location, limit_except

Sets the path, format, and configuration for a buffered log write.
Several logs can be specified on the same configuration level.
Logging to syslog
can be configured by specifying
the “syslog:” prefix in the first parameter.
The special value off cancels all
access_log directives on the current level.
If the format is not specified then the predefined
combined” format is used.

If either the buffer or gzip
(1.3.10, 1.2.7)
parameter is used, writes to log will be buffered.

The buffer size must not exceed the size of an atomic write to a disk file.
For FreeBSD this size is unlimited.

When buffering is enabled, the data will be written to the file:

  • if the next log line does not fit into the buffer;
  • if the buffered data is older than specified by the flush
    parameter (1.3.10, 1.2.7);
  • when a worker process is re-opening log
    files or is shutting down.

If the gzip parameter is used, then the buffered data will
be compressed before writing to the file.
The compression level can be set between 1 (fastest, less compression)
and 9 (slowest, best compression).
By default, the buffer size is equal to 64K bytes, and the compression level
is set to 1.
Since the data is compressed in atomic blocks, the log file can be decompressed
or read by “zcat” at any time.

Example:

access_log /path/to/log.gz combined gzip flush=5m;

For gzip compression to work, nginx must be built with the zlib library.

The file path can contain variables (0.7.6+),
but such logs have some constraints:

  • the user
    whose credentials are used by worker processes should
    have permissions to create files in a directory with
    such logs;
  • buffered writes do not work;
  • the file is opened and closed for each log write.
    However, since the descriptors of frequently used files can be stored
    in a cache, writing to the old file
    can continue during the time specified by the open_log_file_cache
    directive’s valid parameter
  • during each log write the existence of the request’s
    root directory
    is checked, and if it does not exist the log is not
    created.
    It is thus a good idea to specify both
    root
    and access_log on the same configuration level:

    server {
        root       /spool/vhost/data/$host;
        access_log /spool/vhost/logs/$host;
        ...
    

The if parameter (1.7.0) enables conditional logging.
A request will not be logged if the condition evaluates to “0”
or an empty string.
In the following example, the requests with response codes 2xx and 3xx
will not be logged:

map $status $loggable {
    ~^[23]  0;
    default 1;
}

access_log /path/to/access.log combined if=$loggable;
Syntax: log_format
name
[escape=default|json|none]
string ...;
Default:
log_format combined "...";
Context: http

Specifies log format.

The escape parameter (1.11.8) allows setting
json or default characters escaping
in variables, by default, default escaping is used.
The none value (1.13.10) disables escaping.

For default escaping,
characters “"”, “\”,
and other characters with values less than 32 (0.7.0) or above 126 (1.1.6)
are escaped as “\xXX”.
If the variable value is not found,
a hyphen (“-”) will be logged.

For json escaping,
all characters not allowed
in JSON strings
will be escaped:
characters “"” and
\” are escaped as
\"” and “\\”,
characters with values less than 32 are escaped as
\n”,
\r”,
\t”,
\b”,
\f”, or
\u00XX”.

The log format can contain common variables, and variables that
exist only at the time of a log write:

$bytes_sent
the number of bytes sent to a client
$connection
connection serial number
$connection_requests
the current number of requests made through a connection (1.1.18)
$msec
time in seconds with a milliseconds resolution at the time of the log write
$pipe
p” if request was pipelined, “.
otherwise
$request_length
request length (including request line, header, and request body)
$request_time
request processing time in seconds with a milliseconds resolution;
time elapsed between the first bytes were read from the client and
the log write after the last bytes were sent to the client
$status
response status
$time_iso8601
local time in the ISO 8601 standard format
$time_local
local time in the Common Log Format

In the modern nginx versions variables
$status
(1.3.2, 1.2.2),
$bytes_sent
(1.3.8, 1.2.5),
$connection
(1.3.8, 1.2.5),
$connection_requests
(1.3.8, 1.2.5),
$msec
(1.3.9, 1.2.6),
$request_time
(1.3.9, 1.2.6),
$pipe
(1.3.12, 1.2.7),
$request_length
(1.3.12, 1.2.7),
$time_iso8601
(1.3.12, 1.2.7),
and
$time_local
(1.3.12, 1.2.7)
are also available as common variables.

Header lines sent to a client have the prefix
sent_http_”, for example,
$sent_http_content_range.

The configuration always includes the predefined
combined” format:

log_format combined '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';
Syntax: open_log_file_cache
max=N
[inactive=time]
[min_uses=N]
[valid=time];

open_log_file_cache off;
Default:
open_log_file_cache off;
Context: http, server, location

Defines a cache that stores the file descriptors of frequently used logs
whose names contain variables.
The directive has the following parameters:

max
sets the maximum number of descriptors in a cache;
if the cache becomes full the least recently used (LRU)
descriptors are closed
inactive
sets the time after which the cached descriptor is closed
if there were no access during this time;
by default, 10 seconds
min_uses
sets the minimum number of file uses during the time
defined by the inactive parameter
to let the descriptor stay open in a cache;
by default, 1
valid
sets the time after which it should be checked that the file
still exists with the same name; by default, 60 seconds
off
disables caching

Usage example:

open_log_file_cache max=1000 inactive=20s valid=1m min_uses=2;

NGINX Access & Error Logs

NGINX is the most popular web server in the world, powering close to 35% of all sites according to w2techs. As a popular part of your software stack, it is good to understand its logs. NGINX write two specific types of logs: Error.log and Access.log. This article will dive into each one respectively.

Log & Configuration Locations

Before you get started, it’s best to orient yourself with where things are located. Below are the two most important locations to know as it pertains to logs:

Name Description
Log Locationn /var/log/nginx
Config Locationn /etc/nginx/nginx.conf

NGINX Access Logs

The NGINX access log exist to record the activity of all the visitors to your site. You can use these logs to see what pages are being visited, what files are being requested, where users are coming from, and the list goes on. It also contains sensitive information like the users IP, the date and time of the request, and even the type of application being used to access the site (e.g., Chrome vs Firefox, Mobile vs Desktop). This log is especially helpful when you’re trying to better understand your visitor behavior.

By default, all access logs have a standard format are set in the nginx.conf file with the access_log directive. The format looks like this:


access_log log_file log_format;


http {
       ...
       ...
       access_log /var/log/nginx/access.log;
       ...
       ...
}

1. Access Log Format

By default, NGINX will record all all logs in the combined format, which records the following information:


log_format combined '$remote_addr - $remote_user [$time_local] '
    '"$request" $status $body_bytes_sent '
    '"$http_referer" "$http_user_agent"';

You are not limited to this format, however. A format can be changed by specifying the format you want to use. For example, let’s assume you create a custom format and this format is focused on adding the X-Forwarded-For values to the log.

You would create the new log format:


log_format custom '$remote_addr - $remote_user [$time_local] "$request" '
    '$status $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for"';

Now using the same structure above, you simply add the format type to the end of the configuration file:


http {
       ...
       ...
       access_log /var/log/nginx/access.log custom;
       ...
       ...
}

2. Access Log Sample

Here is an example of a log entry recorded in an access log file:


192.168.33.1 - - [15/Oct/2019:19:41:46 +0000] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"

This is how it corresponds to the default log format:

Format Value Description
$remote_addr 192.168.33.1 The IP address of the client making the request.
$remote_user HTTP Authenticated User. When the user name is not set, this field shows -.
[$time_local] [15/Oct/2019:19:41:46 +0000] Local server time.
$request GET / HTTP/1.1 The request type, path and protocol.
$status 200 The server response code
$body_bytes_sent 396 The size of server response in bytes.
$http_referer The URL of the referral.
$http_user_agent Mozilla/5.0 … The user agent of the client (web browser).

NGINX Error Logs

The NGINX error log exists to record any issues, glitches, that the application might encounter. Maybe there is an error in a configuration file, or a conflict with a dependency. Maybe the application is forced to reboot, or fails to initialize. This is the log you want to analyze when something goes wrong as it usually contains a bit more detail and can help point you in the right direction. It’s also extremely helpful to spend time analyzing the warnings, they can be indicative a problem.

By default, all error logs have a standard format and are set in the nginx.conf with the error_log directive. The format looks like this:


error_log log_file log_level;


http {
      ...
      ...
      error_log /var/log/nginx/error.log;
      ...
      ...
}

1. Error Log Sample

The log starts with the date, time, followed by the criticality of the event (error, warn, crit) and the details of the problem. Watching these logs is critical to understanding your server and detecting problems that might be impacting your users.

Example log:


2022/05/14 19:55:51 [error] 6689#6689: *16764375 FastCGI sent in stderr: "PHP message: PHP Warning: Division by zero in /var/www/html/my.php on line 438" while reading upstream, client: 10.1.1.1, server: mytest.com, request: "GET /my.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.2-fpm.sock:", host: "my.test.com"

In this log, it is categorized as [error], because of a warning from the PHP engine due a division by zero.

2. Log Severity

Speaking of which, the NGINX error logs provide 8 unique severity levels you should be aware of.

Name Description
1. emerg Emergency messages when your system may be unstable.
2. alert Alert messages of serious issues.
3. crit Critical issues that need to be taken care of immediately.
4. error An error has occurred. Something went wrong while processing a page.
5. warn A warning messages that you should look into it.
6. notice A simple log notice that you can ignore.
7. info Just an information messages that you might want to know.
8. debug Debugging information used to pinpoint the location of error.

3. Common Error Types

These are the common nginx error logs that you might have on your server:

Name Description
FastCGI sent in stderr Error coming from the FastCGI module (often from your web application).
PHP message: PHP Warning If your web application uses PHP and generates a warning, you may get this type of error. We recommend passing to the developers to investigate and fixing them.
2022/05/14 07:08:37 [crit] 6689#6689: *16721837 SSL_do_handshake() failed (SSL: error:1420918C:SSL routines:tls_early_post_process_client_hello:version too low) while SSL handshaking The SSL_do_handshake happens whenever the client/server can’t complete the SSL handshaking or it is ended too soon in the process. It can happen many times on a busy server and is often not a cause for concern.

NGINX Logs by Domain

By default, NGINX will store all the logs as one log entry on the server, but if you’re deploying multiple domains on the same server this is highly impractical. This is why you have an option to configure logs for each domain (both access and errors) and you do this via the VirtualHosts file on the server.

Using the examples, formatting, provided above, this is how your virtual hosts file would look like for unique domains:


http {
      ...
      ...
      error_log /var/log/nginx/error_log;
      server {
            listen 80;
            server_name domain1.com;
            access_log /var/log/nginx/domain1.access.log;
            error_log /var/log/nginx/domain1.error_log warn;
            ...
      }
      server {
            listen 80;
            server_name domain2.com;
            access_log /var/log/nginx/domain1.access.log;
            error_log /var/log/nginx/domain2.error_log debug;
            ...
      }
}

In the example above you see domain1 and domain2 each have their own access and error logs set. You also notice that you have the option to set the severity level for each domain as well (e.g., warn, debug).


Posted in  
log-guide   nginx-logs    

by Tony Perez (@perezbox)

Понравилась статья? Поделить с друзьями:
  • Журнал ошибок kyocera
  • Журнал ошибок iphone
  • Журнал ошибок bsod
  • Журнал ошибок android
  • Журнал ошибок 1с розница