Postgresql посмотреть ошибки

20.8.1. Where to Log #

log_destination (string) #

PostgreSQL supports several methods for logging server messages, including stderr, csvlog, jsonlog, and syslog. On Windows, eventlog is also supported. Set this parameter to a list of desired log destinations separated by commas. The default is to log to stderr only. This parameter can only be set in the postgresql.conf file or on the server command line.

If csvlog is included in log_destination, log entries are output in comma separated value (CSV) format, which is convenient for loading logs into programs. See Section 20.8.4 for details. logging_collector must be enabled to generate CSV-format log output.

If jsonlog is included in log_destination, log entries are output in JSON format, which is convenient for loading logs into programs. See Section 20.8.5 for details. logging_collector must be enabled to generate JSON-format log output.

When either stderr, csvlog or jsonlog are included, the file current_logfiles is created to record the location of the log file(s) currently in use by the logging collector and the associated logging destination. This provides a convenient way to find the logs currently in use by the instance. Here is an example of this file’s content:

stderr log/postgresql.log
csvlog log/postgresql.csv
jsonlog log/postgresql.json

current_logfiles is recreated when a new log file is created as an effect of rotation, and when log_destination is reloaded. It is removed when none of stderr, csvlog or jsonlog are included in log_destination, and when the logging collector is disabled.

Note

On most Unix systems, you will need to alter the configuration of your system’s syslog daemon in order to make use of the syslog option for log_destination. PostgreSQL can log to syslog facilities LOCAL0 through LOCAL7 (see syslog_facility), but the default syslog configuration on most platforms will discard all such messages. You will need to add something like:

local0.*    /var/log/postgresql

to the syslog daemon’s configuration file to make it work.

On Windows, when you use the eventlog option for log_destination, you should register an event source and its library with the operating system so that the Windows Event Viewer can display event log messages cleanly. See Section 19.12 for details.

logging_collector (boolean) #

This parameter enables the logging collector, which is a background process that captures log messages sent to stderr and redirects them into log files. This approach is often more useful than logging to syslog, since some types of messages might not appear in syslog output. (One common example is dynamic-linker failure messages; another is error messages produced by scripts such as archive_command.) This parameter can only be set at server start.

Note

It is possible to log to stderr without using the logging collector; the log messages will just go to wherever the server’s stderr is directed. However, that method is only suitable for low log volumes, since it provides no convenient way to rotate log files. Also, on some platforms not using the logging collector can result in lost or garbled log output, because multiple processes writing concurrently to the same log file can overwrite each other’s output.

Note

The logging collector is designed to never lose messages. This means that in case of extremely high load, server processes could be blocked while trying to send additional log messages when the collector has fallen behind. In contrast, syslog prefers to drop messages if it cannot write them, which means it may fail to log some messages in such cases but it will not block the rest of the system.

log_directory (string) #

When logging_collector is enabled, this parameter determines the directory in which log files will be created. It can be specified as an absolute path, or relative to the cluster data directory. This parameter can only be set in the postgresql.conf file or on the server command line. The default is log.

log_filename (string) #

When logging_collector is enabled, this parameter sets the file names of the created log files. The value is treated as a strftime pattern, so %-escapes can be used to specify time-varying file names. (Note that if there are any time-zone-dependent %-escapes, the computation is done in the zone specified by log_timezone.) The supported %-escapes are similar to those listed in the Open Group’s strftime specification. Note that the system’s strftime is not used directly, so platform-specific (nonstandard) extensions do not work. The default is postgresql-%Y-%m-%d_%H%M%S.log.

If you specify a file name without escapes, you should plan to use a log rotation utility to avoid eventually filling the entire disk. In releases prior to 8.4, if no % escapes were present, PostgreSQL would append the epoch of the new log file’s creation time, but this is no longer the case.

If CSV-format output is enabled in log_destination, .csv will be appended to the timestamped log file name to create the file name for CSV-format output. (If log_filename ends in .log, the suffix is replaced instead.)

If JSON-format output is enabled in log_destination, .json will be appended to the timestamped log file name to create the file name for JSON-format output. (If log_filename ends in .log, the suffix is replaced instead.)

This parameter can only be set in the postgresql.conf file or on the server command line.

log_file_mode (integer) #

On Unix systems this parameter sets the permissions for log files when logging_collector is enabled. (On Microsoft Windows this parameter is ignored.) The parameter value is expected to be a numeric mode specified in the format accepted by the chmod and umask system calls. (To use the customary octal format the number must start with a 0 (zero).)

The default permissions are 0600, meaning only the server owner can read or write the log files. The other commonly useful setting is 0640, allowing members of the owner’s group to read the files. Note however that to make use of such a setting, you’ll need to alter log_directory to store the files somewhere outside the cluster data directory. In any case, it’s unwise to make the log files world-readable, since they might contain sensitive data.

This parameter can only be set in the postgresql.conf file or on the server command line.

log_rotation_age (integer) #

When logging_collector is enabled, this parameter determines the maximum amount of time to use an individual log file, after which a new log file will be created. If this value is specified without units, it is taken as minutes. The default is 24 hours. Set to zero to disable time-based creation of new log files. This parameter can only be set in the postgresql.conf file or on the server command line.

log_rotation_size (integer) #

When logging_collector is enabled, this parameter determines the maximum size of an individual log file. After this amount of data has been emitted into a log file, a new log file will be created. If this value is specified without units, it is taken as kilobytes. The default is 10 megabytes. Set to zero to disable size-based creation of new log files. This parameter can only be set in the postgresql.conf file or on the server command line.

log_truncate_on_rotation (boolean) #

When logging_collector is enabled, this parameter will cause PostgreSQL to truncate (overwrite), rather than append to, any existing log file of the same name. However, truncation will occur only when a new file is being opened due to time-based rotation, not during server startup or size-based rotation. When off, pre-existing files will be appended to in all cases. For example, using this setting in combination with a log_filename like postgresql-%H.log would result in generating twenty-four hourly log files and then cyclically overwriting them. This parameter can only be set in the postgresql.conf file or on the server command line.

Example: To keep 7 days of logs, one log file per day named server_log.Mon, server_log.Tue, etc., and automatically overwrite last week’s log with this week’s log, set log_filename to server_log.%a, log_truncate_on_rotation to on, and log_rotation_age to 1440.

Example: To keep 24 hours of logs, one log file per hour, but also rotate sooner if the log file size exceeds 1GB, set log_filename to server_log.%H%M, log_truncate_on_rotation to on, log_rotation_age to 60, and log_rotation_size to 1000000. Including %M in log_filename allows any size-driven rotations that might occur to select a file name different from the hour’s initial file name.

syslog_facility (enum) #

When logging to syslog is enabled, this parameter determines the syslog facility to be used. You can choose from LOCAL0, LOCAL1, LOCAL2, LOCAL3, LOCAL4, LOCAL5, LOCAL6, LOCAL7; the default is LOCAL0. See also the documentation of your system’s syslog daemon. This parameter can only be set in the postgresql.conf file or on the server command line.

syslog_ident (string) #

When logging to syslog is enabled, this parameter determines the program name used to identify PostgreSQL messages in syslog logs. The default is postgres. This parameter can only be set in the postgresql.conf file or on the server command line.

syslog_sequence_numbers (boolean) #

When logging to syslog and this is on (the default), then each message will be prefixed by an increasing sequence number (such as [2]). This circumvents the — last message repeated N times — suppression that many syslog implementations perform by default. In more modern syslog implementations, repeated message suppression can be configured (for example, $RepeatedMsgReduction in rsyslog), so this might not be necessary. Also, you could turn this off if you actually want to suppress repeated messages.

This parameter can only be set in the postgresql.conf file or on the server command line.

syslog_split_messages (boolean) #

When logging to syslog is enabled, this parameter determines how messages are delivered to syslog. When on (the default), messages are split by lines, and long lines are split so that they will fit into 1024 bytes, which is a typical size limit for traditional syslog implementations. When off, PostgreSQL server log messages are delivered to the syslog service as is, and it is up to the syslog service to cope with the potentially bulky messages.

If syslog is ultimately logging to a text file, then the effect will be the same either way, and it is best to leave the setting on, since most syslog implementations either cannot handle large messages or would need to be specially configured to handle them. But if syslog is ultimately writing into some other medium, it might be necessary or more useful to keep messages logically together.

This parameter can only be set in the postgresql.conf file or on the server command line.

event_source (string) #

When logging to event log is enabled, this parameter determines the program name used to identify PostgreSQL messages in the log. The default is PostgreSQL. This parameter can only be set in the postgresql.conf file or on the server command line.

20.8.2. When to Log #

log_min_messages (enum) #

Controls which message levels are written to the server log. Valid values are DEBUG5, DEBUG4, DEBUG3, DEBUG2, DEBUG1, INFO, NOTICE, WARNING, ERROR, LOG, FATAL, and PANIC. Each level includes all the levels that follow it. The later the level, the fewer messages are sent to the log. The default is WARNING. Note that LOG has a different rank here than in client_min_messages. Only superusers and users with the appropriate SET privilege can change this setting.

log_min_error_statement (enum) #

Controls which SQL statements that cause an error condition are recorded in the server log. The current SQL statement is included in the log entry for any message of the specified severity or higher. Valid values are DEBUG5, DEBUG4, DEBUG3, DEBUG2, DEBUG1, INFO, NOTICE, WARNING, ERROR, LOG, FATAL, and PANIC. The default is ERROR, which means statements causing errors, log messages, fatal errors, or panics will be logged. To effectively turn off logging of failing statements, set this parameter to PANIC. Only superusers and users with the appropriate SET privilege can change this setting.

log_min_duration_statement (integer) #

Causes the duration of each completed statement to be logged if the statement ran for at least the specified amount of time. For example, if you set it to 250ms then all SQL statements that run 250ms or longer will be logged. Enabling this parameter can be helpful in tracking down unoptimized queries in your applications. If this value is specified without units, it is taken as milliseconds. Setting this to zero prints all statement durations. -1 (the default) disables logging statement durations. Only superusers and users with the appropriate SET privilege can change this setting.

This overrides log_min_duration_sample, meaning that queries with duration exceeding this setting are not subject to sampling and are always logged.

For clients using extended query protocol, durations of the Parse, Bind, and Execute steps are logged independently.

Note

When using this option together with log_statement, the text of statements that are logged because of log_statement will not be repeated in the duration log message. If you are not using syslog, it is recommended that you log the PID or session ID using log_line_prefix so that you can link the statement message to the later duration message using the process ID or session ID.

log_min_duration_sample (integer) #

Allows sampling the duration of completed statements that ran for at least the specified amount of time. This produces the same kind of log entries as log_min_duration_statement, but only for a subset of the executed statements, with sample rate controlled by log_statement_sample_rate. For example, if you set it to 100ms then all SQL statements that run 100ms or longer will be considered for sampling. Enabling this parameter can be helpful when the traffic is too high to log all queries. If this value is specified without units, it is taken as milliseconds. Setting this to zero samples all statement durations. -1 (the default) disables sampling statement durations. Only superusers and users with the appropriate SET privilege can change this setting.

This setting has lower priority than log_min_duration_statement, meaning that statements with durations exceeding log_min_duration_statement are not subject to sampling and are always logged.

Other notes for log_min_duration_statement apply also to this setting.

log_statement_sample_rate (floating point) #

Determines the fraction of statements with duration exceeding log_min_duration_sample that will be logged. Sampling is stochastic, for example 0.5 means there is statistically one chance in two that any given statement will be logged. The default is 1.0, meaning to log all sampled statements. Setting this to zero disables sampled statement-duration logging, the same as setting log_min_duration_sample to -1. Only superusers and users with the appropriate SET privilege can change this setting.

log_transaction_sample_rate (floating point) #

Sets the fraction of transactions whose statements are all logged, in addition to statements logged for other reasons. It applies to each new transaction regardless of its statements’ durations. Sampling is stochastic, for example 0.1 means there is statistically one chance in ten that any given transaction will be logged. log_transaction_sample_rate can be helpful to construct a sample of transactions. The default is 0, meaning not to log statements from any additional transactions. Setting this to 1 logs all statements of all transactions. Only superusers and users with the appropriate SET privilege can change this setting.

Note

Like all statement-logging options, this option can add significant overhead.

log_startup_progress_interval (integer) #

Sets the amount of time after which the startup process will log a message about a long-running operation that is still in progress, as well as the interval between further progress messages for that operation. The default is 10 seconds. A setting of 0 disables the feature. If this value is specified without units, it is taken as milliseconds. This setting is applied separately to each operation. This parameter can only be set in the postgresql.conf file or on the server command line.

For example, if syncing the data directory takes 25 seconds and thereafter resetting unlogged relations takes 8 seconds, and if this setting has the default value of 10 seconds, then a messages will be logged for syncing the data directory after it has been in progress for 10 seconds and again after it has been in progress for 20 seconds, but nothing will be logged for resetting unlogged relations.

Table 20.2 explains the message severity levels used by PostgreSQL. If logging output is sent to syslog or Windows’ eventlog, the severity levels are translated as shown in the table.

Table 20.2. Message Severity Levels

Severity Usage syslog eventlog
DEBUG1 .. DEBUG5 Provides successively-more-detailed information for use by developers. DEBUG INFORMATION
INFO Provides information implicitly requested by the user, e.g., output from VACUUM VERBOSE. INFO INFORMATION
NOTICE Provides information that might be helpful to users, e.g., notice of truncation of long identifiers. NOTICE INFORMATION
WARNING Provides warnings of likely problems, e.g., COMMIT outside a transaction block. NOTICE WARNING
ERROR Reports an error that caused the current command to abort. WARNING ERROR
LOG Reports information of interest to administrators, e.g., checkpoint activity. INFO INFORMATION
FATAL Reports an error that caused the current session to abort. ERR ERROR
PANIC Reports an error that caused all database sessions to abort. CRIT ERROR

20.8.3. What to Log #

Note

What you choose to log can have security implications; see Section 25.3.

application_name (string) #

The application_name can be any string of less than NAMEDATALEN characters (64 characters in a standard build). It is typically set by an application upon connection to the server. The name will be displayed in the pg_stat_activity view and included in CSV log entries. It can also be included in regular log entries via the log_line_prefix parameter. Only printable ASCII characters may be used in the application_name value. Other characters will be replaced with question marks (?).

debug_print_parse (boolean)
debug_print_rewritten (boolean)
debug_print_plan (boolean) #

These parameters enable various debugging output to be emitted. When set, they print the resulting parse tree, the query rewriter output, or the execution plan for each executed query. These messages are emitted at LOG message level, so by default they will appear in the server log but will not be sent to the client. You can change that by adjusting client_min_messages and/or log_min_messages. These parameters are off by default.

debug_pretty_print (boolean) #

When set, debug_pretty_print indents the messages produced by debug_print_parse, debug_print_rewritten, or debug_print_plan. This results in more readable but much longer output than the compact format used when it is off. It is on by default.

log_autovacuum_min_duration (integer) #

Causes each action executed by autovacuum to be logged if it ran for at least the specified amount of time. Setting this to zero logs all autovacuum actions. -1 disables logging autovacuum actions. If this value is specified without units, it is taken as milliseconds. For example, if you set this to 250ms then all automatic vacuums and analyzes that run 250ms or longer will be logged. In addition, when this parameter is set to any value other than -1, a message will be logged if an autovacuum action is skipped due to a conflicting lock or a concurrently dropped relation. The default is 10min. Enabling this parameter can be helpful in tracking autovacuum activity. This parameter can only be set in the postgresql.conf file or on the server command line; but the setting can be overridden for individual tables by changing table storage parameters.

log_checkpoints (boolean) #

Causes checkpoints and restartpoints to be logged in the server log. Some statistics are included in the log messages, including the number of buffers written and the time spent writing them. This parameter can only be set in the postgresql.conf file or on the server command line. The default is on.

log_connections (boolean) #

Causes each attempted connection to the server to be logged, as well as successful completion of both client authentication (if necessary) and authorization. Only superusers and users with the appropriate SET privilege can change this parameter at session start, and it cannot be changed at all within a session. The default is off.

Note

Some client programs, like psql, attempt to connect twice while determining if a password is required, so duplicate connection received messages do not necessarily indicate a problem.

log_disconnections (boolean) #

Causes session terminations to be logged. The log output provides information similar to log_connections, plus the duration of the session. Only superusers and users with the appropriate SET privilege can change this parameter at session start, and it cannot be changed at all within a session. The default is off.

log_duration (boolean) #

Causes the duration of every completed statement to be logged. The default is off. Only superusers and users with the appropriate SET privilege can change this setting.

For clients using extended query protocol, durations of the Parse, Bind, and Execute steps are logged independently.

Note

The difference between enabling log_duration and setting log_min_duration_statement to zero is that exceeding log_min_duration_statement forces the text of the query to be logged, but this option doesn’t. Thus, if log_duration is on and log_min_duration_statement has a positive value, all durations are logged but the query text is included only for statements exceeding the threshold. This behavior can be useful for gathering statistics in high-load installations.

log_error_verbosity (enum) #

Controls the amount of detail written in the server log for each message that is logged. Valid values are TERSE, DEFAULT, and VERBOSE, each adding more fields to displayed messages. TERSE excludes the logging of DETAIL, HINT, QUERY, and CONTEXT error information. VERBOSE output includes the SQLSTATE error code (see also Appendix A) and the source code file name, function name, and line number that generated the error. Only superusers and users with the appropriate SET privilege can change this setting.

log_hostname (boolean) #

By default, connection log messages only show the IP address of the connecting host. Turning this parameter on causes logging of the host name as well. Note that depending on your host name resolution setup this might impose a non-negligible performance penalty. This parameter can only be set in the postgresql.conf file or on the server command line.

log_line_prefix (string) #

This is a printf-style string that is output at the beginning of each log line. % characters begin escape sequences that are replaced with status information as outlined below. Unrecognized escapes are ignored. Other characters are copied straight to the log line. Some escapes are only recognized by session processes, and will be treated as empty by background processes such as the main server process. Status information may be aligned either left or right by specifying a numeric literal after the % and before the option. A negative value will cause the status information to be padded on the right with spaces to give it a minimum width, whereas a positive value will pad on the left. Padding can be useful to aid human readability in log files.

This parameter can only be set in the postgresql.conf file or on the server command line. The default is '%m [%p] ' which logs a time stamp and the process ID.

Escape Effect Session only
%a Application name yes
%u User name yes
%d Database name yes
%r Remote host name or IP address, and remote port yes
%h Remote host name or IP address yes
%b Backend type no
%p Process ID no
%P Process ID of the parallel group leader, if this process is a parallel query worker no
%t Time stamp without milliseconds no
%m Time stamp with milliseconds no
%n Time stamp with milliseconds (as a Unix epoch) no
%i Command tag: type of session’s current command yes
%e SQLSTATE error code no
%c Session ID: see below no
%l Number of the log line for each session or process, starting at 1 no
%s Process start time stamp no
%v Virtual transaction ID (backendID/localXID); see Section 74.1 no
%x Transaction ID (0 if none is assigned); see Section 74.1 no
%q Produces no output, but tells non-session processes to stop at this point in the string; ignored by session processes no
%Q Query identifier of the current query. Query identifiers are not computed by default, so this field will be zero unless compute_query_id parameter is enabled or a third-party module that computes query identifiers is configured. yes
%% Literal % no

The backend type corresponds to the column backend_type in the view pg_stat_activity, but additional types can appear in the log that don’t show in that view.

The %c escape prints a quasi-unique session identifier, consisting of two 4-byte hexadecimal numbers (without leading zeros) separated by a dot. The numbers are the process start time and the process ID, so %c can also be used as a space saving way of printing those items. For example, to generate the session identifier from pg_stat_activity, use this query:

SELECT to_hex(trunc(EXTRACT(EPOCH FROM backend_start))::integer) || '.' ||
       to_hex(pid)
FROM pg_stat_activity;

Tip

If you set a nonempty value for log_line_prefix, you should usually make its last character be a space, to provide visual separation from the rest of the log line. A punctuation character can be used too.

Tip

Syslog produces its own time stamp and process ID information, so you probably do not want to include those escapes if you are logging to syslog.

Tip

The %q escape is useful when including information that is only available in session (backend) context like user or database name. For example:

log_line_prefix = '%m [%p] %q%u@%d/%a '

Note

The %Q escape always reports a zero identifier for lines output by log_statement because log_statement generates output before an identifier can be calculated, including invalid statements for which an identifier cannot be calculated.

log_lock_waits (boolean) #

Controls whether a log message is produced when a session waits longer than deadlock_timeout to acquire a lock. This is useful in determining if lock waits are causing poor performance. The default is off. Only superusers and users with the appropriate SET privilege can change this setting.

log_recovery_conflict_waits (boolean) #

Controls whether a log message is produced when the startup process waits longer than deadlock_timeout for recovery conflicts. This is useful in determining if recovery conflicts prevent the recovery from applying WAL.

The default is off. This parameter can only be set in the postgresql.conf file or on the server command line.

log_parameter_max_length (integer) #

If greater than zero, each bind parameter value logged with a non-error statement-logging message is trimmed to this many bytes. Zero disables logging of bind parameters for non-error statement logs. -1 (the default) allows bind parameters to be logged in full. If this value is specified without units, it is taken as bytes. Only superusers and users with the appropriate SET privilege can change this setting.

This setting only affects log messages printed as a result of log_statement, log_duration, and related settings. Non-zero values of this setting add some overhead, particularly if parameters are sent in binary form, since then conversion to text is required.

log_parameter_max_length_on_error (integer) #

If greater than zero, each bind parameter value reported in error messages is trimmed to this many bytes. Zero (the default) disables including bind parameters in error messages. -1 allows bind parameters to be printed in full. If this value is specified without units, it is taken as bytes.

Non-zero values of this setting add overhead, as PostgreSQL will need to store textual representations of parameter values in memory at the start of each statement, whether or not an error eventually occurs. The overhead is greater when bind parameters are sent in binary form than when they are sent as text, since the former case requires data conversion while the latter only requires copying the string.

log_statement (enum) #

Controls which SQL statements are logged. Valid values are none (off), ddl, mod, and all (all statements). ddl logs all data definition statements, such as CREATE, ALTER, and DROP statements. mod logs all ddl statements, plus data-modifying statements such as INSERT, UPDATE, DELETE, TRUNCATE, and COPY FROM. PREPARE, EXECUTE, and EXPLAIN ANALYZE statements are also logged if their contained command is of an appropriate type. For clients using extended query protocol, logging occurs when an Execute message is received, and values of the Bind parameters are included (with any embedded single-quote marks doubled).

The default is none. Only superusers and users with the appropriate SET privilege can change this setting.

Note

Statements that contain simple syntax errors are not logged even by the log_statement = all setting, because the log message is emitted only after basic parsing has been done to determine the statement type. In the case of extended query protocol, this setting likewise does not log statements that fail before the Execute phase (i.e., during parse analysis or planning). Set log_min_error_statement to ERROR (or lower) to log such statements.

Logged statements might reveal sensitive data and even contain plaintext passwords.

log_replication_commands (boolean) #

Causes each replication command to be logged in the server log. See Section 55.4 for more information about replication command. The default value is off. Only superusers and users with the appropriate SET privilege can change this setting.

log_temp_files (integer) #

Controls logging of temporary file names and sizes. Temporary files can be created for sorts, hashes, and temporary query results. If enabled by this setting, a log entry is emitted for each temporary file, with the file size specified in bytes, when it is deleted. A value of zero logs all temporary file information, while positive values log only files whose size is greater than or equal to the specified amount of data. If this value is specified without units, it is taken as kilobytes. The default setting is -1, which disables such logging. Only superusers and users with the appropriate SET privilege can change this setting.

log_timezone (string) #

Sets the time zone used for timestamps written in the server log. Unlike TimeZone, this value is cluster-wide, so that all sessions will report timestamps consistently. The built-in default is GMT, but that is typically overridden in postgresql.conf; initdb will install a setting there corresponding to its system environment. See Section 8.5.3 for more information. This parameter can only be set in the postgresql.conf file or on the server command line.

20.8.4. Using CSV-Format Log Output #

Including csvlog in the log_destination list provides a convenient way to import log files into a database table. This option emits log lines in comma-separated-values (CSV) format, with these columns: time stamp with milliseconds, user name, database name, process ID, client host:port number, session ID, per-session line number, command tag, session start time, virtual transaction ID, regular transaction ID, error severity, SQLSTATE code, error message, error message detail, hint, internal query that led to the error (if any), character count of the error position therein, error context, user query that led to the error (if any and enabled by log_min_error_statement), character count of the error position therein, location of the error in the PostgreSQL source code (if log_error_verbosity is set to verbose), application name, backend type, process ID of parallel group leader, and query id. Here is a sample table definition for storing CSV-format log output:

CREATE TABLE postgres_log
(
  log_time timestamp(3) with time zone,
  user_name text,
  database_name text,
  process_id integer,
  connection_from text,
  session_id text,
  session_line_num bigint,
  command_tag text,
  session_start_time timestamp with time zone,
  virtual_transaction_id text,
  transaction_id bigint,
  error_severity text,
  sql_state_code text,
  message text,
  detail text,
  hint text,
  internal_query text,
  internal_query_pos integer,
  context text,
  query text,
  query_pos integer,
  location text,
  application_name text,
  backend_type text,
  leader_pid integer,
  query_id bigint,
  PRIMARY KEY (session_id, session_line_num)
);

To import a log file into this table, use the COPY FROM command:

COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;

It is also possible to access the file as a foreign table, using the supplied file_fdw module.

There are a few things you need to do to simplify importing CSV log files:

  1. Set log_filename and log_rotation_age to provide a consistent, predictable naming scheme for your log files. This lets you predict what the file name will be and know when an individual log file is complete and therefore ready to be imported.

  2. Set log_rotation_size to 0 to disable size-based log rotation, as it makes the log file name difficult to predict.

  3. Set log_truncate_on_rotation to on so that old log data isn’t mixed with the new in the same file.

  4. The table definition above includes a primary key specification. This is useful to protect against accidentally importing the same information twice. The COPY command commits all of the data it imports at one time, so any error will cause the entire import to fail. If you import a partial log file and later import the file again when it is complete, the primary key violation will cause the import to fail. Wait until the log is complete and closed before importing. This procedure will also protect against accidentally importing a partial line that hasn’t been completely written, which would also cause COPY to fail.

20.8.5. Using JSON-Format Log Output #

Including jsonlog in the log_destination list provides a convenient way to import log files into many different programs. This option emits log lines in JSON format.

String fields with null values are excluded from output. Additional fields may be added in the future. User applications that process jsonlog output should ignore unknown fields.

Each log line is serialized as a JSON object with the set of keys and their associated values shown in Table 20.3.

Table 20.3. Keys and Values of JSON Log Entries

Key name Type Description
timestamp string Time stamp with milliseconds
user string User name
dbname string Database name
pid number Process ID
remote_host string Client host
remote_port number Client port
session_id string Session ID
line_num number Per-session line number
ps string Current ps display
session_start string Session start time
vxid string Virtual transaction ID
txid string Regular transaction ID
error_severity string Error severity
state_code string SQLSTATE code
message string Error message
detail string Error message detail
hint string Error message hint
internal_query string Internal query that led to the error
internal_position number Cursor index into internal query
context string Error context
statement string Client-supplied query string
cursor_position number Cursor index into query string
func_name string Error location function name
file_name string File name of error location
file_line_num number File line number of the error location
application_name string Client application name
backend_type string Type of backend
leader_pid number Process ID of leader for active parallel workers
query_id number Query ID

20.8.6. Process Title #

These settings control how process titles of server processes are modified. Process titles are typically viewed using programs like ps or, on Windows, Process Explorer. See Section 28.1 for details.

cluster_name (string) #

Sets a name that identifies this database cluster (instance) for various purposes. The cluster name appears in the process title for all server processes in this cluster. Moreover, it is the default application name for a standby connection (see synchronous_standby_names.)

The name can be any string of less than NAMEDATALEN characters (64 characters in a standard build). Only printable ASCII characters may be used in the cluster_name value. Other characters will be replaced with question marks (?). No name is shown if this parameter is set to the empty string '' (which is the default). This parameter can only be set at server start.

update_process_title (boolean) #

Enables updating of the process title every time a new SQL command is received by the server. This setting defaults to on on most platforms, but it defaults to off on Windows due to that platform’s larger overhead for updating the process title. Only superusers and users with the appropriate SET privilege can change this setting.

Как включить журналы базы данных

Время на прочтение
11 мин

Количество просмотров 15K

PostgreSQL — это система управления реляционными базами данных с открытым исходным кодом, которая используется в непрерывной разработке и продакшне уже 30 лет. Почти все крупные технологические компании используют PostgreSQL, поскольку это одна из самых надежных, проверенных в боях систем реляционных баз данных на сегодняшний день.

PostgreSQL является критически важной точкой в вашей инфраструктуре, поскольку в ней хранятся все данные. Для этого важна наглядность, а значит, вы должны понимать, как работает протоколирование в PostgreSQL. Это достигается с помощью журналов и метрик, которые предоставляет PostgreSQL.

В этой статье я объясню все, что вам нужно знать о журналах (логах) PostgreSQL, начиная с того, как их включить и заканчивая тем, как их легко форматировать и анализировать.

Что такое журналы PostgreSQL?

Журналы PostgreSQL — текстовые файлы, в которых отображается информация о том, что в данный момент происходит в вашей системе баз данных. Это включает в себя сведения о том, кто имеет доступ и к какому компоненту, какие ошибки произошли, что изменилось в настройках, какие запросы находятся в процессе выполнения и какие транзакции выполняются.

Чтобы получить общую картину всех журналов, вы можете разместить их  централизованно, а затем организовать возможность поиска среди них. Парсинг позволяет извлекать важную информацию и метрики, которые затем можно нанести на график для лучшей визуализации в виде точек.

В этой статье мы покажем вам, как изменить настройки PostgreSQL с помощью файла конфигурации и интерфейса командной строки. Рекомендуется вносить все эти изменения исключительно с помощью файла конфигурации, иначе ваши изменения могут быть потеряны при перезагрузке сервера.

Местоположение журнала PostgreSQL

Из коробки PostgreSQL будет показывать журналы в stderr, что не очень удобно, так как они будут смешиваться с другими процессами, которые также ведут логирование в stderr. Чтобы PostgreSQL мог создавать собственные логи, необходимо включить параметр logging_collector. Когда вы это сделаете, журналы начнут отправляться в стандартное местоположение, определенное вашей ОС. Ниже приведены директории журналов по умолчанию для нескольких различных операционных систем:

  • Система на базе Debian: e/var/log/postgresql/postgresql-x.x.main.log. X.x.

  • Система на базе Red Hat: /var/lib/pgsql/data/pg_log

  • Windows: C:\Program Files\PostgreSQL\9.3\data\pg_log

Чтобы поменять место хранения файлов журнала при включенном сборщике (коллекторе) логов, вы можете использовать параметр log_directory для указания пользовательского каталога.

Обратите внимание, что иногда ведение журнала может быть затруднительно в PostgreSQL. Коллектор логов не позволит потерять ни одного сообщения журнала, поэтому при высокой нагрузке он может блокировать процессы сервера, что приведет к проблемам. Вместо него можно использовать системный журнал (syslog), так как он позволяет отбрасывать некоторые сообщения и не блокирует систему. Чтобы отключить коллектор логов, вы можете настроить опцию на off:

logging_collector off

В зависимости от условий использования, вам может понадобиться изменить местоположение журналов PostgreSQL. Обычно используются такие варианты, как запись в syslog, CSV, Windows Event и Docker, о которых речь пойдет ниже.

Syslog

Вы можете легко настроить PostgreSQL на ведение журнала в syslog. Вам нужно сделать это на демоне syslog с помощью следующей конфигурации:

local0.* /var/log/postgresql

Вы можете использовать такие параметры, как syslog_facility, syslog_indent, syslog_sequence_number в конфигурационном файле PostgreSQL для форматирования логов.

CSV лог

Если вы хотите загрузить журналы в инструмент анализа или программу, можно сохранить их в CSV-файл. CSV хорошо определен, что делает этот процесс простым. Для перевода журналов в CSV необходимо добавить следующую строку в конфигурацию PostgreSQL:

csvlog /log/postgresql.csv

Вы также можете создать таблицу дополнительно к журналам, а затем использовать SQL для запроса по определенным условиям.

Журнал событий Windows

Для систем PostgreSQL, работающих под Windows, вы можете отправлять логи в журнал событий (event) Windows, используя следующую конфигурацию:

log_destination = 'stderr, eventlog'

Обязательно зарегистрируйте систему источника событий в ОС Windows, чтобы она могла получать и показывать вам сообщения журнала событий с помощью программы просмотра событий Windows. Для этого выполните команду:

regsvr32 pgsql_library_directory/pgevent.dll

Docker

В настоящее время многие инструменты и базы данных запускаются как Docker-приложения, включая PostgreSQL. Вы также можете легко запустить Docker-версию PostgreSQL на Kubernetes или любой другой платформе оркестрации контейнеров. Однако в таких случаях не стоит вносить изменения непосредственно в поды или контейнеры, поскольку эти изменения могут быть потеряны при перезапуске подов. Вместо этого необходимо передавать конфигурации во время запуска этих контейнеров.

Чтобы включить логирование, необходимо передать конфигурации с помощью ConfigMaps в Kubernetes. Читайте этот блог, чтобы развернуть PostgreSQL на Kubernetes и включить/выключить различные настройки.

Что важно записывать в журнал?

Запись большого количества информации в журнал может привести к пустой трате времени, если вы не сможете определить, какие логи важны, а какие нет. Очень важно уменьшить шум в журналах, чтобы ускорить отладку — это также сэкономит ваше время и ресурсы для их хранения.

Журналы должны показать вам медленные запросы, уровни логов и то, как отловить критическую информацию с минимальным объемом логирования. Этого можно добиться с помощью фильтров, наиболее распространенными из которых являются пороговые значения журнала, его уровни, время выполнения оператора и семплинг. Давайте немного углубимся в каждый из них.

Пороговые значения для медленного запроса

PostgreSQL может регистрировать запросы, которые занимают больше времени, чем определенный порог. Определение медленных запросов в журнале помогает обнаружить проблемы с базой данных и причины задержек в работе вашего приложения.

Чтобы включить эту функцию, необходимо отредактировать файл postgresql.conf. Найдите строку log_min_duration_statement и настройте ее в соответствии с вашими потребностями. Например, приведенный ниже оператор будет регистрировать все запросы, которые выполняются более 1 секунды:

log_min_duration_statement = 1000

После этого сохраните файл и перезагрузите PostgreSQL. Ваши настройки будут применены, и вы сможете увидеть логи медленных запросов в файлах журнала PostgreSQL.

Вы также можете задать эти параметры динамически с помощью интерфейса запросов PostgreSQL, выполнив следующую команду:

ALTER DATABASE db SET log_min_duration_statement = ‘1000ms';

Время выполнения оператора

Вы можете легко регистрировать продолжительность выполнения каждого оператора в PostgreSQL. Для этого добавьте приведенный ниже оператор в свою конфигурацию, чтобы включить протоколирование каждого оператора:

log_statement all

Другим вариантом решения этой задачи является запуск следующего оператора PostgreSQL:

ALTER DATABASE db SET log_statement = ‘all';

Обратите внимание, что это включит логирование всех запрошенных операторов, что может оказаться не слишком полезным и попросту будет создавать много шума.

Вместо этого, возможно, вы захотите вести журнал по типу запроса, например, DDL или MOD. DDL состоит из операторов CREATE, ALTER и DROP, а MOD включает в себя DDL плюс другие операторы модификации.

Семплинг

При включенном семплинге вы можете записывать в журнал примеры операторов, которые переходят определенный порог. Если ваш сервер генерирует огромное количество журналов в связи с различными событиями, то вы не захотите регистрировать все, что выходит за порог. Можно делать это выборочно. Это помогает поддерживать меньший объем ввода/вывода при логировании и уменьшить шум в журналах, что облегчает определение того, какие типы операторов вызывают проблему.

Этими порогами и выборкой можно управлять с помощью таких параметров в файле postgresql.conf, как log_min_duration_sample, log_statement_sample_rate и log_transaction_sample_rate. Обратитесь к документации PostgreSQL, чтобы узнать, как правильно их использовать. У вас также есть возможность внести данные изменения через командную строку PostgreSQL.

Обратите внимание, что подобное решение может стать неожиданной ловушкой, так как в результате семплинга можно пропустить единственный оператор, создающий проблему. В таких случаях вы не сможете найти причину ошибки, и отладка займет больше времени, чем обычно.

Уровни журналов PostgreSQL

PostgreSQL предлагает несколько уровней оповещения журнала в зависимости от серьезности события. Вы можете изменить уровень журнала PostgreSQL с помощью параметра log_min_error_statement в конфигурационном файле PostgreSQL, выбрав любой из приведенных ниже:

  • DEBUG1, DEBUG2, DEBUG3… DEBUG5: Предоставляет разработчикам более подробную информацию.

  • INFO: Извлекает конкретные данные, запрошенные пользователем, подобно вербозному выводу

  • NOTICE: Предлагает пользователям полезную информацию, например, об усечении идентификатора.

  • WARNING: Выдает предупреждения о вероятных проблемах

  • ERROR: Регистрирует ошибки, включая те, которые вызывают прерывание любой команды.

  • LOG: Регистрирует данные, например, активность контрольных точек, что может быть полезно для администратора.

  • FATAL: Возникает при ошибках, которые привели к прерыванию текущего сеанса работы.

  • PANIC: Возникает при ошибках, которые приводят к прерыванию всех сеансов базы данных.

Если вы отправляете журналы в Windows eventlog (журнал событий) или syslog (системный журнал), уровень серьезности (log-severity) журнала будет изменен следующим образом:

  • DEBUG1… DEBUG5 будут преобразованы в DEBUG в syslog и INFORMATION в eventlog.

  • INFO будет INFO в syslog и INFORMATION в eventlog.

  • NOTICE будет NOTICE в syslog и INFORMATION в eventlog.

  • WARNING будет NOTICE в syslog и WARNING в eventlog.

  • ERROR будет WARNING в syslog и ERROR в eventlog.

  • LOG будет INFO в syslog и INFORMATION в eventlog.

  • FATAL будет ERR в syslog и ERROR в eventlog.

  • PANIC будет CRIT в syslog и ERROR в eventlog.

Помимо уровней, очень важно понимать, какие типы журналов генерируются PostgreSQL. Это поможет вам понять, какие именно журналы следует просматривать при возникновении определенной проблемы.

Типы журналов

Существует несколько типов журналов PostgreSQL, которые необходимо учитывать при дебаггинге Их можно разделить на два типа: журналы для администратора, и журналы для пользователя приложения.

Журналы, специфичные для администратора, помогают управлять сервером PostgreSQL. Если сервер работает неправильно, они могут указать причину этого и помочь в устранении неполадок.

Существует два типа журналов, специфичных для администратора:

  • Журналы запуска: Здесь отображаются все важные события и любые проблемы (например, связанные с неправильной конфигурацией) в процессе запуска вашего сервера PostgreSQL.

  • Журналы сервера: Они помогут вам определить, что происходит с сервером PostgreSQL во время работы с точки зрения администратора. Они располагаются в стандартном месте вашей инсталляции или в том месте, которое вы указали в конфигурационном файле PostgreSQL.

Когда речь заходит о журналах, специфичных для пользователей приложений, следует обратить внимание на такие важные журналы PostgreSQL:

  • Журналы запросов показывают все запросы, которые были сделаны на сервере; вы можете увидеть зарегистрированные запросы, если у вас включен log_statement.

  • Журналы транзакций — это записи всех событий, происходящих с базой данных; они соответствуют стандарту WAL (write ahead log), который не предназначен для чтения человеком. WAL — это способ хранения записей всех действий, выполняемых с базой данных, и может быть использован для восстановления после аварии. Плагин pg_receivexlog может отображать журналы транзакций, передаваемые вашим сервером PostgreSQL.

  • Журналы соединений полезны для выявления любых нежелательных подключений к серверу. Вы можете включить log_connections в файле postgresql.conf для фиксации каждой попытки подключения к серверу; log_disconnections позволит вам увидеть всех клиентов, которые отключились от сервера.

  • Журналы ошибок помогут вам определить, создают ли какие-либо из ваших запросов нежелательные проблемы на сервере; log_in_error_statement управляет уровнем важности регистрации сообщений об ошибках.

  • Журналы аудита и доступа очень важны с точки зрения администратора. Первые показывают изменения, внесенные в базу данных, а вторые определяют, кто какие запросы делал; их можно включить с помощью конфигурации log_statement или плагина PostgreSQL, например, pgAudit.

Большинство из этих типов журналов находятся в стандартных местах хранения по умолчанию или там, куда вы их определите в файле postgresql.conf. Есть также несколько проектов с открытым исходным кодом, которые я люблю использовать вместе с PostgreSQL для лучшего анализа журналов, например pgBadger.

Просто вести журнал — это еще не все случаи. Вам также нужно подумать о том, как вы будете архивировать или выполнять ротацию журналов. PostgreSQL поддерживает ротацию журналов, о которой пойдет речь в следующем разделе.

Ротация журналов PostgreSQL

PostgreSQL может выполнять ротацию журналов с помощью некоторых базовых параметров конфигурации. Благодаря таким параметрам, как log_rotation_age, log_rotation_size и log_truncate_on_rotation, вы можете легко настроить, в какой момент вы хотите произвести ротацию журналов. Например:

log_rotation_age 60 #default unit is minutes, this will rotate logs every
log_rotation_age 300 #rotate the logs after the time mentioned.

Вы также можете использовать CLI для настройки этой конфигурации.

Как уже говорилось, изучение журналов — необходимый шаг в выявлении проблем, а для этого нужно понимать форматирование журналов. В PostgreSQL вы можете легко определить формат журнала в соответствии с вашими потребностями.

Как форматировать журналы

В PostgreSQL есть возможность использовать формат CSV и сгенерировать CSV файл, который можно использовать для добавления логов в таблице, и в дополнение к всему использовать SQL.

Кроме того, параметр log_line_prefix позволяет форматировать начало каждой строки журнала в файле postgresql.conf или через командную строку. Настраиваемые параметры включают имя приложения, имя пользователя, имя базы данных, удаленный хост, тип бэкенда, идентификатор процесса и т.д. Весь список параметров доступен в документации PostgreSQL. Например:

log_line_prefix = '%m [%p] %u@%d/%a '

Приведенный префикс log_line_prefix означает, что журналы будут начинаться со времени в миллисекундах, затем идентификатор процесса, имя пользователя, имя базы данных и имя приложения.

Форматирование журнала, пороговые значения, семплинг, уровни и типы журналов — все это поможет вам в процессе дебаггинга. Но в идеале вам нужен инструмент, который позволяет агрегировать и анализировать все эти журналы и просматривать результаты через одну панель, а не заходить на каждый сервер. Одним из таких инструментов является Sematext. Давайте рассмотрим, как вы можете получить преимущества от ведения журналов PostgreSQL с помощью Sematext.

Ведение журналов PostgreSQL с помощью Sematext

Ведение журналов PostgreSQL с помощью Sematext

Ведение журналов PostgreSQL с помощью Sematext

Sematext Logs — это решение для управления журналами и их мониторинга, которое позволяет вам объединять журналы из различных источников данных вашей инфраструктуры в одном месте для просмотра и анализа.

Sematext обладает функцией автоматического обнаружения сервисов, поэтому вам просто нужно установить агент Sematext на свои серверы, выполнить базовую настройку, и ваши журналы PostgreSQL начнут в него стекаться. Они будут представлены на интуитивно понятной, готовой к использованию информационной панели (дашборд). Вы даже можете легко создать собственный дашборд, настроить оповещения и отправлять их по различным каналам передачи уведомлений, таким как Gmail, Slack или PagerDuty.

Sematext также предлагает такие функции, как обнаружение аномалий, что помогает вам заранее выявлять проблемы, а затем принимать меры для их предотвращения. Для более глубокого понимания вы можете сопоставить журналы PostgreSQL с метриками PostgreSQL, чтобы быстрее обнаружить узкие места. Таким образом, вы получаете вид с высоты птичьего полета на ваши установки PostgreSQL, что облегчает поиск и отладку неисправностей.

Sematext Logs (журналы) являются частью Sematext Cloud (облако), полнофункционального решения для ведения журналов и мониторинга, которое позволяет вам получить возможность обзора и интеграции всей вашей ИТ-среды. Помимо баз данных, оно поддерживает интеграцию с широким спектром инструментов, включая HAProxy, Apache Tomcat, JVM и Kubernetes. Кроме того, вы получаете поддержку деплоя Kubernetes, поэтому вам будет проще контролировать свою установку в среде Kubernetes.

Заключение

Следить за журналами PostgreSQL — важная часть работы по устранению неполадок в базе данных. Понимая, как делаются запросы и выполнятся операторы, а также трафик, соединения, ошибки и другие изменения или события на вашем сервере, вы можете легко докопаться до проблематичных процессов и обнаружить первопричину затруднений с производительностью.

Вы можете отслеживать журналы различными способами, например, используя less или tail для файлов журналов, но это будет сложно сделать, если журналы разбросаны по нескольким файлам и машинам. Вам нужны журналы в одном месте, и такое решение, как Sematext Logs, может помочь вам достичь этого.

В преддверии старта курса «Observability — мониторинг, логирование, трейсинг» хотим порекомендовать два абсолютно бесплатных вебинара от OTUS, регистрация на которые доступна по ссылкам ниже.

  • Отказоустойчивый Prometheus. Thanos, VictoriaMetrics

  • Организация мониторинга с помощью Grafana stack

log_destination (string)

PostgreSQL supports several methods
for logging server messages, including
stderr, csvlog and
syslog. On Windows,
eventlog is also supported. Set this
parameter to a list of desired log destinations separated by
commas. The default is to log to stderr
only.
This parameter can only be set in the postgresql.conf
file or on the server command line.

If csvlog is included in log_destination,
log entries are output in comma separated
value
(CSV) format, which is convenient for
loading logs into programs.
See Section 19.8.4 for details.
logging_collector must be enabled to generate
CSV-format log output.

When either stderr or
csvlog are included, the file
current_logfiles is created to record the location
of the log file(s) currently in use by the logging collector and the
associated logging destination. This provides a convenient way to
find the logs currently in use by the instance. Here is an example of
this file’s content:

stderr log/postgresql.log
csvlog log/postgresql.csv

current_logfiles is recreated when a new log file
is created as an effect of rotation, and
when log_destination is reloaded. It is removed when
neither stderr
nor csvlog are included
in log_destination, and when the logging collector is
disabled.

Note

On most Unix systems, you will need to alter the configuration of
your system’s syslog daemon in order
to make use of the syslog option for
log_destination. PostgreSQL
can log to syslog facilities
LOCAL0 through LOCAL7 (see syslog_facility), but the default
syslog configuration on most platforms
will discard all such messages. You will need to add something like:

local0.*    /var/log/postgresql

to the syslog daemon’s configuration file
to make it work.

On Windows, when you use the eventlog
option for log_destination, you should
register an event source and its library with the operating
system so that the Windows Event Viewer can display event
log messages cleanly.
See Section 18.11 for details.

logging_collector (boolean)

This parameter enables the logging collector, which
is a background process that captures log messages
sent to stderr and redirects them into log files.
This approach is often more useful than
logging to syslog, since some types of messages
might not appear in syslog output. (One common
example is dynamic-linker failure messages; another is error messages
produced by scripts such as archive_command.)
This parameter can only be set at server start.

Note

It is possible to log to stderr without using the
logging collector; the log messages will just go to wherever the
server’s stderr is directed. However, that method is
only suitable for low log volumes, since it provides no convenient
way to rotate log files. Also, on some platforms not using the
logging collector can result in lost or garbled log output, because
multiple processes writing concurrently to the same log file can
overwrite each other’s output.

Note

The logging collector is designed to never lose messages. This means
that in case of extremely high load, server processes could be
blocked while trying to send additional log messages when the
collector has fallen behind. In contrast, syslog
prefers to drop messages if it cannot write them, which means it
may fail to log some messages in such cases but it will not block
the rest of the system.

log_directory (string)

When logging_collector is enabled,
this parameter determines the directory in which log files will be created.
It can be specified as an absolute path, or relative to the
cluster data directory.
This parameter can only be set in the postgresql.conf
file or on the server command line.
The default is log.

log_filename (string)

When logging_collector is enabled,
this parameter sets the file names of the created log files. The value
is treated as a strftime pattern,
so %-escapes can be used to specify time-varying
file names. (Note that if there are
any time-zone-dependent %-escapes, the computation
is done in the zone specified
by log_timezone.)
The supported %-escapes are similar to those
listed in the Open Group’s strftime
specification.
Note that the system’s strftime is not used
directly, so platform-specific (nonstandard) extensions do not work.
The default is postgresql-%Y-%m-%d_%H%M%S.log.

If you specify a file name without escapes, you should plan to
use a log rotation utility to avoid eventually filling the
entire disk. In releases prior to 8.4, if
no % escapes were
present, PostgreSQL would append
the epoch of the new log file’s creation time, but this is no
longer the case.

If CSV-format output is enabled in log_destination,
.csv will be appended to the timestamped
log file name to create the file name for CSV-format output.
(If log_filename ends in .log, the suffix is
replaced instead.)

This parameter can only be set in the postgresql.conf
file or on the server command line.

log_file_mode (integer)

On Unix systems this parameter sets the permissions for log files
when logging_collector is enabled. (On Microsoft
Windows this parameter is ignored.)
The parameter value is expected to be a numeric mode
specified in the format accepted by the
chmod and umask
system calls. (To use the customary octal format the number
must start with a 0 (zero).)

The default permissions are 0600, meaning only the
server owner can read or write the log files. The other commonly
useful setting is 0640, allowing members of the owner’s
group to read the files. Note however that to make use of such a
setting, you’ll need to alter log_directory to
store the files somewhere outside the cluster data directory. In
any case, it’s unwise to make the log files world-readable, since
they might contain sensitive data.

This parameter can only be set in the postgresql.conf
file or on the server command line.

log_rotation_age (integer)

When logging_collector is enabled,
this parameter determines the maximum lifetime of an individual log file.
After this many minutes have elapsed, a new log file will
be created. Set to zero to disable time-based creation of
new log files.
This parameter can only be set in the postgresql.conf
file or on the server command line.

log_rotation_size (integer)

When logging_collector is enabled,
this parameter determines the maximum size of an individual log file.
After this many kilobytes have been emitted into a log file,
a new log file will be created. Set to zero to disable size-based
creation of new log files.
This parameter can only be set in the postgresql.conf
file or on the server command line.

log_truncate_on_rotation (boolean)

When logging_collector is enabled,
this parameter will cause PostgreSQL to truncate (overwrite),
rather than append to, any existing log file of the same name.
However, truncation will occur only when a new file is being opened
due to time-based rotation, not during server startup or size-based
rotation. When off, pre-existing files will be appended to in
all cases. For example, using this setting in combination with
a log_filename like postgresql-%H.log
would result in generating twenty-four hourly log files and then
cyclically overwriting them.
This parameter can only be set in the postgresql.conf
file or on the server command line.

Example: To keep 7 days of logs, one log file per day named
server_log.Mon, server_log.Tue,
etc, and automatically overwrite last week’s log with this week’s log,
set log_filename to server_log.%a,
log_truncate_on_rotation to on, and
log_rotation_age to 1440.

Example: To keep 24 hours of logs, one log file per hour, but
also rotate sooner if the log file size exceeds 1GB, set
log_filename to server_log.%H%M,
log_truncate_on_rotation to on,
log_rotation_age to 60, and
log_rotation_size to 1000000.
Including %M in log_filename allows
any size-driven rotations that might occur to select a file name
different from the hour’s initial file name.

syslog_facility (enum)

When logging to syslog is enabled, this parameter
determines the syslog
facility to be used. You can choose
from LOCAL0, LOCAL1,
LOCAL2, LOCAL3, LOCAL4,
LOCAL5, LOCAL6, LOCAL7;
the default is LOCAL0. See also the
documentation of your system’s
syslog daemon.
This parameter can only be set in the postgresql.conf
file or on the server command line.

syslog_ident (string)

When logging to syslog is enabled, this parameter
determines the program name used to identify
PostgreSQL messages in
syslog logs. The default is
postgres.
This parameter can only be set in the postgresql.conf
file or on the server command line.

syslog_sequence_numbers (boolean)

When logging to syslog and this is on (the
default), then each message will be prefixed by an increasing
sequence number (such as [2]). This circumvents
the — last message repeated N times — suppression
that many syslog implementations perform by default. In more modern
syslog implementations, repeated message suppression can be configured
(for example, $RepeatedMsgReduction
in rsyslog), so this might not be
necessary. Also, you could turn this off if you actually want to
suppress repeated messages.

This parameter can only be set in the postgresql.conf
file or on the server command line.

syslog_split_messages (boolean)

When logging to syslog is enabled, this parameter
determines how messages are delivered to syslog. When on (the
default), messages are split by lines, and long lines are split so
that they will fit into 1024 bytes, which is a typical size limit for
traditional syslog implementations. When off, PostgreSQL server log
messages are delivered to the syslog service as is, and it is up to
the syslog service to cope with the potentially bulky messages.

If syslog is ultimately logging to a text file, then the effect will
be the same either way, and it is best to leave the setting on, since
most syslog implementations either cannot handle large messages or
would need to be specially configured to handle them. But if syslog
is ultimately writing into some other medium, it might be necessary or
more useful to keep messages logically together.

This parameter can only be set in the postgresql.conf
file or on the server command line.

event_source (string)

When logging to event log is enabled, this parameter
determines the program name used to identify
PostgreSQL messages in
the log. The default is PostgreSQL.
This parameter can only be set in the postgresql.conf
file or on the server command line.

20.8.1. Where to Log

log_destination (string)

PostgreSQL supports several methods for logging server messages, including stderr, csvlog and syslog. On Windows, eventlog is also supported. Set this parameter to a list of desired log destinations separated by commas. The default is to log to stderr only. This parameter can only be set in the postgresql.conf file or on the server command line.

If csvlog is included in log_destination, log entries are output in comma separated value (CSV) format, which is convenient for loading logs into programs. See Section 20.8.4 for details. logging_collector must be enabled to generate CSV-format log output.

When either stderr or csvlog are included, the file current_logfiles is created to record the location of the log file(s) currently in use by the logging collector and the associated logging destination. This provides a convenient way to find the logs currently in use by the instance. Here is an example of this file’s content:

stderr log/postgresql.log
csvlog log/postgresql.csv

current_logfiles is recreated when a new log file is created as an effect of rotation, and when log_destination is reloaded. It is removed when neither stderr nor csvlog are included in log_destination, and when the logging collector is disabled.

Note

On most Unix systems, you will need to alter the configuration of your system’s syslog daemon in order to make use of the syslog option for log_destination. PostgreSQL can log to syslog facilities LOCAL0 through LOCAL7 (see syslog_facility), but the default syslog configuration on most platforms will discard all such messages. You will need to add something like:

local0.*    /var/log/postgresql

to the syslog daemon’s configuration file to make it work.

On Windows, when you use the eventlog option for log_destination, you should register an event source and its library with the operating system so that the Windows Event Viewer can display event log messages cleanly. See Section 19.12 for details.

logging_collector (boolean)

This parameter enables the logging collector, which is a background process that captures log messages sent to stderr and redirects them into log files. This approach is often more useful than logging to syslog, since some types of messages might not appear in syslog output. (One common example is dynamic-linker failure messages; another is error messages produced by scripts such as archive_command.) This parameter can only be set at server start.

Note

It is possible to log to stderr without using the logging collector; the log messages will just go to wherever the server’s stderr is directed. However, that method is only suitable for low log volumes, since it provides no convenient way to rotate log files. Also, on some platforms not using the logging collector can result in lost or garbled log output, because multiple processes writing concurrently to the same log file can overwrite each other’s output.

Note

The logging collector is designed to never lose messages. This means that in case of extremely high load, server processes could be blocked while trying to send additional log messages when the collector has fallen behind. In contrast, syslog prefers to drop messages if it cannot write them, which means it may fail to log some messages in such cases but it will not block the rest of the system.

log_directory (string)

When logging_collector is enabled, this parameter determines the directory in which log files will be created. It can be specified as an absolute path, or relative to the cluster data directory. This parameter can only be set in the postgresql.conf file or on the server command line. The default is log.

log_filename (string)

When logging_collector is enabled, this parameter sets the file names of the created log files. The value is treated as a strftime pattern, so %-escapes can be used to specify time-varying file names. (Note that if there are any time-zone-dependent %-escapes, the computation is done in the zone specified by log_timezone.) The supported %-escapes are similar to those listed in the Open Group’s strftime specification. Note that the system’s strftime is not used directly, so platform-specific (nonstandard) extensions do not work. The default is postgresql-%Y-%m-%d_%H%M%S.log.

If you specify a file name without escapes, you should plan to use a log rotation utility to avoid eventually filling the entire disk. In releases prior to 8.4, if no % escapes were present, PostgreSQL would append the epoch of the new log file’s creation time, but this is no longer the case.

If CSV-format output is enabled in log_destination, .csv will be appended to the timestamped log file name to create the file name for CSV-format output. (If log_filename ends in .log, the suffix is replaced instead.)

This parameter can only be set in the postgresql.conf file or on the server command line.

log_file_mode (integer)

On Unix systems this parameter sets the permissions for log files when logging_collector is enabled. (On Microsoft Windows this parameter is ignored.) The parameter value is expected to be a numeric mode specified in the format accepted by the chmod and umask system calls. (To use the customary octal format the number must start with a 0 (zero).)

The default permissions are 0600, meaning only the server owner can read or write the log files. The other commonly useful setting is 0640, allowing members of the owner’s group to read the files. Note however that to make use of such a setting, you’ll need to alter log_directory to store the files somewhere outside the cluster data directory. In any case, it’s unwise to make the log files world-readable, since they might contain sensitive data.

This parameter can only be set in the postgresql.conf file or on the server command line.

log_rotation_age (integer)

When logging_collector is enabled, this parameter determines the maximum amount of time to use an individual log file, after which a new log file will be created. If this value is specified without units, it is taken as minutes. The default is 24 hours. Set to zero to disable time-based creation of new log files. This parameter can only be set in the postgresql.conf file or on the server command line.

log_rotation_size (integer)

When logging_collector is enabled, this parameter determines the maximum size of an individual log file. After this amount of data has been emitted into a log file, a new log file will be created. If this value is specified without units, it is taken as kilobytes. The default is 10 megabytes. Set to zero to disable size-based creation of new log files. This parameter can only be set in the postgresql.conf file or on the server command line.

log_truncate_on_rotation (boolean)

When logging_collector is enabled, this parameter will cause PostgreSQL to truncate (overwrite), rather than append to, any existing log file of the same name. However, truncation will occur only when a new file is being opened due to time-based rotation, not during server startup or size-based rotation. When off, pre-existing files will be appended to in all cases. For example, using this setting in combination with a log_filename like postgresql-%H.log would result in generating twenty-four hourly log files and then cyclically overwriting them. This parameter can only be set in the postgresql.conf file or on the server command line.

Example: To keep 7 days of logs, one log file per day named server_log.Mon, server_log.Tue, etc, and automatically overwrite last week’s log with this week’s log, set log_filename to server_log.%a, log_truncate_on_rotation to on, and log_rotation_age to 1440.

Example: To keep 24 hours of logs, one log file per hour, but also rotate sooner if the log file size exceeds 1GB, set log_filename to server_log.%H%M, log_truncate_on_rotation to on, log_rotation_age to 60, and log_rotation_size to 1000000. Including %M in log_filename allows any size-driven rotations that might occur to select a file name different from the hour’s initial file name.

syslog_facility (enum)

When logging to syslog is enabled, this parameter determines the syslog facility to be used. You can choose from LOCAL0, LOCAL1, LOCAL2, LOCAL3, LOCAL4, LOCAL5, LOCAL6, LOCAL7; the default is LOCAL0. See also the documentation of your system’s syslog daemon. This parameter can only be set in the postgresql.conf file or on the server command line.

syslog_ident (string)

When logging to syslog is enabled, this parameter determines the program name used to identify PostgreSQL messages in syslog logs. The default is postgres. This parameter can only be set in the postgresql.conf file or on the server command line.

syslog_sequence_numbers (boolean)

When logging to syslog and this is on (the default), then each message will be prefixed by an increasing sequence number (such as [2]). This circumvents the — last message repeated N times — suppression that many syslog implementations perform by default. In more modern syslog implementations, repeated message suppression can be configured (for example, $RepeatedMsgReduction in rsyslog), so this might not be necessary. Also, you could turn this off if you actually want to suppress repeated messages.

This parameter can only be set in the postgresql.conf file or on the server command line.

syslog_split_messages (boolean)

When logging to syslog is enabled, this parameter determines how messages are delivered to syslog. When on (the default), messages are split by lines, and long lines are split so that they will fit into 1024 bytes, which is a typical size limit for traditional syslog implementations. When off, PostgreSQL server log messages are delivered to the syslog service as is, and it is up to the syslog service to cope with the potentially bulky messages.

If syslog is ultimately logging to a text file, then the effect will be the same either way, and it is best to leave the setting on, since most syslog implementations either cannot handle large messages or would need to be specially configured to handle them. But if syslog is ultimately writing into some other medium, it might be necessary or more useful to keep messages logically together.

This parameter can only be set in the postgresql.conf file or on the server command line.

event_source (string)

When logging to event log is enabled, this parameter determines the program name used to identify PostgreSQL messages in the log. The default is PostgreSQL. This parameter can only be set in the postgresql.conf file or on the server command line.

20.8.2. When to Log

log_min_messages (enum)

Controls which message levels are written to the server log. Valid values are DEBUG5, DEBUG4, DEBUG3, DEBUG2, DEBUG1, INFO, NOTICE, WARNING, ERROR, LOG, FATAL, and PANIC. Each level includes all the levels that follow it. The later the level, the fewer messages are sent to the log. The default is WARNING. Note that LOG has a different rank here than in client_min_messages. Only superusers can change this setting.

log_min_error_statement (enum)

Controls which SQL statements that cause an error condition are recorded in the server log. The current SQL statement is included in the log entry for any message of the specified severity or higher. Valid values are DEBUG5, DEBUG4, DEBUG3, DEBUG2, DEBUG1, INFO, NOTICE, WARNING, ERROR, LOG, FATAL, and PANIC. The default is ERROR, which means statements causing errors, log messages, fatal errors, or panics will be logged. To effectively turn off logging of failing statements, set this parameter to PANIC. Only superusers can change this setting.

log_min_duration_statement (integer)

Causes the duration of each completed statement to be logged if the statement ran for at least the specified amount of time. For example, if you set it to 250ms then all SQL statements that run 250ms or longer will be logged. Enabling this parameter can be helpful in tracking down unoptimized queries in your applications. If this value is specified without units, it is taken as milliseconds. Setting this to zero prints all statement durations. -1 (the default) disables logging statement durations. Only superusers can change this setting.

This overrides log_min_duration_sample, meaning that queries with duration exceeding this setting are not subject to sampling and are always logged.

For clients using extended query protocol, durations of the Parse, Bind, and Execute steps are logged independently.

Note

When using this option together with log_statement, the text of statements that are logged because of log_statement will not be repeated in the duration log message. If you are not using syslog, it is recommended that you log the PID or session ID using log_line_prefix so that you can link the statement message to the later duration message using the process ID or session ID.

log_min_duration_sample (integer)

Allows sampling the duration of completed statements that ran for at least the specified amount of time. This produces the same kind of log entries as log_min_duration_statement, but only for a subset of the executed statements, with sample rate controlled by log_statement_sample_rate. For example, if you set it to 100ms then all SQL statements that run 100ms or longer will be considered for sampling. Enabling this parameter can be helpful when the traffic is too high to log all queries. If this value is specified without units, it is taken as milliseconds. Setting this to zero samples all statement durations. -1 (the default) disables sampling statement durations. Only superusers can change this setting.

This setting has lower priority than log_min_duration_statement, meaning that statements with durations exceeding log_min_duration_statement are not subject to sampling and are always logged.

Other notes for log_min_duration_statement apply also to this setting.

log_statement_sample_rate (floating point)

Determines the fraction of statements with duration exceeding log_min_duration_sample that will be logged. Sampling is stochastic, for example 0.5 means there is statistically one chance in two that any given statement will be logged. The default is 1.0, meaning to log all sampled statements. Setting this to zero disables sampled statement-duration logging, the same as setting log_min_duration_sample to -1. Only superusers can change this setting.

log_transaction_sample_rate (floating point)

Sets the fraction of transactions whose statements are all logged, in addition to statements logged for other reasons. It applies to each new transaction regardless of its statements’ durations. Sampling is stochastic, for example 0.1 means there is statistically one chance in ten that any given transaction will be logged. log_transaction_sample_rate can be helpful to construct a sample of transactions. The default is 0, meaning not to log statements from any additional transactions. Setting this to 1 logs all statements of all transactions. Only superusers can change this setting.

Note

Like all statement-logging options, this option can add significant overhead.

Table 20.2 explains the message severity levels used by PostgreSQL. If logging output is sent to syslog or Windows’ eventlog, the severity levels are translated as shown in the table.

Table 20.2. Message Severity Levels

Severity Usage syslog eventlog
DEBUG1 .. DEBUG5 Provides successively-more-detailed information for use by developers. DEBUG INFORMATION
INFO Provides information implicitly requested by the user, e.g., output from VACUUM VERBOSE. INFO INFORMATION
NOTICE Provides information that might be helpful to users, e.g., notice of truncation of long identifiers. NOTICE INFORMATION
WARNING Provides warnings of likely problems, e.g., COMMIT outside a transaction block. NOTICE WARNING
ERROR Reports an error that caused the current command to abort. WARNING ERROR
LOG Reports information of interest to administrators, e.g., checkpoint activity. INFO INFORMATION
FATAL Reports an error that caused the current session to abort. ERR ERROR
PANIC Reports an error that caused all database sessions to abort. CRIT ERROR

20.8.3. What to Log

Note

What you choose to log can have security implications; see Section 25.3.

application_name (string)

The application_name can be any string of less than NAMEDATALEN characters (64 characters in a standard build). It is typically set by an application upon connection to the server. The name will be displayed in the pg_stat_activity view and included in CSV log entries. It can also be included in regular log entries via the log_line_prefix parameter. Only printable ASCII characters may be used in the application_name value. Other characters will be replaced with question marks (?).

debug_print_parse (boolean)
debug_print_rewritten (boolean)
debug_print_plan (boolean)

These parameters enable various debugging output to be emitted. When set, they print the resulting parse tree, the query rewriter output, or the execution plan for each executed query. These messages are emitted at LOG message level, so by default they will appear in the server log but will not be sent to the client. You can change that by adjusting client_min_messages and/or log_min_messages. These parameters are off by default.

debug_pretty_print (boolean)

When set, debug_pretty_print indents the messages produced by debug_print_parse, debug_print_rewritten, or debug_print_plan. This results in more readable but much longer output than the compact format used when it is off. It is on by default.

log_autovacuum_min_duration (integer)

Causes each action executed by autovacuum to be logged if it ran for at least the specified amount of time. Setting this to zero logs all autovacuum actions. -1 (the default) disables logging autovacuum actions. If this value is specified without units, it is taken as milliseconds. For example, if you set this to 250ms then all automatic vacuums and analyzes that run 250ms or longer will be logged. In addition, when this parameter is set to any value other than -1, a message will be logged if an autovacuum action is skipped due to a conflicting lock or a concurrently dropped relation. Enabling this parameter can be helpful in tracking autovacuum activity. This parameter can only be set in the postgresql.conf file or on the server command line; but the setting can be overridden for individual tables by changing table storage parameters.

log_checkpoints (boolean)

Causes checkpoints and restartpoints to be logged in the server log. Some statistics are included in the log messages, including the number of buffers written and the time spent writing them. This parameter can only be set in the postgresql.conf file or on the server command line. The default is off.

log_connections (boolean)

Causes each attempted connection to the server to be logged, as well as successful completion of both client authentication (if necessary) and authorization. Only superusers can change this parameter at session start, and it cannot be changed at all within a session. The default is off.

Note

Some client programs, like psql, attempt to connect twice while determining if a password is required, so duplicate connection received messages do not necessarily indicate a problem.

log_disconnections (boolean)

Causes session terminations to be logged. The log output provides information similar to log_connections, plus the duration of the session. Only superusers can change this parameter at session start, and it cannot be changed at all within a session. The default is off.

log_duration (boolean)

Causes the duration of every completed statement to be logged. The default is off. Only superusers can change this setting.

For clients using extended query protocol, durations of the Parse, Bind, and Execute steps are logged independently.

Note

The difference between enabling log_duration and setting log_min_duration_statement to zero is that exceeding log_min_duration_statement forces the text of the query to be logged, but this option doesn’t. Thus, if log_duration is on and log_min_duration_statement has a positive value, all durations are logged but the query text is included only for statements exceeding the threshold. This behavior can be useful for gathering statistics in high-load installations.

log_error_verbosity (enum)

Controls the amount of detail written in the server log for each message that is logged. Valid values are TERSE, DEFAULT, and VERBOSE, each adding more fields to displayed messages. TERSE excludes the logging of DETAIL, HINT, QUERY, and CONTEXT error information. VERBOSE output includes the SQLSTATE error code (see also Appendix A) and the source code file name, function name, and line number that generated the error. Only superusers can change this setting.

log_hostname (boolean)

By default, connection log messages only show the IP address of the connecting host. Turning this parameter on causes logging of the host name as well. Note that depending on your host name resolution setup this might impose a non-negligible performance penalty. This parameter can only be set in the postgresql.conf file or on the server command line.

log_line_prefix (string)

This is a printf-style string that is output at the beginning of each log line. % characters begin escape sequences that are replaced with status information as outlined below. Unrecognized escapes are ignored. Other characters are copied straight to the log line. Some escapes are only recognized by session processes, and will be treated as empty by background processes such as the main server process. Status information may be aligned either left or right by specifying a numeric literal after the % and before the option. A negative value will cause the status information to be padded on the right with spaces to give it a minimum width, whereas a positive value will pad on the left. Padding can be useful to aid human readability in log files.

This parameter can only be set in the postgresql.conf file or on the server command line. The default is '%m [%p] ' which logs a time stamp and the process ID.

Escape Effect Session only
%a Application name yes
%u User name yes
%d Database name yes
%r Remote host name or IP address, and remote port yes
%h Remote host name or IP address yes
%b Backend type no
%p Process ID no
%P Process ID of the parallel group leader, if this process is a parallel query worker no
%t Time stamp without milliseconds no
%m Time stamp with milliseconds no
%n Time stamp with milliseconds (as a Unix epoch) no
%i Command tag: type of session’s current command yes
%e SQLSTATE error code no
%c Session ID: see below no
%l Number of the log line for each session or process, starting at 1 no
%s Process start time stamp no
%v Virtual transaction ID (backendID/localXID) no
%x Transaction ID (0 if none is assigned) no
%q Produces no output, but tells non-session processes to stop at this point in the string; ignored by session processes no
%Q Query identifier of the current query. Query identifiers are not computed by default, so this field will be zero unless compute_query_id parameter is enabled or a third-party module that computes query identifiers is configured. yes
%% Literal % no

The backend type corresponds to the column backend_type in the view pg_stat_activity, but additional types can appear in the log that don’t show in that view.

The %c escape prints a quasi-unique session identifier, consisting of two 4-byte hexadecimal numbers (without leading zeros) separated by a dot. The numbers are the process start time and the process ID, so %c can also be used as a space saving way of printing those items. For example, to generate the session identifier from pg_stat_activity, use this query:

SELECT to_hex(trunc(EXTRACT(EPOCH FROM backend_start))::integer) || '.' ||
       to_hex(pid)
FROM pg_stat_activity;

Tip

If you set a nonempty value for log_line_prefix, you should usually make its last character be a space, to provide visual separation from the rest of the log line. A punctuation character can be used too.

Tip

Syslog produces its own time stamp and process ID information, so you probably do not want to include those escapes if you are logging to syslog.

Tip

The %q escape is useful when including information that is only available in session (backend) context like user or database name. For example:

log_line_prefix = '%m [%p] %q%u@%d/%a '

Note

The %Q escape always reports a zero identifier for lines output by log_statement because log_statement generates output before an identifier can be calculated, including invalid statements for which an identifier cannot be calculated.

log_lock_waits (boolean)

Controls whether a log message is produced when a session waits longer than deadlock_timeout to acquire a lock. This is useful in determining if lock waits are causing poor performance. The default is off. Only superusers can change this setting.

log_recovery_conflict_waits (boolean)

Controls whether a log message is produced when the startup process waits longer than deadlock_timeout for recovery conflicts. This is useful in determining if recovery conflicts prevent the recovery from applying WAL.

The default is off. This parameter can only be set in the postgresql.conf file or on the server command line.

log_parameter_max_length (integer)

If greater than zero, each bind parameter value logged with a non-error statement-logging message is trimmed to this many bytes. Zero disables logging of bind parameters for non-error statement logs. -1 (the default) allows bind parameters to be logged in full. If this value is specified without units, it is taken as bytes. Only superusers can change this setting.

This setting only affects log messages printed as a result of log_statement, log_duration, and related settings. Non-zero values of this setting add some overhead, particularly if parameters are sent in binary form, since then conversion to text is required.

log_parameter_max_length_on_error (integer)

If greater than zero, each bind parameter value reported in error messages is trimmed to this many bytes. Zero (the default) disables including bind parameters in error messages. -1 allows bind parameters to be printed in full. If this value is specified without units, it is taken as bytes.

Non-zero values of this setting add overhead, as PostgreSQL will need to store textual representations of parameter values in memory at the start of each statement, whether or not an error eventually occurs. The overhead is greater when bind parameters are sent in binary form than when they are sent as text, since the former case requires data conversion while the latter only requires copying the string.

log_statement (enum)

Controls which SQL statements are logged. Valid values are none (off), ddl, mod, and all (all statements). ddl logs all data definition statements, such as CREATE, ALTER, and DROP statements. mod logs all ddl statements, plus data-modifying statements such as INSERT, UPDATE, DELETE, TRUNCATE, and COPY FROM. PREPARE, EXECUTE, and EXPLAIN ANALYZE statements are also logged if their contained command is of an appropriate type. For clients using extended query protocol, logging occurs when an Execute message is received, and values of the Bind parameters are included (with any embedded single-quote marks doubled).

The default is none. Only superusers can change this setting.

Note

Statements that contain simple syntax errors are not logged even by the log_statement = all setting, because the log message is emitted only after basic parsing has been done to determine the statement type. In the case of extended query protocol, this setting likewise does not log statements that fail before the Execute phase (i.e., during parse analysis or planning). Set log_min_error_statement to ERROR (or lower) to log such statements.

Logged statements might reveal sensitive data and even contain plaintext passwords.

log_replication_commands (boolean)

Causes each replication command to be logged in the server log. See Section 53.4 for more information about replication command. The default value is off. Only superusers can change this setting.

log_temp_files (integer)

Controls logging of temporary file names and sizes. Temporary files can be created for sorts, hashes, and temporary query results. If enabled by this setting, a log entry is emitted for each temporary file when it is deleted. A value of zero logs all temporary file information, while positive values log only files whose size is greater than or equal to the specified amount of data. If this value is specified without units, it is taken as kilobytes. The default setting is -1, which disables such logging. Only superusers can change this setting.

log_timezone (string)

Sets the time zone used for timestamps written in the server log. Unlike TimeZone, this value is cluster-wide, so that all sessions will report timestamps consistently. The built-in default is GMT, but that is typically overridden in postgresql.conf; initdb will install a setting there corresponding to its system environment. See Section 8.5.3 for more information. This parameter can only be set in the postgresql.conf file or on the server command line.

20.8.4. Using CSV-Format Log Output

Including csvlog in the log_destination list provides a convenient way to import log files into a database table. This option emits log lines in comma-separated-values (CSV) format, with these columns: time stamp with milliseconds, user name, database name, process ID, client host:port number, session ID, per-session line number, command tag, session start time, virtual transaction ID, regular transaction ID, error severity, SQLSTATE code, error message, error message detail, hint, internal query that led to the error (if any), character count of the error position therein, error context, user query that led to the error (if any and enabled by log_min_error_statement), character count of the error position therein, location of the error in the PostgreSQL source code (if log_error_verbosity is set to verbose), application name, backend type, process ID of parallel group leader, and query id. Here is a sample table definition for storing CSV-format log output:

CREATE TABLE postgres_log
(
  log_time timestamp(3) with time zone,
  user_name text,
  database_name text,
  process_id integer,
  connection_from text,
  session_id text,
  session_line_num bigint,
  command_tag text,
  session_start_time timestamp with time zone,
  virtual_transaction_id text,
  transaction_id bigint,
  error_severity text,
  sql_state_code text,
  message text,
  detail text,
  hint text,
  internal_query text,
  internal_query_pos integer,
  context text,
  query text,
  query_pos integer,
  location text,
  application_name text,
  backend_type text,
  leader_pid integer,
  query_id bigint,
  PRIMARY KEY (session_id, session_line_num)
);

To import a log file into this table, use the COPY FROM command:

COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;

It is also possible to access the file as a foreign table, using the supplied file_fdw module.

There are a few things you need to do to simplify importing CSV log files:

  1. Set log_filename and log_rotation_age to provide a consistent, predictable naming scheme for your log files. This lets you predict what the file name will be and know when an individual log file is complete and therefore ready to be imported.

  2. Set log_rotation_size to 0 to disable size-based log rotation, as it makes the log file name difficult to predict.

  3. Set log_truncate_on_rotation to on so that old log data isn’t mixed with the new in the same file.

  4. The table definition above includes a primary key specification. This is useful to protect against accidentally importing the same information twice. The COPY command commits all of the data it imports at one time, so any error will cause the entire import to fail. If you import a partial log file and later import the file again when it is complete, the primary key violation will cause the import to fail. Wait until the log is complete and closed before importing. This procedure will also protect against accidentally importing a partial line that hasn’t been completely written, which would also cause COPY to fail.

20.8.5. Process Title

These settings control how process titles of server processes are modified. Process titles are typically viewed using programs like ps or, on Windows, Process Explorer. See Section 28.1 for details.

cluster_name (string)

Sets a name that identifies this database cluster (instance) for various purposes. The cluster name appears in the process title for all server processes in this cluster. Moreover, it is the default application name for a standby connection (see synchronous_standby_names.)

The name can be any string of less than NAMEDATALEN characters (64 characters in a standard build). Only printable ASCII characters may be used in the cluster_name value. Other characters will be replaced with question marks (?). No name is shown if this parameter is set to the empty string '' (which is the default). This parameter can only be set at server start.

update_process_title (boolean)

Enables updating of the process title every time a new SQL command is received by the server. This setting defaults to on on most platforms, but it defaults to off on Windows due to that platform’s larger overhead for updating the process title. Only superusers can change this setting.

PostgreSQL logs are a valuable resource for troubleshooting problems, tracking
performance, and auditing database activity. Before deploying your application
to production, it’s necessary to fine-tune the logging configuration to ensure
that you’re recording the right amount of information to diagnose issues but not
to the point of slowing down essential database operations.

To achieve the ideal balance, a deep understanding of the various PostgreSQL
logging directives is key. Armed with this knowledge, coupled with an
understanding of how the logs will be used, you’d be well equipped to tailor the
logging settings to precisely suit your monitoring and troubleshooting needs.

The wrong time to be configuring Postgres’ logging settings is when there’s
already a problem with your application. Being proactive about your database
logs is the best way get the most out of them.

This article discusses the different types of PostgreSQL logs, how to configure
the many logging parameters it offers, and how to interpret log messages. It
also provide some tips on how to use PostgreSQL logs to troubleshoot problems
and improve database performance.

Prerequisites

You need PostgreSQL installed and
running on your computer. Ensure that you have
PostgreSQL 15 or later as it
introduced the ability output structured logs in JSON format.

To check the client version, use the following command:

psql --version # client version
psql (PostgreSQL) 15.3 (Ubuntu 15.3-1.pgdg22.04+1)

Viewing the server version can be accomplished by executing:

sudo -u postgres psql -c 'SHOW server_version;'
          server_version
----------------------------------
 15.3 (Ubuntu 15.3-1.pgdg22.04+1)

Setting up a sample database

To follow along with the examples in this article, you can set up the popular
Chinook sample database
which represents a digital media store, including tables for artists, albums,
media tracks, invoices, and customers.

Start by downloading
the database file
to your computer:

curl -LO https://gist.github.com/ayoisaiah/ebf628402706de0b6053699f9e0776ed/raw/43a4ce34a3c7ef397d2cf235f20f9217985f18e1/chinook.sql

Next, launch the psql interface using the default postgres user:

Create the chinook database as follows:

CREATE DATABASE chinook OWNER postgres;

Connect to the chinook database you just created:

Finally, execute the chinook.sql file to create the tables and populate them
with data:

Once the command finishes executing, you can run the following query to confirm
that the data is all there:

 albumid |                 title                 | artistid
---------+---------------------------------------+----------
       1 | For Those About To Rock We Salute You |        1
       2 | Balls to the Wall                     |        2
       3 | Restless and Wild                     |        2
       4 | Let There Be Rock                     |        1
       5 | Big Ones                              |        3
       6 | Jagged Little Pill                    |        4
       7 | Facelift                              |        5
       8 | Warner 25 Anos                        |        6
       9 | Plays Metallica By Four Cellos        |        7
      10 | Audioslave                            |        8
(10 rows)

You may now quit the psql interface:

Now that you have some data to play with, let’s proceed to the next section
where you’ll learn about where to find the logs generated by the PostgreSQL
server.

TIP: For testing purposes, you can type SET log_statement = 'all' in
psql so that all your database queries are recorded in the sever log.

Where are PostgreSQL logs stored?

The PostgreSQL server outputs its logs to the standard error stream by default.
You can confirm this by executing the following query in psql:

 log_destination
-----------------
 stderr
(1 row)

It also provides a logging collector process that is responsible for capturing
the logs sent to the standard error and routing then to log files. The behavior
of this logging collector is controlled by the aptly named logging_collector
setting.

Let’s go ahead and check the status of the logging_collector config through
the psql interface:

 logging_collector
-------------------
 off
(1 row)

The logging_collector is off by default so the PostgreSQL logs (which are
sent to the server’s standard error stream) are not handled by the logging
collector process. This means that the destination of the logs depend on where
the server’s stderr stream is directed to.

On Ubuntu, you may use the pg_lsclusters command to locate the log file for
all the PostgreSQL clusters running on your machine:

The Log file column indicates the location of the file:

Ver Cluster Port Status Owner    Data directory              Log file
14  main    5432 online postgres /var/lib/postgresql/15/main /var/log/postgresql/postgresql-15-main.log

If logging_collector is on on your machine, you may execute the following
query in psql to find out the location of the file:

SELECT pg_current_logfile();
   pg_current_logfile
------------------------
 log/postgresql-Thu.log
(1 row)

The specified file (log/postgresql-Thu.log in this case) is relative to the
PostgreSQL data directory, which you can locate by entering the following query:

   data_directory
---------------------
 /var/lib/pgsql/data
(1 row)

The complete path to the log file in this example is:

/var/lib/pgsql/data/log/postgresql-Thu.log

Once you’ve located the PostgreSQL log file, proceed to the next section where
we’ll examine the various types of logs that you might encounter when browsing
the contents of the file.

Types of PostgreSQL logs

There are various kind of PostgreSQL logs that you’re likely to encounter in the
log file depending on your configuration. This section will cover the most
important ones and what they look like.

Note that the examples below intentionally omit the log line prefix (everything
before <log_message> below) for the sake of brevity. Only the <log_message>
part is shown.

2023-07-30 08:31:50.628 UTC [2176] postgres@chinook <log_message>

1. Startup and shutdown logs

These logs describe the startup and shutdown processes of the PostgreSQL server.
Startup logs include the server version, IP address, port and UNIX socket. They
also report the last time the server was shutdown and indicate its readiness to
accept new connections:

LOG:  starting PostgreSQL 15.3 (Ubuntu 15.3-0ubuntu0.22.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0, 64-bit
LOG:  listening on IPv4 address "127.0.0.1", port 5432
LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
LOG:  database system was shut down at 2023-07-27 17:45:08 UTC
LOG:  database system is ready to accept connections

On the other hand, shutdown logs are entries describing why and how the server
was shut down which could come in handy when investigating unexpected database
failures. The logs below describe a
fast shutdown procedure
when a SIGINT signal is sent to the server:

LOG:  received fast shutdown request
LOG:  aborting any active transactions
LOG:  background worker "logical replication launcher" (PID 82894) exited with exit code 1
LOG:  shutting down
LOG:  database system is shut down

2. Query logs

These logs represent the various SQL queries that are executed against a
PostgreSQL database including SELECT, INSERT, UPDATE, DELETE, and more.

STATEMENT:  select c.firstname, c.lastname, i.invoiceid, i.invoicedate, i.billingcountry
        from customer as c, invoice as i
        where c.country = 'Brazil' and
        select distinct billingcountry from invoice;
LOG:  statement: select distinct billingcountry from invoice;
LOG:  statement: select albumid, title from album where artistid = 2;

3. Query duration logs

Closely related to query logs are duration logs that track how long a query took
to complete:

LOG:  duration: 13.020 ms
LOG:  duration: 13.504 ms  statement: UPDATE album SET title = 'Audioslave' WHERE albumid = 10;

4. Error logs

Error logs help you identify queries that lead to error conditions on the
server. Such conditions include query-specific mistakes like constraint
violations or mismatched data types to more severe problems such as deadlocks
and resource exhaustion. Here are examples of such logs:

ERROR:  relation "custome" does not exist at character 15
STATEMENT:  select * from custome where country = 'Brazil';
ERROR:  duplicate key value violates unique constraint "pk_album"
DETAIL:  Key (albumid)=(10) already exists.
STATEMENT:  UPDATE album SET albumid = 10 WHERE albumid = 2;

5. Connection and disconnection logs

Information about client connections and disconnections to the database can also
be recorded in the logs. These records include the source IP address, username,
database name, and connection status. In the case of disconnections, they also
include the total session duration.

LOG:  connection received: host=[local]
LOG:  connection authenticated: identity="postgres" method=peer (/etc/postgresql/15/main/pg_hba.conf:90)
LOG:  connection authorized: user=postgres database=chinook application_name=psql
LOG:  disconnection: session time: 0:00:08.602 user=postgres database=chinook host=[local]

6. Checkpoint and Write-Ahead Logs (WAL)

A checkpoint represents the moment when the Write Ahead Log (WAL) flushes
modified data from memory (shared buffers) to disk. These logs look like this:

LOG:  checkpoint starting: time
LOG:  checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.005 s, sync=0.002 s, total=0.025 s; sync files=2, longest=0.001 s, average=0.001 s; distance=0 kB, estimate=0 kB

Turning on the logging collector

As mentioned earlier, the logging collector is a background process that
captures log messages sent to the standard error and redirects them into files.
When you enable the collector, PostgreSQL logs will no longer be redirected to
the /var/log/postgresql/postgresql-15-main.log file (on Ubuntu) but will be
stored in a separate directory.

Before you can start modifying the PostgreSQL configuration, you need to locate
its location on your machine with the following query:

               config_file
-----------------------------------------
 /etc/postgresql/15/main/postgresql.conf

Open the path to the file in your favourite text editor with root privileges:

sudo nano /etc/postgresql/15/main/postgresql.conf

This file contains several configurable parameters and comments start with the
# character.

Now, find the logging_collector option, uncomment the line and turn it on:

/etc/postgresql/15/main/postgresql.conf

Copied!

Save the file, then restart your PostgreSQL server in the terminal (ensure to do
this after modifying any setting in this file):

sudo systemctl restart postgresql

Afterwards, view the final lines in the
/var/log/postgresql/postgresql-15-main.log file:

tail /var/log/postgresql/postgresql-15-main.log

You should observe two lines confirming that the logging collector is turned on
and that future log output is now being placed in a «log» directory:

. . .
LOG:  redirecting log output to logging collector process
HINT:  Future log output will appear in directory "log".

This «log» directory is controlled by the log_directory configuration:

/etc/postgresql/15/main/postgresql.conf

Copied!

From the comment above, the log_directory controls where PostgreSQL logs are
written to. When a relative path is supplied, it is relative to the value of the
data_directory setting:

/etc/postgresql/15/main/postgresql.conf

Copied!

data_directory = '/var/lib/postgresql/15/main'

Hence, we can deduce that the log files should be in the
/var/lib/postgresql/15/main/log/ directory. Execute the command below with
root privileges to find out:

sudo ls /var/lib/postgresql/15/main/log/

You should observe at least one log file in the directory indicating that the
logging collector is working correctly:

postgresql-2023-07-30_172237.log

In the next section, you’ll learn how to customize the log file name and related
options.

Customizing the log files

PostgreSQL allows the customization of the log filename through the
log_filename option. The default is shown below:

/etc/postgresql/15/main/postgresql.conf

Copied!

log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'

This configuration produces filenames containing the date and time of file
creation. The supported % escapes are listed in
Open Group’s strftime specification.

To keep things simple for the purpose of this tutorial you can change the log
filename to the following:

/etc/postgresql/15/main/postgresql.conf

Copied!

log_filename = 'postgresql-%a.log'

This will produce files with the abbreviated name of the day like
postgresql-Thu.log etc. Note that although the filename ends with .log, if
JSON or CSV formatting is enabled, the file produced will end in .json and
.csv respectively (postgresql-Thu.json for example).

If you want to control how each log file is created, you can change the
log_file_mode option:

/etc/postgresql/15/main/postgresql.conf

Copied!

log_file_mode = 0600 # the default

The 0600 permission is commonly used for sensitive files that should only be
accessible and modifiable by the file’s owner. With this setting, you can only
access, view or modify this file by switching to the postgres user or using
root privileges. This is a good security measure since PostgreSQL logs often
contain sensitive information.

Setting up log file rotation

As you probably already know, logging into files can quickly eat up all your
disk space if an appropriate log rotation
policy isn’t in place,
and this is especially true for PostgreSQL logs when
query logging is turned on.

Conveniently, it provides a few configuration options for controlling how its
log files are rotated:

  • log_rotation_age: This specifies the maximum age of a log file before it is
    rotated. The default value is 24 hours (1d). It can accept an integer value
    with a unit like m for minutes, d for days, w for weeks, etc. If the
    unit is not provided, minutes is assumed. You can also disable time-based log
    rotation by setting it to 0.

  • log_rotation_size: This value determines the maximum size of a log file
    before it is rotated in favor of a new log file. The default value is 10
    megabytes (10MB). Set this to 0 if you want to disable size-based log
    rotation.

  • log_truncate_on_rotation: This option only applies when time-based rotation
    is applied to a log file. It is used to overwrite existing any log file of the
    same name instead of appending to the file. This is useful when you have a
    log_filename pattern like postgresql-%a.log which produces files like
    postgresql-Mon.log, postgresql-Tue.log etc. Enabling this option (together
    with setting log_rotation_age to 7d) ensures that each week’s logs
    overwrites the previous week but the filenames remain the same.

As you can see, the options above provide basic log file rotation capabilities.
If you need more customization options like the ability to automatically
compress rotated files, turn off PostgreSQL’s log file rotation options and use
the logrotate utility
instead.

Log formatting

PostgreSQL’s log format is controlled by the log_destination setting which is
set to stderr by default. You can use also the jsonlog and csvlog options
to output the logs in JSON and CSV formats respectively. More on this soon.

/etc/postgresql/15/main/postgresql.conf

Copied!

log_destination = 'stderr'

When stderr is enabled, you can modify its output through the
log_line_prefix option which holds a printf-style string that determines the
information that is will be included at the beginning of each record. Type the
following in psql to find out its current value:

 log_line_prefix
------------------
 %m [%p] %q%u@%d
(1 row)

This value ensures that each log record sent to the stderr includes the
following details:

  • %m: The time of the event with milliseconds.
  • %p: The process ID of the specific PostgreSQL instance that created the log.
  • %q: This token produces no output but informs background processes to ignore
    everything after this point. The other tokens (%u@%d) are only available in
    session processes.
  • %u: The connected user that triggered the event.
  • %d: The database the user is connected to.

A log entry produced using the format described above looks like this:

2023-07-30 08:31:50.628 UTC [2176] postgres@chinook LOG:  statement: select albumid, title from album where artistid = 2;

Note that everything after postgres@chinook is not controlled by the
log_line_prefix and cannot be customized. It consists of the log level and the
log message which is a SELECT statement in this case.

Customizing the log format

The only way to modify the stderr log format is through the log_line_prefix
setting. Please see the
full table of escape sequences
available to you. We recommend including the following variables:

  • %a: The application name (set by using the application_name parameter in
    the PostgreSQL connection string).
  • %p: The process ID of the PostgreSQL instance.
  • %u: The connected user.
  • %d: The database name.
  • %m: The time of the event with milliseconds (or %n if you prefer a UNIX
    epoch).
  • %q: Separate session-only variables from those that are valid in all
    contexts.
  • %i: The command tag identifying the SQL query that was executed.
  • %e: The relevant
    PostgreSQL error code.
  • %c: The ID of the current session.

To make the logs easier to read and parse, you can prefix each variable with a
key like this:

/etc/postgresql/15/main/postgresql.conf

Copied!

log_line_prefix = 'time=%m pid=%p error=%e sess_id=%c %qtag=%i usr=%u db=%d app=%a '

It’s advisable to use a space (or some other character) before the closing quote
to separate the log prefix from the actual log message. With this configuration
in place, you will observe logs in the following format after restarting the
PostgreSQL server:

time=2023-07-30 22:02:27.929 UTC pid=17678 error=00000 sess_id=64c6ddf3.450e LOG:  database system is ready to accept connections
time=2023-07-30 22:03:33.613 UTC pid=17769 error=00000 sess_id=64c6de20.4569 tag=idle usr=postgres db=chinook app=psql LOG:  statement: UPDATE album SET title = 'Audioslave' WHERE albumid = 10;

The first record above is created by a non-session process (a.k.a background
process). These processes perform maintenance activities, background tasks, and
other internal functions to support the database’s operation. The inclusion of
the %q escape excludes everything after it since such sequences are invalid in
non-session processes. If %q is excluded, the tag, usr, db, and app
keys would be present in the log but empty.

On the other hand, session processes (a.k.a user backend processes) are those
that are directly tied to a specific user and database, and they handle the
actual queries and commands issued by the client application. In such contexts,
all escape sequences are valid so the %q escape will have no effect.

Customizing the time zone

The log_timezone setting controls the time zone used for recording timestamp
information. The default in Ubuntu is Etc/UTC which is
UTC time. If you
want the time to correspond to your server time, change this value accordingly:

/etc/postgresql/15/main/postgresql.conf

Copied!

log_timezone = 'Europe/Helsinki'

However,
I recommend sticking with UTC time
for ease of log correlation and normalization.

Structured logging in JSON

PostgreSQL has supported logging in the CSV format since the
v8.3 release way back in
February 2008, but JSON was only recently supported in
v15 release.
You can now format PostgreSQL logs as structured JSON by adding jsonlog to the
log_destination config like this:

/etc/postgresql/15/main/postgresql.conf

Copied!

log_destination = 'stderr,jsonlog'

When you restart the server and view the log directory, you will observe that
two new log files with the same name but different extensions (.log, and
.json). These files contain the same logs but in different formats. The .log
file follows the stderr format as before, while the .json file predictably
contains JSON logs.

Here’s a comparison of the two files with the same logs:

/var/lib/postgresql/15/main/log/postgresql-Sun.log

Copied!

time=2023-07-30 22:48:01.817 UTC pid=18254 error=00000 sess_id=64c6e836.474e tag=idle usr=postgres db=chinook app=psql LOG:  statement: SHOW data_directory;
time=2023-07-30 22:49:21.808 UTC pid=18254 error=00000 sess_id=64c6e836.474e tag=idle usr=postgres db=chinook app=psql LOG:  statement: TABLE albu limit 10;
time=2023-07-30 22:49:21.808 UTC pid=18254 error=42P01 sess_id=64c6e836.474e tag=SELECT usr=postgres db=chinook app=psql ERROR:  relation "albu" does not exist at character 7
time=2023-07-30 22:49:21.808 UTC pid=18254 error=42P01 sess_id=64c6e836.474e tag=SELECT usr=postgres db=chinook app=psql STATEMENT:  TABLE albu limit 10;

These logs are formatted according to the log_line_prefix setting as
demonstrated earlier. On the other hand, the JSON logs contain all the
available fields
except those with null values:

/var/lib/postgresql/15/main/log/postgresql-Sun.json

Copied!

{"timestamp":"2023-07-30 22:48:01.817 UTC","user":"postgres","dbname":"chinook","pid":18254,"remote_host":"[local]","session_id":"64c6e836.474e","line_num":1,"ps":"idle","session_start":"2023-07-30 22:46:14 UTC","vxid":"4/3","txid":0,"error_severity":"LOG","message":"statement: SHOW data_directory;","application_name":"psql","backend_type":"client backend","query_id":0}
{"timestamp":"2023-07-30 22:49:21.808 UTC","user":"postgres","dbname":"chinook","pid":18254,"remote_host":"[local]","session_id":"64c6e836.474e","line_num":2,"ps":"idle","session_start":"2023-07-30 22:46:14 UTC","vxid":"4/4","txid":0,"error_severity":"LOG","message":"statement: TABLE albu limit 10;","application_name":"psql","backend_type":"client backend","query_id":0}
{"timestamp":"2023-07-30 22:49:21.808 UTC","user":"postgres","dbname":"chinook","pid":18254,"remote_host":"[local]","session_id":"64c6e836.474e","line_num":3,"ps":"SELECT","session_start":"2023-07-30 22:46:14 UTC","vxid":"4/4","txid":0,"error_severity":"ERROR","state_code":"42P01","message":"relation \"albu\" does not exist","statement":"TABLE albu limit 10;","cursor_position":7,"application_name":"psql","backend_type":"client backend","query_id":0}

A keen observer will notice that the error=42P01 event is spread over two log
entries in the stderr format. However, this same event is captured in a single
entry in the JSON format.

The obvious advantage to
logging in JSON
is that the logs can be automatically parsed and analyzed by log management
tools. However, the fields cannot be customized in any way. You can’t exclude a
field or customize its name unlike the stderr format which is more flexible in
that sense. If you really need such features, you can adopt a log shipper to
transform the logs before sending it off to its final destination.

The rest of the examples will continue to be displayed in the stderr
format but without the log_line_prefix for brevity
.

How to log PostgreSQL queries

The log_statement setting controls what SQL queries are recorded in the server
log files. Its valid options are as follows:

  • none (default): No SQL queries are recorded.
  • ddl: Logs only Data Definition Language (DDL) statements, such as CREATE,
    ALTER, and DROP statements that modify the database schema.
  • mod: In addition to DDL statements, this value logs Data Modification
    Language (DML) statements such as INSERT, UPDATE, and DELETE.
  • all: Capture all SQL queries made to the server except those that fail
    before the execution phase due to parsing errors (see
    log_min_error_statement).

/etc/postgresql/15/main/postgresql.conf

Copied!

Using the all setting in production will generate an enormous amount of data
on busy systems which could slow down the database due to the overhead of
writing every single query to disk. If your motivation for logging all queries
is for auditing purposes, consider using
pgAudit instead since it provides better
controls for configuring audit logging.

Logging query duration

One of the main use cases for recording query logs is to identify slow-running
queries that need optimization. However, the log_statement option does not
track query duration in its output. You need to use the log_duration setting
for this purpose. It is a boolean meaning that it can either be off (0) or on
1:

/etc/postgresql/15/main/postgresql.conf

Copied!

Restart the server and view the logs once again. You should start observing
entries containing a duration value in milliseconds:

LOG:  statement: UPDATE album SET title = 'Audioslave' WHERE albumid = 10;

LOG: duration: 18.498 ms

When log_duration is turned on, duration logs are produced for all
statements executed by the server without exception (even a simple SELECT 1).
This will produce a huge amount of logs which could harm performance and consume
disk space unnecessarily.

Another problem with log_duration is that depending on the log_statement
configuration, you may not know what query produced the duration log. To test
this out, set log_statement to ddl, and keep log_duration enabled. Restart
the server and enter the following query in psql:

UPDATE album SET title = 'Audioslave' WHERE albumid = 10;

You will observe the following log upon successful completion of the query:

Crucially, the statement that produced this entry is not recorded at all making
it effectively useless. If you must use log_duration, ensure that the
log_statement is also set to all so that each duration log can be correlated
with the statement that produced it.

An alternative approach here is turning log_duration off and enabling
log_min_duration_statement instead:

/etc/postgresql/15/main/postgresql.conf

Copied!

log_duration = 0
log_min_duration_statement = 0

The log_min_duration_statement setting is not a boolean. It accepts an integer
value in milliseconds and produces logs for queries whose execution time exceeds
the specified value. When it is set to 0 as above, the duration of all
completed statements will be recorded.

Apply the updated configuration above and restart the PostgreSQL server, keeping
log_statement in ddl mode. Afterwards, repeat the UPDATE query. You should
observe a duration log that looks like this:

LOG:  duration: 13.504 ms  statement: UPDATE album SET title = 'Audioslave' WHERE albumid = 10;

This is an improvement on the log_duration output because the query that
produced the duration log is visible.

One weakness remains though and that is your inability to see the query in the
log file before it finishes. If the server crashes before the completion of a
query, you’ll have no idea what queries were running before the crash since it
wasn’t recorded.

However, the logging behavior changes if the query produces its own log. To
observe this, change log_statement to all then restart the server. Run
SELECT pg_sleep(5) in psql and observe the logs.

You should see the following line immediately:

LOG:  statement: SELECT pg_sleep(5);

Five seconds later, the following line will appear:

LOG:  duration: 5005.975 ms

With this configuration, if a problem occurs before the five seconds have
elapsed, you’ll at least know what queries may have played a role in the
incident.

The
PostgreSQL documentation
recommends that you include the process and session IDs in the log_line_prefix
so that you can easily correlate both entries when reading and analysing the
logs.

Logging slow-running queries

Logging the duration of all queries on your server is guaranteed to flood your
log files with numerous trivial entries that don’t add much value. It is more
useful to log only slow running queries so that can investigate and optimize
them accordingly. It also reduces the pressure on your PostgreSQL server since
the volume of entries that will be logged will be drastically reduced.

What constitutes a «slow query» obviously depends on the type of application and
specific workloads. Ensure to use a value that’ll capture resource-intensive
queries that can cause performance bottlenecks. Queries that join large tables,
perform complex calculations, or involve recursion are prime candidates to be
monitored.

A threshold of a couple hundred milliseconds for interactive applications could
be good starting point, then you can adjust as you go along:

/etc/postgresql/15/main/postgresql.conf

Copied!

log_min_duration_statement = 250

With this configuration, only queries whose execution exceeds the specified
value (250ms) will be logged. For example:

LOG:  statement: SELECT pg_sleep(1);
LOG:  duration: 1013.675 ms

Message severity levels in PostgreSQL

PostgreSQL uses the following log levels to indicate the
severity of each log record it produces:

  • DEBUG5, DEBUG4, DEBUG3, DEBUG2, DEBUG1: These levels are meant for
    logging detailed information that is useful for troubleshooting where DEBUG5
    is the most verbose, and DEBUG1 is the least verbose.
  • INFO: Reports information that was implicitly requested by the user.
  • NOTICE: Reports details that might be helpful to users.
  • WARNING: Warns about potential problems such as using COMMIT outside of a
    transaction block.
  • ERROR: Reports errors that abort the current command.
  • LOG: Reports general database activity.
  • FATAL: Reports errors that abort the current session while other sessions
    remain active.
  • PANIC: Reports severe errors that abort all existing database sessions.

Here’s an example of what messages tagged with each severity level look like:

DEBUG:  server process (PID 13557) exited with exit code 0
INFO:  vacuuming "chinook.public.album"
NOTICE:  identifier "very_very_very_very_very_very_very_very_long_table_name_with_more_than_63_characters" will be truncated to "very_very_very_very_very_very_very_very_long_table_name_with_mo"
WARNING:  SET LOCAL can only be used in transaction blocks
LOG:  statement: UPDATE album SET title = 'Audioslave' WHERE albumid = 10;
ERROR:  relation "albu" does not exist at character 7
FATAL:  role "root" does not exist
PANIC: database system shutdown requested

The default log level is set to WARNING through the log_min_messages
setting, and I recommend keeping it that way.

/etc/postgresql/15/main/postgresql.conf

Copied!

log_min_messages = warning

Reducing log verbosity

PostgreSQL allows you to control the verbosity of event logs through the
log_error_verbosity setting, which, unlike the name suggests, works for each
logged message. The possible values are terse, default, and verbose.

With the default setting, you’ll observe extra DETAIL, HINT, QUERY, or
CONTEXT information for certain queries. For example, when you query
DROP TABLE album; against the chinook database in psql, you should observe
the following logs:

LOG:  statement: DROP table album;
ERROR:  cannot drop table album because other objects depend on it
DETAIL:  constraint fk_trackalbumid on table track depends on table album
HINT:  Use DROP ... CASCADE to drop the dependent objects too.
STATEMENT:  DROP table album;

All five entries here were produced by the same query. If log_error_verbosity
is terse, the DETAIL and HINT lines will be dropped. The verbose option
includes everything in default coupled with lines that more low level details
about the error such as its source code file name, function name, and line
number:

LOCATION:  reportDependentObjects, dependency.c:1189

Note that if you’re logging in JSON, the DETAIL, HINT and STATEMENT are
included in the same log entry and LOCATION is split into func_name,
file_name and file_line_num properties:

{"timestamp":"2023-08-03 15:56:26.988 UTC","user":"postgres","dbname":"chinook","pid":14155,"remote_host":"[local]","session_id":"64cbce27.374b","line_num":5,"ps":"DROP TABLE","session_start":"2023-08-03 15:56:23 UTC","vxid":"3/8","txid":16414,"error_severity":"ERROR","state_code":"2BP01","message":"cannot drop table album because other objects depend on it","detail":"constraint fk_trackalbumid on table track depends on table album","hint":"Use DROP ... CASCADE to drop the dependent objects too.","statement":"DROP table album;","func_name":"reportDependentObjects","file_name":"dependency.c","file_line_num":1189,"application_name":"psql","backend_type":"client backend","query_id":0}

Logging connections and disconnections

The following settings control how PostgreSQL logs client connections and
disconnections. Both are turned off by default:

/etc/postgresql/15/main/postgresql.conf

Copied!

log_connections = off
log_disconnections = off

Turning log_connections on (1) will log each attempted connection to the
server as well as the completion of the client authentication and authorization:

/etc/postgresql/15/main/postgresql.conf

Copied!

LOG:  connection received: host=[local]
LOG:  connection authenticated: identity="postgres" method=peer (/etc/postgresql/15/main/pg_hba.conf:90)
LOG:  connection authorized: user=postgres database=chinook application_name=psql

This information can be useful from a security perspective since you can see
who’s connecting to your systems and from where. It can also result in a large
amount of log entries if there are several short-lived connections to the server
and connection pooling isn’t being used.

The log_disconnections option works similarly, and it includes the session
duration:

/etc/postgresql/15/main/postgresql.conf

Copied!

LOG:  disconnection: session time: 0:00:02.824 user=postgres database=chinook host=[local]

Creating custom log messages

So far, we’ve discussed the various types of logs produced by the PostgreSQL
server and how to customize them. If you want to create your own log messages in
your SQL queries or when writing trigger functions using
PL/pgSQL, you
may use the
RAISE
statement as follows:

CREATE OR REPLACE FUNCTION custom_logs_func()
RETURNS TRIGGER LANGUAGE PLPGSQL AS $$
BEGIN
    RAISE LOG 'This is an informational message';
    RAISE WARNING 'Something unexpected happened';
    RAISE EXCEPTION 'An unexpected error';
END;
$$;

CREATE OR REPLACE TRIGGER UPDATE_LAST_EDITED_TIME BEFORE UPDATE ON ALBUM
FOR EACH ROW EXECUTE FUNCTION custom_logs_func();

Depending on your log_min_messages setting, you will observe the following in
the log file when custom_logs_func() is executed:

LOG:  This is an informational message
WARNING:  Something unexpected happened
ERROR:  An unexpected error

Analyzing log files with PgBadger

PgBadger is a command-line log analysis
tool for PostgreSQL log files. It is designed to parse and analyze PostgreSQL
log files, extracting valuable insights and generating detailed reports on the
database’s activity, performance, and usage patterns. It can produce
comprehensive HTML reports with a host of statistics such as the following:

  • Total number of log entries parsed and processed.
  • Top slowest queries and their execution times.
  • Number of connections over time.
  • Most frequent error messages and their occurrence count.
  • Commit and rollback statistics for transactions.
  • Tables with the highest temporary file usage.
  • Histogram of query and session duration.
  • Users and applications involved in top queries..
  • The most time consuming prepare/bind queries
  • Number of autovacuum processes and their execution times.
  • and more!

Once you install pgbadger, check out its
recommended logging configuration settings
and configure them accordingly in your PostgreSQL config file. Note that those
settings are tailored to produced as much information as possible so that
pgBadger can effectively analyze database activity.

The tool provides a host of options, but the most basic setup to generate HTML
reports is to provide the configured log line prefix, the log file, and the
desired name/location of the report. At the time of writing, it only works with
the stderr and CSV formats, but not JSON.

sudo pgbadger --prefix 'time=%m pid=%p error=%e sess_id=%c %qtag=%i usr=%u db=%d app=%a ' /var/lib/postgresql/15/main/log/postgresql-Thu.log -o postgresql.html
[========================>] Parsed 289019 bytes of 289019 (100.00%), queries: 102, events: 42
LOG: Ok, generating html report...

When you open the HTML file in your browser, you should observe a report that
looks like this:

Please see the
examples on the PgBadger website
and its documentation for more
details.

Final thoughts

PostgreSQL logs is a powerful way to gain insight into your database activity
and performance when configured correctly. It may not be the glamorous thing to
do, but sticking with it through it is sure to reap dividends. Setting it up
correctly may involve some trial and error, but I hope this article has set you
on the right path to getting it right.

Thanks for reading, and happy logging!

Author's avatar

Article by

Ayooluwa Isaiah

Ayo is the Head of Content at Better Stack. His passion is simplifying and communicating complex technical ideas effectively. His work was featured on several esteemed publications including LWN.net, Digital Ocean, and CSS-Tricks. When he’s not writing or coding, he loves to travel, bike, and play tennis.

Centralize all your logs into one place.

Analyze, correlate and filter logs with SQL.

Create actionable

dashboards.

Share and comment with built-in collaboration.

Got an article suggestion?
Let us know

Next article

How To Start Logging With MariaDB

Learn how to start logging with MariaDB and go from basics to best practices in no time.

Licensed under CC-BY-NC-SA

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Понравилась статья? Поделить с друзьями:
  • Postgresql логи ошибок
  • Powershell поймать ошибку
  • Postgresql ошибка функция не существует
  • Powershell убрать вывод ошибки
  • Postgresql код ошибки 28000