1. Overview
The Linux operating system, and many applications that run on it, do a lot of logging. These logs are invaluable for monitoring and troubleshooting your system.
What you’ll learn
- Viewing logs with a simple GUI tool
- Basic command-line commands for working with log files
What you’ll need
- Ubuntu Desktop or Server
- Very basic command-line knowledge (
cd
,ls
, etc.)
Originally authored by Ivan Fonseca.
How will you use this tutorial?
-
Only read through it
Read it and complete the exercises
What is your current level of experience?
-
Novice
Intermediate
Proficient
2. Log files locations
There are many different log files that all serve different purposes. When trying to find a log about something, you should start by identifying the most relevant file. Below is a list of common log file locations.
System logs
System logs deal with exactly that — the Ubuntu system — as opposed to extra applications added by the user. These logs may contain information about authorizations, system daemons and system messages.
Authorization log
Location: /var/log/auth.log
Keeps track of authorization systems, such as password prompts, the sudo
command and remote logins.
Daemon Log
Location: /var/log/daemon.log
Daemons are programs that run in the background, usually without user interaction. For example, display server, SSH sessions, printing services, bluetooth, and more.
Debug log
Location: /var/log/debug
Provides debugging information from the Ubuntu system and applications.
Kernel log
Location: /var/log/kern.log
Logs from the Linux kernel.
System log
Location: /var/log/syslog
Contains more information about your system. If you can’t find anything in the other logs, it’s probably here.
Application logs
Some applications also create logs in /var/log
. Below are some examples.
Apache logs
Location: /var/log/apache2/
(subdirectory)
Apache creates several log files in the /var/log/apache2/
subdirectory. The access.log
file records all requests made to the server to access files. error.log
records all errors thrown by the server.
X11 server logs
Location: /var/log/Xorg.0.log
The X11 server creates a seperate log file for each of your displays. Display numbers start at zero, so your first display (display 0) will log to Xorg.0.log
. The next display (display 1) would log to Xorg.1.log
, and so on.
Non-human-readable logs
Not all log files are designed to be read by humans. Some were made to be parsed by applications. Below are some of examples.
Login failures log
Location: /var/log/faillog
Contains info about login failures. You can view it with the faillog
command.
Last logins log
Location: /var/log/lastlog
Contains info about last logins. You can view it with the lastlog
command.
Login records log
Location: /var/log/wtmp
Contains login info used by other utilities to find out who’s logged in. To view currently logged in users, use the who
command.
This is not an exhaustive list!
You can search the web for more locations relevant to what you’re trying to debug. There is also a longer list here.
3. Viewing logs using GNOME System Log Viewer
The GNOME System Log Viewer provides a simple GUI for viewing and monitoring log files. If you’re running Ubuntu 17.10 or above, it will be called Logs. Otherwise, it will be under the name System Log.
System Log Viewer interface
The log viewer has a simple interface. The sidebar on the left shows a list of open log files, with the contents of the currently selected file displayed on the right.
The log viewer not only displays but also monitors log files for changes. The bold text (as seen in the screenshot above) indicates new lines that have been logged after opening the file. When a log that is not currently selected is updated, it’s name in the file list will turn bold (as shown by auth.log
in the screenshot above).
Clicking on the cog at the top right of the window will open a menu allowing you to change some display settings, as well as open and close log files.
There is also a magnifying glass icon to the right of the cog that allows you to search within the currently selected log file.
More information
If you wish to learn more about the GNOME System Log Viewer, you may visit the official documentation.
4. Viewing and monitoring logs from the command line
It is also important to know how to view logs in the command line. This is especially useful when you’re remotely connected to a server and don’t have a GUI.
The following commands will be useful when working with log files from the command line.
Viewing files
The most basic way to view files from the command line is using the cat
command. You simply pass in the filename, and it outputs the entire contents of the file: cat file.txt
.
This can be inconvenient when dealing with large files (which isn’t uncommon for logs!). We could use an editor, although that may be overkill just to view a file. This is where the less
command comes in. We pass it the filename (less file.txt
), and it will open the file in a simple interface. From here, we can use the arrow keys (or j/k if you’re familiar with Vim) to move through the file, use /
to search, and press q
to quit. There are a few more features, all of which are described by pressing h
to open the help.
Viewing the start or end of a file
We may also want to quickly view the first or last n
number of lines of a file. This is where the head
and tail
commands come in handy. These commands work much like cat
, although you can specify how many lines from the start/end of the file you want to view. To view the first 15 lines of a file, we run head -n 15 file.txt
, and to view the last 15, we run tail -n 15 file.txt
. Due to the nature of log files being appended to at the bottom, the tail
command will generally be more useful.
Monitoring files
To monitor a log file, you may pass the -f
flag to tail
. It will keep running, printing new additions to the file, until you stop it (Ctrl + C). For example: tail -f file.txt
.
Searching files
One way that we looked at to search files is to open the file in less
and press /
. A faster way to do this is to use the grep
command. We specify what we want to search for in double quotes, along with the filename, and grep
will print all the lines containing that search term in the file. For example, to search for lines containing “test” in file.txt
, you would run grep "test" file.txt
.
If the result of a grep
search is too long, you may pipe it to less
, allowing you to scroll and search through it: grep "test" file.txt | less
.
Editing files
The simplest way to edit files from the command line is to use nano
. nano
is a simple command line editor, which has all the most useful keybindings printed directly on screen. To run it, just give it a filename (nano file.txt
). To close or save a file, press Ctrl + X. The editor will ask you if you want to save your changes. Press y
for yes or n
for no. If you choose yes, it will ask you for the filename to save the file as. If you are editing an existing file, the filename will already be there. Simply leave it as it is and it will save to the proper file.
5. Conclusion
Congratulations, you now have enough knowledge of log file locations, usage of the GNOME System Log Viewer and basic command line commands to properly monitor and trouble-shoot problems that arise on your system.
Further reading
- The Ubuntu Wiki has an article that goes more in-depth into Ubuntu log files.
- This DigitalOcean Community article covers viewing Systemd logs
Was this tutorial useful?
Thank you for your feedback.
Приветствую читателей своего сайта. Сегодня я всесторонне рассмотрю тему тему логов в Ubuntu — ошибки, загрузка, системные логи, cron и остальное. Постараюсь дать обзорную информацию по основным моментам в этой теме. Материал в основном рассчитан на новичков, но возможно восполнит пробелы и специалистов.
Традиционно логи в Linux хранятся в директории /var/log. Вот описание стандартных лог файлов Ubuntu, которые там присутствуют. Кстати, если вы только планируете устанавливать ubuntu, то можете воспользоваться моей подробной статьей на этот счет — установка ubuntu server. Так же вам может быть интересен мой обзор и сравнение сервера убунту с другими linux системами — Ubuntu Server — обзор для начинающих, сравнение, отзывы.
- syslog или messages. Последнего чаще всего нет и вместо него только syslog. Это традиционные глобальные системные журналы операционной системы linux. Сюда пишутся события загрузки, ядра системы, системы инициализации systemd и т.д.
- auth.log — лог авторизации и аутентификации в системе.
- dmesg — в этом логе хранится информация о загрузке ядра и драйверов оборудования.
- alternatives.log — лог файл программы update-alternatives. Не знаю, за какие такие заслуги ей выделили отдельный лог файл, а cron, к примеру, нет.
- kern.log — лог сообщений ядра ubuntu, да и любой другой linux системы.
- maillog — сообщения почтовой системы. Обычно postfix или exim. Если на сервере ubuntu они не установлены, то и почтового лога не будет.
- dpkg.log — логирование работы пакетных менеджеров ubuntu. Обычно это apt или apt-get.
- lastlog и wtmp — информация о прошлых авторизациях пользователей.
Лог загрузки
Начнем с самого начала. В момент загрузки системы записывается вся основная информация, имеющая к ней отношение. Если у вас будут какие-то ошибки во время старта сервера, вы сможете их увидеть в этом логе. Посмотреть лог загрузки Ubuntu можно следующим образом.
sudo dmesg
У вас получится очень длинный вывод всего того, что происходило с системой на старте. Если ищите что-то конкретное, то можете сделать фильтрацию вывода с помощью grep. Допустим, вам надо узнать информацию только о диске.
sudo dmesg | grep sda
Вы увидите лог загрузки системы ubuntu, содержащий информацию только о диске sda. Аналогичным образом можно фильтровать вывод по другим темам. Например, посмотреть все ошибки, которые были во время загрузки.
sudo dmesg | grep error
И так далее. Информация, которую выводит команда dmesg, хранится в log файле /var/log/dmesg.
Логи авторизации, в том числе ssh
Для того, чтобы узнать, кто и когда проходил авторизацию на сервере ubuntu, можно воспользоваться логами из файла /var/log/auth.log. Авторизация по ssh там будет выглядеть следующим образом.
sshd[2774]: Accepted publickey for root from 21.17.214.129 port 2673 ssh2: RSA SHA256:MCDja9Ve7rYZCzeVGpYXpqRxfAanWwVkcd+lU3GS sshd[2774]: pam_unix(sshd:session): session opened for user root by (uid=0) systemd-logind[628]: New session 6 of user root.
Здесь мы видим ip адрес, с которого произошло подключение и слепок сертификата, так как аутентификация была произведена с его помощью. Если хотите повысить уровень логирования подключений по ssh и получать больше информации, то можете отредактировать конфигурационный файл sshd — /etc/ssh/sshd_config, добавив туда следующий параметр.
LogLevel VERBOSE
Не забудьте перезапустить службу sshd для принятия изменений:
sudo systemctl restart sshd
После этого логирование подключений по ssh будет более подробное.
Лог локального входа в ubuntu тоже хранится в файле auth.log. Информация о подключении через консоль выглядит следующим образом.
login[680]: pam_unix(login:session): session opened for user root by LOGIN(uid=0) systemd-logind[628]: New session 9 of user root. login[3094]: ROOT LOGIN on '/dev/tty1'
Устройство /dev/tty1 говорит о том, что вход локальный.
Вы можете быстро посмотреть информацию о последних входах в систему с помощью команды last. Эта информация хранится в бинарном логе /var/log/lastlog.
Примерно то же самое можно увидеть с помощью utmpdump.
sudo utmpdump /var/log/wtmp
Логи ошибок в Ubuntu
Рассмотрим теперь вопрос с расположением лога ошибок в Ubuntu. Как такового отдельного error log в традиционных linux системах нет. И Убунта тут не исключение. Ошибки придется искать по системным и программным логам выборкой ключевых слов. Обычно используют следующие фразы:
- error или err
- critical или crit
- debug
- warn
Например, посмотрим в логе загрузки dmesg все сообщения уровня предупреждений (warn).
sudo dmesg -l warn
А теперь проверим ошибки в системном логе.
sudo cat /var/log/syslog | grep error
Видим некоторые ошибки в службе systemd-resolved.
Cron logs
Часто хочется проверить лог запуска периодических заданий cron. В Ubuntu, как мне кажется, сделали не удобно. По умолчанию, cron logs не выделены в отдельный файл. Искать их стоит в общем системном логе syslog. Например, в Centos существует отдельный лог-файл /var/log/cron, где собрана вся информация о запущенных заданиях. Предлагаю сделать так же в Ubuntu.
Для этого открываем конфигурационный файл /etc/rsyslog.d/50-default.conf и добавляем туда следующую информацию.
cron.* /var/log/cron.log
По умолчанию, она присутствует в конфиге, но закомментирована. Вам нужно убрать # в начале строки, чтобы раскомментировать ее. Так же я рекомендую сделать так, чтобы эти логи не дублировались в общий системный лог. Для этого немного измените следующую строку.
*.*;auth,authpriv.none,cron.none -/var/log/syslog
Я добавил в нее cron.none, чтобы логи cron не писались больше в системный лог syslog. После этого перезапустите службы rsyslog и cron и проверяйте изменения.
sudo systemctl restart rsyslog sudo systemctl restart cron
Проверяем cron.log
sudo cat /var/log/cron.log
Теперь у нас все логи Cron в Ubuntu будут в отдельном файле.
Лог действий пользователя
Мне часто задают вопросы, как посмотреть лог действий пользователя в системе или как узнать, какие программы он запускал. По умолчанию, такие действия не логируются в ubuntu. Для этого нужно устанавливать какое-то дополнительное программное обеспечение. Я даже не знаю, кто умеет это делать. Обычно если надо фиксировать действия пользователя, включается лог работы sudo.
Для того, чтобы включить логирование действий пользователя через sudo, редактируем файл /etc/sudoers. Добавляем туда строку.
Defaults logfile=/var/log/sudo.log
Теперь выполните какую-нибудь команду через sudo.
sudo cat /var/log/cron.log
Проверяем sudo.log.
Nov 25 23:10:36 : root : TTY=pts/3 ; PWD=/root ; USER=root ; COMMAND=/usr/bin/cat /var/log/cron.log
Выполненная команда пользователя сохранена в логе sudo.log. Теперь никто не сможет выполнить незаметно административные действия на сервере. Конечно, человек с полными правами сможет изменить любой лог файл, удалив свои действия при желании. Для этого важные логи нужно отправлять куда-то в другое место, но это уже тема отдельной статьи.
На сегодня по логам в Ubuntu у меня все. Желаю вам логов без ошибок и вечного аптайма (шутка, надо ставить обновы и перезагружаться).
Время на прочтение
6 мин
Количество просмотров 427K
Невозможно представить себе пользователя и администратора сервера, или даже рабочей станции на основе Linux, который никогда не читал лог файлы. Операционная система и работающие приложения постоянно создают различные типы сообщений, которые регистрируются в различных файлах журналов. Умение определить нужный файл журнала и что искать в нем поможет существенно сэкономить время и быстрее устранить ошибку.
Журналирование является основным источником информации о работе системы и ее ошибках. В этом кратком руководстве рассмотрим основные аспекты журналирования операционной системы, структуру каталогов, программы для чтения и обзора логов.
Основные лог файлы
Все файлы журналов, можно отнести к одной из следующих категорий:
- приложения;
- события;
- службы;
- системный.
Большинство же лог файлов содержится в директории /var/log
.
- /var/log/syslog или /var/log/messages содержит глобальный системный журнал, в котором пишутся сообщения с момента запуска системы, от ядра Linux, различных служб, обнаруженных устройствах, сетевых интерфейсов и много другого.
- /var/log/auth.log или /var/log/secure — информация об авторизации пользователей, включая удачные и неудачные попытки входа в систему, а также задействованные механизмы аутентификации.
- /var/log/dmesg — драйвера устройств. Одноименной командой можно просмотреть вывод содержимого файла. Размер журнала ограничен, когда файл достигнет своего предела, старые сообщения будут перезаписаны более новыми. Задав ключ
--level=
можно отфильтровать вывод по критерию значимости.
Поддерживаемые уровни журналирования (приоритеты):
emerg - система неиспользуемая
alert - действие должно быть произведено немедленно
crit - условия критичности
err - условия ошибок
warn - условия предупреждений
notice - обычные, но значимые условия
info - информационный
debug - отладочные сообщения
(5:520)$ dmesg -l err
[1131424.604352] usb 1-1.1: 2:1: cannot get freq at ep 0x1
[1131424.666013] usb 1-1.1: 1:1: cannot get freq at ep 0x81
[1131424.749378] usb 1-1.1: 1:1: cannot get freq at ep 0x81
- /var/log/alternatives.log — Вывод программы
update-alternatives
, в котором находятся символические ссылки на команды или библиотеки по умолчанию. - /var/log/anaconda.log — Записи, зарегистрированные во время установки системы.
- /var/log/audit — Записи, созданные службой аудита
auditd
. - /var/log/boot.log — Информация, которая пишется при загрузке операционной системы.
- /var/log/cron — Отчет службы
crond
об исполняемых командах и сообщения от самих команд. - /var/log/cups — Все, что связано с печатью и принтерами.
- /var/log/faillog — Неудачные попытки входа в систему. Очень полезно при проверке угроз в системе безопасности, хакерских атаках, попыток взлома методом перебора. Прочитать содержимое можно с помощью команды
faillog
. - var/log/kern.log — Журнал содержит сообщения от ядра и предупреждения, которые могут быть полезны при устранении ошибок пользовательских модулей встроенных в ядро.
- /var/log/maillog/ или /var/log/mail.log — Журнал почтового сервера, используемого на ОС.
- /var/log/pm-powersave.log — Сообщения службы экономии заряда батареи.
- /var/log/samba/ — Логи файлового сервера
Samba
, который используется для доступа к общим папкам Windows и предоставления доступа пользователям Windows к общим папкам Linux. - /var/log/spooler — Для представителей старой школы, содержит сообщения USENET. Чаще всего бывает пустым и заброшенным.
- /var/log/Xorg.0.log — Логи X сервера. Чаще всего бесполезны, но если в них есть строки начинающиеся с EE, то следует обратить на них внимание.
Для каждого дистрибутива будет отдельный журнал менеджера пакетов.
- /var/log/yum.log — Для программ установленных с помощью
Yum
в RedHat Linux. - /var/log/emerge.log — Для
ebuild
-ов установленных изPortage
с помощьюemerge
в Gentoo Linux. - /var/log/dpkg.log — Для программ установленных с помощью
dpkg
в Debian Linux и всем семействе родственных дистрибутивах.
И немного бинарных журналов учета пользовательских сессий.
- /var/log/lastlog — Последняя сессия пользователей. Прочитать можно командой
last
. - /var/log/tallylog — Аудит неудачных попыток входа в систему. Вывод на экран с помощью утилиты
pam_tally2
. - /var/log/btmp — Еже один журнал записи неудачных попыток входа в систему. Просто так, на всякий случай, если вы еще не догадались где следует искать следы активности взломщиков.
- /var/log/utmp — Список входов пользователей в систему на данный момент.
- /var/log/wtmp — Еще один журнал записи входа пользователей в систему. Вывод на экран командой
utmpdump
.
(5:535)$ sudo utmpdump /var/log/wtmp
[5] [02187] [l0 ] [ ] [4.0.5-gentoo ] [0.0.0.0 ] [Вт авг 11 16:50:07 2015]
[1] [00000] [~~ ] [shutdown] [4.0.5-gentoo ] [0.0.0.0 ] [Вт авг 11 16:50:08 2015]
[2] [00000] [~~ ] [reboot ] [3.18.12-gentoo ] [0.0.0.0 ] [Вт авг 11 16:50:57 2015]
[8] [00368] [rc ] [ ] [3.18.12-gentoo ] [0.0.0.0 ] [Вт авг 11 16:50:57 2015]
[1] [20019] [~~ ] [runlevel] [3.18.12-gentoo ] [0.0.0.0 ] [Вт авг 11 16:50:57 2015]
И другие журналы
Так как операционная система, даже такая замечательная как Linux, сама по себе никакой ощутимой пользы не несет в себе, то скорее всего на сервере или рабочей станции будет крутится база данных, веб сервер, разнообразные приложения. Каждое приложения или служба может иметь свой собственный файл или каталог журналов событий и ошибок. Всех их естественно невозможно перечислить, лишь некоторые.
- /var/log/mysql/ — Лог базы данных MySQL.
- /var/log/httpd/ или /var/log/apache2/ — Лог веб сервера Apache, журнал доступа находится в
access_log
, а ошибки — вerror_log
. - /var/log/lighthttpd/ — Лог веб сервера lighttpd.
В домашнем каталоге пользователя могут находится журналы графических приложений, DE.
- ~/.xsession-errors — Вывод
stderr
графических приложений X11.
Initializing "kcm_input" : "kcminit_mouse"
Initializing "kcm_access" : "kcminit_access"
Initializing "kcm_kgamma" : "kcminit_kgamma"
QXcbConnection: XCB error: 3 (BadWindow), sequence: 181, resource id: 10486050, major code: 20 (GetProperty), minor code: 0
kf5.kcoreaddons.kaboutdata: Could not initialize the equivalent properties of Q*Application: no instance (yet) existing.
QXcbConnection: XCB error: 3 (BadWindow), sequence: 181, resource id: 10486050, major code: 20 (GetProperty), minor code: 0
Qt: Session management error: networkIdsList argument is NULL
- ~/.xfce4-session.verbose-log — Сообщения рабочего стола XFCE4.
Чем просматривать — lnav
Почти все знают об утилите less
и команде tail -f
. Также для этих целей сгодится редактор vim
и файловый менеджер Midnight Commander. У всех есть свои недостатки: less
неважно обрабатывает журналы с длинными строками, принимая их за бинарники. Midnight Commander годится только для беглого просмотра, когда нет необходимости искать по сложному шаблону и переходить помногу взад и вперед между совпадениями. Редактор vim
понимает и подсвечивает синтаксис множества форматов, но если журнал часто обновляется, то появляются отвлекающие внимания сообщения об изменениях в файле. Впрочем это легко можно обойти с помощью <:view /path/to/file>
.
Недавно я обнаружил еще одну годную и многообещающую, но слегка еще сыроватую, утилиту — lnav, в расшифровке Log File Navigator.
Установка пакета как обычно одной командой.
$ aptitude install lnav #Debian/Ubuntu/LinuxMint
$ yum install lnav #RedHat/CentOS
$ dnf install lnav #Fedora
$ emerge -av lnav #Gentoo, нужно добавить в файл package.accept_keywords
$ yaourt -S lnav #Arch
Навигатор журналов lnav понимает ряд форматов файлов.
- Access_log веб сервера.
- CUPS page_log
- Syslog
- glog
- dpkg.log
- strace
- Произвольные записи с временными отметками
- gzip, bzip
- Журнал VMWare ESXi/vCenter
Что в данном случае означает понимание форматов файлов? Фокус в том, что lnav больше чем утилита для просмотра текстовых файлов. Программа умеет кое что еще. Можно открывать несколько файлов сразу и переключаться между ними.
(5:471)$ sudo lnav /var/log/pm-powersave.log /var/log/pm-suspend.log
Программа умеет напрямую открывать архивный файл.
(5:471)$ lnav -r /var/log/Xorg.0.log.old.gz
Показывает гистограмму информативных сообщений, предупреждений и ошибок, если нажать клавишу <i>
. Это с моего syslog-а.
Mon May 02 20:25:00 123 normal 3 errors 0 warnings 0 marks
Mon May 02 22:40:00 2 normal 0 errors 0 warnings 0 marks
Mon May 02 23:25:00 10 normal 0 errors 0 warnings 0 marks
Tue May 03 07:25:00 96 normal 3 errors 0 warnings 0 marks
Tue May 03 23:50:00 10 normal 0 errors 0 warnings 0 marks
Wed May 04 07:40:00 96 normal 3 errors 0 warnings 0 marks
Wed May 04 08:30:00 2 normal 0 errors 0 warnings 0 marks
Wed May 04 10:40:00 10 normal 0 errors 0 warnings 0 marks
Wed May 04 11:50:00 126 normal 2 errors 1 warnings 0 marks
Кроме этого поддерживается подсветка синтаксиса, дополнение по табу и разные полезности в статусной строке. К недостаткам можно отнести нестабильность поведения и зависания. Надеюсь lnav будет активно развиваться, очень полезная программа на мой взгляд.
Использованные материалы
- lnav — An Advanced Log File viewer for Linux
- What Are Linux Logs? How to View Them, Most Important Directories, and More
- Как посмотреть логи в Linux
All log files are located in /var/log
directory. In that directory, there are specific files for each type of logs. For example, system logs, such as kernel activities are logged in syslog
file.
Some of the most common log files in that directory is :
-
In directory
apt
there is a filehistory.log
which saves all the package installation and removal information even the initial system build as Live CD. You can open this file to see this very interesting file. -
In directory
dist-upgrade
there is a fileapt.log
which logs the information during distribution upgrades -
In directory
installer
the log files which are created during installation can be found. -
There is an
apport.log
file which saves information about crashes in your system and reporting them. -
The file
auth.log
includes information about the authentication activities such as when you authenticate as root user via sudo. -
The file
dpkg.log
saves the low level details of package installation and removal related withdpkg
. You might be aware that theapt
system depends ondpkg
for package installation and removal. -
boot.log
includes information of each booting. -
kern.log
saves kernel information such as warnings, errors etc. -
alternatives.log
includes the history of all the alternatives set by various packages and their removal viaupdate-alternatives
command. -
Another important log file is
Xorg.log
which include information about the graphics driver, its failures, warnings etc.
Some other types of Log files may be there depending on your installed packages. For example, My system also includes a log files epoptes.log
which will only be there if you install epoptes
package.
Changes after systemd
With the advent of systemd
, logging is mostly handled by journalctl
utility and store the logs in binary format in /var/lib/systemd/catalog/database
file. This file enumerates all logs including kernel, boot and application logs and provides required logs via journalctl
utility.
Here is a good article on journalctl
on how you can use it to fetch required log info.
https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-view-and-manipulate-systemd-logs#setting-the-system-time
Ubuntu provides extensive logging capabilities, so most of the activities happening in the system are tracked via logs. Ubuntu logs are valuable sources of information about the state of your Ubuntu operating system and the applications deployed on it. The majority of the logs are in plain text ASCII format and easily readable. This makes them a great tool to use for troubleshooting and identifying the root causes associated with system failures or application errors.
Due to the wide variety of system and application logs available, choosing the appropriate log sources and locating them within your system can be a daunting task. In this post, I’ll be taking you through the many types of Ubuntu logs, as well as how to view and analyze them.
What Are Ubuntu Logs?
Ubuntu logs record the events occurring in the hardware, software, and operating system, which aid in determining the underlying cause of any problems the system may experience.
In Ubuntu, there are two types of logs:
- System logs that provide insights on operating system functionalities like access control and system processes.
- Application logs are generated by the applications deployed in the system, offering information on their state.
Ubuntu Log Files Location: Where Are They Stored?
The root directory of the majority of log files is /var/log, while most system logs are generated by the syslog log daemon to capture activities. The OS will use syslog, whereas the majority of programs will log data to files in the /var/log subdirectory.
System Logs
System logs include details about the system and certain applications. Syslog usually stores its log files at /var/log/messages or /var/log/syslog.
Below are some key system logs and how to find them to help gain more insights into the health of your Ubuntu system and the applications running on it
Authorization Log
This log contains information about the authorization processes that happen in the system. This includes sudo commands, SSH logins, or any authentication modules used for collecting user credentials. You can find the authorization log file at /var/log/auth.log.
Daemon Log
Ubuntu systems have a number of daemons, which are effectively background services that offer certain functionalities, monitor the system, or support other processes. Some of the common daemons are systemd, logind, gcfsv, etc. They could also be application-specific daemons like mysqld. These logs are routed to the /var/log/daemon.log file and can help you investigate issues associated with their respective daemons.
Debug Log
Systems use the syslogd protocol to send log messages to a central location. The debug log stores the debug messages sent to syslogd at the DEBUG level by applications or Ubuntu system components. You can find the debug log at /var/log/debug.
Kernel Log
This log captures events or messages related to the Linux kernel. The information available in this log helps administrators investigate kernel-level issues in Ubuntu and you can find it at /var/log/kern/log.
Kernel Ring Buffer
The kernel ring buffer is a data structure that lets you store information related to kernel operations in Ubuntu. It is not a log file but rather a constant-size buffer where the oldest messages are removed to make space for new messages. The messages can be queries using the following command:
dmesg | less
System Log
The system log contains exhaustive information about the state of an Ubuntu system. It’s found at /var/log/syslog, the default location referred to by administrators when required information is not found in other log files. Applications or services that do not have their own log files use this file to store logging information. It also contains logs that used to be saved in /var/log/messages in older versions of Ubuntu.
Service Log
In Ubuntu, service logs are system logs that capture information about the state of your services. You can look into an event logged by the daemon systemd-journald if logs about a specific service are not found in /var/log. This daemon publishes a single stream combining the log outputs from all services to the central journal in /{run,var}/log/journal/. To view data for a specific service, use the following command:
journalctl -f -u {service name}
Cron Log
The cron daemon reads crontab files and runs scheduled operations in the background. The logs it produces are useful for debugging and auditing cron jobs. The Ubuntu crontab log can be found in /var/log/syslog. The following command lets you view cron job-specific entries in the file:
grep CRON /var/log/syslog
Network Log
The NetworkManager service manages network connectivity in Ubuntu systems. By default, the log files generated by NetworkManager are streamed to /var/log/syslog. For the latest versions of Ubuntu, you can view the logs with the command:
journalctl /usr/sbin/NetworkManager
Audit Log
Audit information about Ubuntu systems is provided by auditd, the Linux audit daemon. The default location of the auditd log file is /var/log/audit/audit.log. This also allows you to view commands like ausearch and aureport to parse and analyze audit logs.
Startup Log
System boot logs are located in /var/log/boot.log. To read the most recent boot log messages on Ubuntu operating systems that use systemd, use the journalctl command as below:
journalctl -b
Crash Log
The Ubuntu Kernel Crash Dump mechanism collects the state and memory of the kernel when a system crashes. This data is saved to /var/crash and helps in identifying the root cause of the crash.
Firewall Log
Firewall logs are necessary for identifying odd network activities, spotting attacks, and debugging firewall rules. The Ubuntu firewall logs associated with the default firewall service UFW can be found in the /var/log folder. You can manage the location of the logs by editing the Ubuntu syslog config file available at /etc/syslog.conf.
Journal Log
Journald is the log management service daemon in Ubuntu. Go ahead and edit the location of the logging file in /etc/systemd/journald.conf. To view the logging data captured by journald, use the journalctl
command.
Error Log
Error logs provide useful information about issues happening in application and system components. Error log files specific to an application or service are available in the /var/log folder.
SSH Log
Information about user logins and usage of sudo commands in SSH is also stored in the authorization logs. You can search the files to get information about SSH logs using the following command:
grep sshd /var/log/auth.log | less
Application Logs
I usually recommend you analyze application logs as the first step when attempting to identify the source of an application error or, in general, to review your app’s health. Let’s explore the logging capabilities of some common applications.
Apache Log
Apache generates two types of logs to help provide valuable insights into the health of the Apache server, its resource utilization patterns, and associated metrics:
- Access logs include the server’s response codes, resource access information, time of access, source IP address, and other details. In Ubuntu, Apache access logs can be found at /var/log/apache2/access.log.
- Error logs log any errors that happen while the Apache server processes requests. These are very helpful to troubleshoot Apache server issues and can contain insights into the root cause. In Ubuntu machines, Apache error logs are available at /var/log/apache2/error.log.
Read the Apache logs guide to learn more.
NGINX Log
NGINX provides multiple log files that the administrator can configure to provide insights into activities on the server:
- Access logs provide information about the resource access pattern on the server, i.e., response code, user agent, source IP address, etc. The default log location is /var/log/nginx/access.log.
- Error logs contain all NGINX errors, related to both configuration or access. You can find them at /var/log/nginx/error.log.
If you’re using NGINX, read NGINX logging guide to learn more about how to work with NGINX logs.
Docker Log
As Docker containers are ephemeral, capturing the logs requires additional configuration.
The logs sent by containers to stdout or stderr should be forwarded by a logging driver to the log destination. The logs are streamed to the /var/lib/docker/containers/ directory of the container host machines. Access them using the following command:
$ docker logs [OPTIONS] <CONTAINER-NAME OR ID>
Read our blog post on Docker logs for a deep dive into Docker logging.
HAProxy Log
HAproxy logs provide useful information to troubleshoot the errors encountered by clients accessing the applications with an HAProxy frontend. In the event of an error, a corresponding log will be written in HAProxy log files. Ubuntu systems, you can locate this log at /var/log/haproxy.log.
Check out the HAProxy logging guide if HAProxy is part of your technology stack.
PostgreSQL Log
By default, PostgreSQL logs are streamed to stderr. The logs will, however, be written to the default OS log directory if the logging_collector parameter of the service is enabled. Log files of the PostgreSQL server in Ubuntu can be found at /var/log/postgresql/postgresql-x.x.main.log. X.x.
If you’re running PostgreSQL on Ubuntu, read our post on PostgreSQL logging to learn how to best handle its logs.
Other Useful Logs
In addition to the abovementioned logs, there are some additional Ubuntu server logs that the system generates but are not human-readable. These logs are also stored in the /var/log folder.
Login Failure Log
These logs contain information about the latest login failures in the system. The logs files are located at /var/log/faillog. The data here can be parsed using the faillog command to view the latest login failures.
Last Logins Log
This log lists the most recent logins to the system. The log file is located at /var/log/lastlogin. To parse the log and view a list of last logins to the system, use the following command:
lastlog | less
Login Records Log
This log stores information about users currently logged into the system. The log file is located at /var/log/wtmp and can be parsed by command line utilities such as the command who.
Logs provide a wealth of information to help troubleshoot issues or errors in the Ubuntu system or applications deployed in it. But to view and analyze the log files, additional tools are required, some of which I’ll cover here.
GNOME Logs App
GNOME is a commonly used desktop GUI interface for Ubuntu. In the latest versions of Ubuntu GNOME, a log viewer utility named “Logs” is available out of the box. It’s a GUI utility that helps you view a number of log files including your hardware, system, application, security, etc.
In the Ubuntu Dash, search for “Logs” to open the utility. The utility contains different tabs to view the logs. For example, you can click on the “System” tab to view the system logs. You can also view additional details about specific logs by clicking on them or search for logs using keywords.
Log File Viewer
The log file viewer is another GUI-based tool that lets you view and analyze logs and is available in older versions of Ubuntu. To access the utility, search for “Log Viewer” in the Ubuntu Dash. It has several categories like syslog, auth.log, kern.log, etc. listed on the left panel. You can use these to explore respective log types. You can also search for specific logs using keywords.
Terminal
Administrators comfortable with the command-line mode of analyzing logs can use the Ubuntu terminal. There are several commands available to view the log files in Ubuntu:
tail
to view the last few lines of the log filedmesg
to view log messages from Kernel’s buffercat
to open and view log filesmore
for when you want to browse through the log file one screen at a time; use space bar to go to the next pageless
also displays content of log files one screen at a time but has additional options to scroll forward, backward, and searchgrep
to search for specific keywords within a log; helpful if you are looking for information on specific incidents/errors in the log file, for example,dmesg |grep [keyword]
Journalctl Command
In earlier sections, I talked about journald, which is the systemd’s logging service or, in other words, the log management daemon in Ubuntu. Journalctl is the utility available for viewing and querying these logs since they’re stored in binary format by journald. You can use it without any parameters to view all the content of the systemd journal:
journalctl
As all the logs in the journal will be overwhelming to comprehend, use appropriate filters for the output. For instance, to view only messages within a specified timeframe, input the following command:
journalctl --since “2 hours ago”
To view logs associated with a specific systemd unit, use the -u switch. For example, to view logs from the MySQL service:
journalctl -u mysql.service
You can also view log messages related to activities of a specific user with the UID of the user as a switch, as shown below:
journalctl _UID=209
You cam use the –since and –until switches to view logs within a time period:
journalctl --since "2022-07-21 23:15:00" --until "2022-07-21 23:20:00"
Ubuntu System Logging Daemon Configuration
Syslogd, also known as sysklogd, is the default system logging daemon in Ubuntu. It performs the task of receiving messages from different system sources and sends them to the defined log file destination. Besides log messages, additional useful metadata like timestamps for when the logs are generated, source system hostnames, etc. are also captured in these log files.
You can configure the log files destination and the priority level of logs by editing the /etc/syslog.conf file. The selector field in the configuration file specifies the logging facility and the required log priority level. For example, you can choose the auth logging facility and the priority as “info” or “warning”.
The log destination file is specified by the action field. This could be the default Ubuntu server log files location, i.e., /var/log/syslog, or the name of a centralized logging server.
Ubuntu Log Rotation
With multiple logs generated every minute by different system and application components, the log file data can easily consume the entire disk space in your machine. So it’s important to compress and archive log files periodically. This process is called log rotation. Initially, logs are renamed after a specified timeframe and a new log file gets created for the latest logs. The logs are then compressed to save disk space. The utility used to manage this log rotation process is called “logrotate”.
Using Logrotate
Configure the logrotate utility via the file /etc/logrotate.conf. With the default settings in logrotate.conf, Ubuntu rotates log files weekly for files owned by the Ubuntu syslog server group and root user, with a default retention of four logs at a time.
When current logs are rotated, new empty log files are created to which the logs get streamed. The logrotate utility is invoked from the cron script /etc/cron.daily/logrotate.
You can add a log rotation configuration specific to applications to /etc/logrotate.d/. This is also where you can define the log rotation settings of system tools like APT, syslog, and dpkg or for specific applications like MySQL and Apache2.
Useful Commands for Working with Ubuntu Logs
Let’s look at some of the basic commands you can use in an Ubuntu terminal to explore log files.
Change Log Directory
The first step for accessing logs would be to switch to the log file directly using the following command:
cd /var/log
Search Log Files
Log files can contain a lot of information, so you might sometimes have to use the search option to drill down to the relevant log content, or you can use the grep command. For example, to search for all lines with the word “error” in a log file:
grep “error' kern.log
Edit Log Files
Simple text editors like nano let you edit and make changes in the log files. Note that to save changes, you will have to use the sudo command, which is not a recommended practice. A sample command for editing the log file is:
nano test.log
Monitor Log Files
As log files are continuously getting updated, use the following command to monitor those changes in real time:
tail -f error.log
While the utilities and tools discussed above can be helpful in analyzing logs in individual servers, when it comes to large-scale infrastructure deployments you need to consider custom-built solutions for Ubuntu log management and analysis like Sematext Logs. You may have to correlate events and logs across multiple systems, and the manual analysis of logs could become cumbersome.
We have also created a comprehensive guide to the best Ubuntu server monitoring tools that you may find handy.
Ubuntu Log Analysis and Monitoring with Sematext
Sematext Logs is a log management tool that can provide out-of-the-box integration for monitoring Ubuntu logs. It’s a full-spectrum solution, capable of collecting logs and events in real time from your Ubuntu systems, then parsing and analyzing them to give deeper visibility into your Ubuntu hosts. It allows you to set up threat or anomaly detection alerts to be warned about any potential issues upfront, which helps bring down troubleshooting time.
Simply install the Sematext agent for an intuitive UI that lets you select the type of logs you want to monitor to cut out the noise and focus on relevant information. For example, the agent is capable of collecting logs from important system components like systemd-journal as well as specific applications like NGINX.
Sematext – Ubuntu logs overview
Sematext – Ubuntu journald log view
You can configure the Ubuntu logging options, input, output, and parser in the agent configuration file as per your log monitoring requirements. Moreover, being part of the Sematext Cloud monitoring suite, it allows you to also correlate the data from your logs with additional context from performance metrics for faster troubleshooting and debugging. Sematext Cloud even helps you build an actionable dashboard for a bird’s-eye view of the state of your Ubuntu system.
Watch the video below to learn more about what Sematext Logs can do for you or jumpstart your logging journey by trying out the 14-day free trial!
Conclusion
Ubuntu logs provide a wealth of information that can help you troubleshoot your operating system as well as application errors. You need to select the right log files for review to reduce the turnaround time. Ensure that the log files are configured to capture the level of detail your organization requires. Ubuntu’s built-in tools for viewing logs can help with the first level of analysis. This approach works well if you have only a few systems to monitor.
However, large-scale log ingestion and management needs enterprise-scale specialized tools like Sematext Logs. When your Ubuntu landscape is expansive, this requirement becomes even more prevalent, especially due to the sheer volume of logs generated. Without the right tools at your disposal, finding the source of outages and correcting them could prove to be a difficult undertaking.
Author Bio
Jean-Christophe Dansereau
Jean-Christophe Dansereau is a Canadian software engineer and tech writer, specializing in backend and cloud engineering. He helps tech companies develop and scale applications and write technical blogs to allow organizations to communicate their message.
Start Free Trial