Configuring the system log
Configuring the system log
Syslog is a de facto standard solution for logging messages in UNIX-derived systems, such as the operating system of the Appliance. Programs use the syslog system call to send arbitrary messages to the syslog service. In addition to the message itself, two parameters are provided to the syslog system call: the facility and the level. The facility refers to the type of program that required the logging of a message. Facilities are named after typical UNIX services such as mail, ftp, or cron, subsystems such as the kernel, the printer, or the clock, and others are reserved for local use. The level indicates the importance or seriousness of the message. Possible values for the level are critical, warning, notice, etc.
The Nexthink components in the Appliance use the system log service to keep a record of significant occurrences, including:
- Audit trail events
- System alerts
- Investigation-based global alerts
- Internal state of the Engine
For writing to the system log, the Appliance relies on the rsyslog package, which has become the default logging service in many Linux distributions. Although it adds new advanced features, rsyslog still keeps backwards compatibility with the configuration files of the original syslog daemon. If you are familiar with the configuration of rsyslog, you may easily customize the output of the logs written by the Nexthink components and adapt them to your needs.
From this point on, we may refer to rsyslog as syslog when we talk about the logging service in general and not about specific features of rsyslog.
Default configuration and log files
The configuration file for rsyslog is found in /etc/rsyslog.conf. For the sake of clarity, the specific modifications of Nexthink to the configuration of rsyslog are stored in a separate file, which is found in /etc/nexthink/nx_rsyslog.conf. This file is applied to the main configuration file by means of an include directive in /etc/rsyslog.conf.
The default configuration of Nexthink dispatches log messsages to different files depending on their content. Find these log files under /var/log/nexthink:
|audit.log||Audit trail events|
|engine.log||Internal state of the Engine|
To write information about its internal state, the Portal does not use the syslog service but its own logging tools. The internal logs of the Portal are found under /var/nexthink/portal/log.
Nexthink uses UTF-8 encoding for its log messages. Rsyslog preserves the encoding.
Configuration of alerts
For its own logging purposes, Nexthink uses two of the facilities reserved for local use by syslog: local5 and local6. The choice of facility has an influence only on the format of logs linked to global alerts. Choose between modern (local6) or legacy (local5) format for the log messages of global alerts by setting a parameter in the configuration file of the Engine:
<syslog> <legacy_alert_format>true</legacy_alert_format> </syslog>
When the parameter is set to true, the Engine logs all syslog-enabled global alerts to the local5 facility in the legacy format. When set to false, the Engine logs the same alerts in modern format to the local6 facility. In either case, the Engine uses the level notice for logging alerts.
The part of the syslog configuration file /etc/nexthink/nx_rsyslog.conf which is relevant for alerts is shown below:
$template RFC5424format,"<%pri%>1 %timestamp:::date-rfc3339% %hostname% %programname% %procid%%msg%\n" ... # alerts local5.=notice -/var/log/nexthink/alert.log; ... # alerts local6.=notice -/var/log/nexthink/alert.log; RFC5424format
The first line defines an output format for syslog messages by means of a template. The template is named RFC5424format because it follows the recommended format for syslog messages which is described in the most recent Internet standard about the syslog protocol: RFC 5424. The template defines the output to be composed of a priority number followed by the timestamp, the host name, the program name, the id of the process which issued the syslog message and the message itself. Once defined in this way, a template can be applied to one or several message filters.
For alerts, you can see that we declare two filters in the syslog configuration file, depending on the facility specified to log the alert. Both filters are instructed to write their output to the same file: /var/log/nexthink/alert.log. The minus sign before the file name is there to improve the performance of the syslog daemon. It indicates that syslog output to the file is buffered, so the syslog system will not directly write to the filesystem but to a buffer in memory and then really write to the disk once the buffer is full. The two filters however accept messages from different facilities. If the facility used is local5, rsyslog will use the default syslog output format. On the other hand, if the facility used is local6, rsyslog will use the output format defined by the template RFC5424format for every logged alert.
Logging to a remote server
The syslog protocol lets you send log messages through the network to be consumed by syslog servers other than the local Appliance.
To send log messages to a remote syslog server, modify each line in the syslog configuration file of Nexthink by substituting the name of the log file for the name or IP address of the receiving server. The name of the server must be preceded by a single or a double at-sign (@ or @@), depending on whether you want to send the log messages via UDP or TCP, respectively. Follow the name or IP address of the remote server by a colon (:) and the port number where the server is listening for syslog messages. For example, to send log messages of alerts to remote servers:
# Send alert logs to file local6.=notice -/var/log/nexthink/alert.log; RFC5424format # Send alert logs to a server listening to UDP port 514 local6.=notice @udp-server.example.com:514; RFC5424format # Send alert logs to a server listening to TCP port 10514 local6.=notice @@tcp-server.example.com:10514; RFC5424format
Note that you do not have to choose between saving the logs in a file and send them to a remote server. It is possible to do both by repeating the same line in the syslog configuration changing the destination of the logs. Check the rsyslog documentation for options when sending log messages through the network, specially via TCP
Logging accesses to the CLI
Besides user access to the Nexthink components such as the Finder and the Portal, the access to the command line interface of the Appliance is an event of interest in the audit trail.
To filter the syslog messages related to accesses to the CLI of the Appliance and send them to a destination of your choice, specify the programs that control the command line inside conditional statements in the configuration file of syslog:
# Log access to the CLI if $programname == 'sshd' then -/var/log/nexthink/audit.log if $programname == 'sudo' then @udp-server.example.com:514 if $programname == 'login' then @@tcp-server.example.com:10514
These programs control remote access (sshd) to the Appliance, logging in (login) to the Appliance, and execute as the superuser (sudo) in the Appliance. In the example above, each program is sending its output to a different destination, but you can send the output of all programs to the same destination.
Restarting the syslog service
After any modification to the configuration file of syslog, restart the service for the changes to be effective:
sudo systemctl restart rsyslog