Databases, Systems & Networks

Récemment, je me suis aperçu que mon dernier fichier /var/log/auth.log n’avait pas été renouvelé depuis le 17/04/2011… Evidemment, ça n’aide pas à analyser sereinement les fichiers de logs, vu la taille que ça génère: 105306586 c’est-à-dire plus de 100Mo !

Je voulais conserver 26 semaines de logs pour analyser les tentatives d’intrusion et calculer leur évolution au fur et à mesure que je renforçais la protection de mon serveur.

Tout simple:

• créer un fichier séparé pour auth.log qu’on appelle /etc/logrotate.d/auth:

/var/log/auth.log

{
        rotate 26
        weekly
        missingok
        notifempty
        compress
        delaycompress
        sharedscripts
        postrotate
                reload rsyslog >/dev/null 2>&1 || true
        endscript
}

• retirer la référence à /var/log/auth.log dans /etc/logrotate.d/auth
• redémarrer le service rsyslog:

/etc/init.d/rsyslog restart

Lire la suite…

An important aspect of any firewall are the log files. Iptables on Linux provides logging functionality, however by default, it will get outputted to the /var/log/messages log file. This can clutter things up, and make it hard to check the logs.

If you want to change the file that IPTables logs to, you need to set up your iptables rules to output a log prefix. Rsyslog will then be configured to pick up this prefix, and output the information to a custom log file, containing just the iptables log information.
Install rsyslog if it is not already installed.

$ sudo apt-get install -y rsyslog

Lire la suite…

Linux operating system and many applications create special files commonly referred to as “logs” to record their operational events. These system logs or application-specific log files are an essential tool when it comes to understanding and troubleshooting the behavior of the operating system and third-party applications. However, log files are not precisely what you would call “light” or “easy” reading, and analyzing raw log files by hand is often time-consuming and tedious. For that reason, any utility that can convert raw log files into a more user-friendly log digest is a great boon for sysadmins.

logwatch is an open-source log parser and analyzer written in Perl, which can parse and convert raw log files into a structured format, making a customizable report based on your use cases and requirements. In logwatch, the focus is on producing more easily consumable log summary, not on real-time log processing and monitoring. As such, logwatch is typically invoked as an automated cron task with desired time and frequency, or manually from the command line whenever log processing is needed. Once a log report is generated, logwatch can email the report to you, save it to a file, or display it on the screen.

A logwatch report is fully customizable in terms of verbosity and processing coverage. The log processing engine of logwatch is extensible, in a sense that if you want to enable logwatch for a new application, you can write a log processing script (in Perl) for the application’s log file, and plug it under logwatch.

One downside of logwatch is that it does not include in its report detailed timestamp information available in original log files. You will only know that a particular event was logged in a requested range of time, and you will have to access original log files to get exact timing information.

Installing Logwatch

On Debian and derivatives:

# aptitude install logwatch

On Red Hat-based distributions:

# yum install logwatch

Configuring Logwatch

During installation, the main configuration file (logwatch.conf) is placed in /etc/logwatch/conf. Configuration options defined in this file override system-wide settings defined in /usr/share/logwatch/default.conf/logwatch.conf.

If logwatch is launched from the command line without any arguments, the custom options defined in /etc/logwatch/conf/logwatch.conf will be used. However, if any command-line arguments are specified with logwatch command, those arguments in turn override any default/custom settings in /etc/logwatch/conf/logwatch.conf.

In this article, we will customize several default settings of logwatch by editing /etc/logwatch/conf/logwatch.conf file.

Detail = <Low, Med, High, or a number>

“Detail” directive controls the verbosity of a logwatch report. It can be a positive integer, or High, Med, Low, which correspond to 10, 5, and 0, respectively.

MailTo = youremailaddress@yourdomain.com

“MailTo” directive is used if you want to have a logwatch report emailed to you. To send a logwatch report to multiple recipients, you can specify their email addresses separated with a space. To be able to use this directive, however, you will need to configure a local mail transfer agent (MTA) such as sendmail or Postfix on the server where logwatch is running.

Range = <Yesterday|Today|All>

“Range” directive specifies the time duration of a logwatch report. Common values for this directive are Yesterday, Today or All. When “Range = All” is used, “Archive = yes” directive is also needed, so that all archived versions of a given log file (e.g., /var/log/maillog, /var/log/maillog.X, or /var/log/maillog.X.gz) are processed.

Besides such common range values, you can also use more complex range options such as the following.

• Range = “2 hours ago for that hour”
• Range = “-5 days”
• Range = “between -7 days and -3 days”
• Range = “since September 15, 2014”
• Range = “first Friday in October”
• Range = “2014/10/15 12:50:15 for that second”

To be able to use such free-form range examples, you need to install Date::Manip Perl module from CPAN. Refer to this post for CPAN module installation instructions.

“Service” option specifies one or more services to monitor using logwath. All available services are listed in /usr/share/logwatch/scripts/services, which cover essential system services (e.g., pam, secure, iptables, syslogd), as well as popular application services such as sudo, sshd, http, fail2ban, samba. If you want to add a new service to the list, you will have to write a corresponding log processing Perl script, and place it in this directory.

If this option is used to select specific services, you need to comment out the line “Service = All” in /usr/share/logwatch/default.conf/logwatch.conf.

Format = <text|html>

“Format” directive specifies the format (e.g., text or HTML) of a logwatch report.

Output = <file|mail|stdout>

“Output” directive indicates where a logwatch report should be sent. It can be saved to a file (file), emailed (mail), or shown to screen (stdout).

Lire la suite…

Source: michael.bouvy.net

Update of December 6th : although Logstash does the job as a log shipper, you might consider replacing it with Lumberjack / Logstash Forwarder, which needs way less resources, and keep Logstash on your indexer to collect, transform and index your logs data (into ElasticSearch) : check out my latest blog post on the topic.

Kibana Dashboard

Even if you manage a single Linux server, you probably already know how hard it is to keep an eye on what’s going on with your server, and especially tracking logs data. And this becomes even worse when you have several (physical or virtual) servers to administrate.

Although Munin is very helpful monitoring various informations from my servers / VMs, I felt the need of something more, and bit less static / more interactive.

There are 3 kind of logs I especially wanted to track :

• Apache 2 access logs
• iptables logs
• Syslogs

After searching arround on the internet for a great tool that would help me, I read about the open source log management tool Logstash which seems to perfectly suit a (major) part of my needs : logs collecting / processing.

For the purpose of this post, I will take the following network architecture and assume and I want to collect my Apache, iptables, system logs from servers 1/2/3 (“shippers”) on server 4 (“indexer”) and visualize them :

As you can see, I am using 4 complementary applications, the role of each one being :

• Logstash : logs collector, processor and shipper (to Redis) on log “shippers” 1-3 ; logs indexer on server 4 (reads from Redis, writes to Elasticsearch)
• Redis : logs data broker, receiving data from log “shippers” 1-3
• Elasticsearch : logs data persistent storage
• Kibana : (time-based) logs data visualization (graphs, tables, etc.)

Lire la suite…

A syslog server represents a central log monitoring point on a network, to which all kinds of devices including Linux or Windows servers, routers, switches or any other hosts can send their logs over network. By setting up a syslog server, you can filter and consolidate logs from different hosts and devices into a single location, so that you can view and archive important log messages more easily.

On most Linux distributions, rsyslog is the standard syslog daemon that comes pre-installed. Configured in a client/server architecture, rsyslog can play both roles; as a syslog server rsyslog can gather logs from other devices, and as a syslog client, rsyslog can transmit its internal logs to a remote syslog server.

In this tutorial, we cover how to configure a centralized syslog server using rsyslog on Linux. Before we go into the details, it is instructive to go over syslog standard first.

Basic of Syslog Standard

When logs are collected with syslog mechanism, three important things must be taken into consideration:

• Facility level: what type of processes to monitor
• Severity (priority) level: what type of log messages to collect
• Destination: where to send or record log messages

Let’s take a look at how the configuration is defined in more detail.

The facility levels define a way to categorize internal system processes. Some of the common standard facilities in Linux are:

• auth: messages related to authentication (login)
• cron: messages related to scheduled processes or applications
• daemon: messages related to daemons (internal servers)
• kernel: messages related to the kernel
• mail: messages related to internal mail servers
• syslog: messages related to the syslog daemon itself
• lpr: messages related to print servers
• local0 – local7: messages defined by user (local7 is usually used by Cisco and Windows servers)

The severity (priority) levels are standardized, and defined by using standard abbreviation and an assigned number with number 7 being the highest level of all. These levels are:

• emerg: Emergency – 0
• alert: Alerts – 1
• crit: Critical – 2
• err: Errors – 3
• warn: Warnings – 4
• notice: Notification – 5
• info: Information – 6
• debug: Debugging – 7

Finally, the destination statement enforces a syslog client to perform one of three following tasks: (1) save log messages on a local file, (2) route them to a remote syslog server over TCP/UDP, or (3) send them to stdout such as a console.

In rsyslog, syslog configuration is structured based on the following schema.

[facility-level].[severity-level]  [destination]

Lire la suite…