Configure Elasticsearch and filebeat for index Microsoft Internet Information Services (IIS) logs in Ingest mode.


Configure Elasticsearch and filebeat for index Microsoft Internet Information Services (IIS) logs in Ingest mode.

The configuration discussed in this article is for direct sending of IIs Logs via Filebeat to Elasticsearch servers in “ingest” mode, without intermediaries. If you use Logstash you may find the Template and grok filter used in Pipeline useful but the configuration will be different for Logstash.

Requeriments

First we need an Elasticsearch and Kibana running. We are going to see how configure filebeats and necessary pipelines and templates for Internet Information Server.

Required config for IIS

In IIS we need only config log properties for log all fields (select all fields) and in W3C format. It is very important for config to work. All templates and pipelines in this post are configured for this IIS log config.

Attention. Attached at the end of this post there is a zip file with all configuration files.

Some basic operations that you need to know.

We can visualize Elasticsearch existing index with command:

[root@es5 ~]# curl 'localhost:9200/_cat/indices?v'
health status index   uuid                   pri rep docs.count docs.deleted store.size pri.store.si                     ze
yellow open   .kibana cy6tNq1gTUWkPOs7Lu4FFg   1   1          1            0      3.1kb          3.1                     kb

We can delete an Elasticsearch index with:

curl -XDELETE 'http://localhost:9200/index_name-*/'

i.e. Deleting files for index “filebeat” (files filebeat-*)

curl -XDELETE 'http://localhost:9200/filebeat-*/'

View all pipelines from Kibana DEV Tools console

GET _ingest/pipeline/*

Delete pipeline from Kibana DEV Tools console.

DELETE _ingest/pipeline/pipeline-name

Delete index with this commands is not recommended in a productiĆ³n server. You must use a tool as Curator. In test / lab you can use -XDELETE option.

Internet Information server Pipeline for Elasticsearch Ingest mode

We need a Pipeline in Elasticsearch for “translate” log inputs for IIS to “fields” in ES. If you are using Logstash (and not Elastisearch Ingest Nodes) yo don’t use a Pipeline. Confiration is different and is not cover in this post but the GROK filter used in this Pipeline can be useful.

We must create our pipeline “iis_w3c” from Kibana DEV Tools console:

PUT _ingest/pipeline/iis_w3c
{
  "description": "iis_w3c",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{TIMESTAMP_ISO8601:date_time} %{WORD:s_sitename} %{HOSTNAME:s_computername} %{IP:s_ip} %{WORD:cs_method} %{URIPATH:cs_uri_stem} %{NOTSPACE:cs_uri_query} %{NUMBER:s_port} %{NOTSPACE:cs_username} %{IPORHOST:c_ip} %{NOTSPACE:cs_version} %{NOTSPACE:cs_user_agent} %{NOTSPACE:cs_cookie} %{NOTSPACE:cs_referer} %{NOTSPACE:cs_host} %{NUMBER:sc_status} %{NUMBER:sc_substatus} %{NUMBER:sc_win32_status} %{NUMBER:sc_bytes} %{NUMBER:cs_bytes} %{NUMBER:time_taken}"]
      }
    },
    {
      "date": {
        "field": "date_time",
        "formats": [ "yyyy-MM-dd HH:mm:ss" ]
      }
    }   
  ],
  "on_failure" : [
    {
      "set" : {
        "field" : "_index",
        "value" : "failed-{{ _index }}"
      }
    }
  ]
}

Now you can view pipelines in ES with:

GET _ingest/pipeline/*

** Attention ** In the pipeline is defined an Index “failed- *” that will be created in the case that the log lines indexed do not match the GROK regular expression. On this way we prevent Filebeat from stopping or discarding the sending of logs in the correct format to Elasticsearch.

Testing grok filter in IIS Pipeline

We can test the grok filter in the pipeline directly in Kibana DEV Tools console. You can change the “message” for a real line from your IIS Log file.

POST _ingest/pipeline/_simulate
{
  "pipeline" :
  {
  "description": "tets_iis",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:serviceName} %{HOSTNAME:virtual_host} %{IP:serverIP} %{WORD:method} %{URIPATH:uriStem} %{NOTSPACE:uriQuery} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientIP} %{NOTSPACE:protocolVersion} %{NOTSPACE:userAgent} %{NOTSPACE:cookie} %{NOTSPACE:referer} %{NOTSPACE:requestHost} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:win32response} %{NUMBER:bytesSent} %{NUMBER:bytesReceived} %{NUMBER:timetaken}"]
      }
    }]},
  "docs": [
    {
      "_index": "index",
      "_type": "type",
      "_id": "id",
      "_source": {
        "message": "2017-04-06 08:15:38 W3SVC1 server-iis8 10.10.48.91 GET /sga/pepito.html - 80 - 10.10.45.43 HTTP/1.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/57.0.2987.133+Safari/537.36 ROUTEID1=.server-001;+ROUTEID2=.server-002 https://www.fulanito.com/sga/pepito.html www.fulanito.com 200 0 0 51340 637 15"
      }
    }
  ]
}

The result must be a response from ES with all fields correctly identified.

{
  "docs": [
    {
      "doc": {
        "_index": "index",
        "_type": "type",
        "_id": "id",
        "_source": {
          "log_timestamp": "2017-04-06 08:15:38",
          "referer": "https://www.fulanito.com/sga/pepito.html",
          "win32response": "0",
          "requestHost": "www.fulanito.com",
          "timetaken": "15",
          "clientIP": "10.10.45.43",
          "serverIP": "10.10.48.91",
          "protocolVersion": "HTTP/1.1",
          "virtual_host": "server-iis8",
          "method": "GET",
          "cookie": "ROUTEID1=.server-001;+ROUTEID2=.server-002",
          "uriStem": "/sga/pepito.html",
          "userAgent": "Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/57.0.2987.133+Safari/537.36",
          "bytesSent": "51340",
          "message": "2017-04-06 08:15:38 W3SVC1 server-iis8 10.10.48.91 GET /sga/pepito.html - 80 - 10.10.45.43 HTTP/1.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/57.0.2987.133+Safari/537.36 ROUTEID1=.server-001;+ROUTEID2=.server-002 https://www.fulanito.com/sga/pepito.html www.fulanito.com 200 0 0 51340 637 15",
          "serviceName": "W3SVC1",
          "bytesReceived": "637",
          "uriQuery": "-",
          "port": "80",
          "response": "200",
          "subresponse": "0",
          "username": "-"
        },
        "_ingest": {
          "timestamp": "2017-05-17T14:05:32.641Z"
        }
      }
    }
  ]
}

Testing IIS Pipeline

We can test our “iis_w3c” pipeline directly with simulate.

POST _ingest/pipeline/iis_w3c/_simulate
{
  "docs": [
    {
      "_index": "index",
      "_type": "type",
      "_id": "id",
      "_source": {
        "message": "2017-04-06 08:15:38 W3SVC1 server-iis8 10.10.48.91 GET /sga/pepito.html - 80 - 10.10.45.43 HTTP/1.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/57.0.2987.133+Safari/537.36 ROUTEID1=.server-001;+ROUTEID2=.server-002 https://www.fulanito.com/sga/pepito.html www.fulanito.com 200 0 0 51340 637 15"
      }
    }
  ]
}

With identical result as previus grok filter test.

Template for Internet Information Server logs.

Now we need a template for IIS logs. The Template includes the creation of the required @timestamp field, the IP addresses with the IP type field and some numeric fields as such. All text fields will be of type “keyword” except the message that will be analyzed. The template is made from an original Filebeat. With this definition you will create the fields in ES with the appropriate type.

We can create template from DEV tools console, curl command or from filebeat agent. In this case we are going to create template from a file and load it in Elasticsearch with curl command.

In a file iis_access.template.json

{
  "mappings": {
    "_default_": {
      "_all": {
        "norms": false
      },
      "_meta": {
        "version": "5.3.1"
      },
      "dynamic_templates": [
        {
          "strings_as_keyword": {
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "match_mapping_type": "string"
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "c_ip": {
          "type": "ip"
        },
        "s_ip": {
          "type": "ip"
        },
        "sc_bytes": {
          "type": "integer"
        },
        "cs_bytes": {
          "type": "integer"
        },
        "time_taken": {
          "type": "integer"
        },
        "beat": {
          "properties": {
            "hostname": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "name": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "version": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        },
        "input_type": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "message": {
          "norms": false,
          "type": "text"
        },
        "offset": {
          "type": "long"
        },
        "source": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "tags": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "type": {
          "ignore_above": 1024,
          "type": "keyword"
        }
      }
    }
  },
  "order": 0,
  "settings": {
    "index.refresh_interval": "5s"
  },
  "template": "iis_access-*"
}

With this file in Elasticsearch server shell run:

curl -XPUT 'http://localhost:9200/_template/iis_access' -d@./iis_access.template.json

We get an create response like {"acknowledged":true}

We can view template now in ES.

curl -XGET 'http://localhost:9200/_template/iis_access?pretty'

Filebeat config in IIS server

Install filebeat as normal windows app and stop Filebeat service if it is running.

Minimal configuration file “filebeat.yml” can be:

filebeat.prospectors:
- input_type: log
  paths:
    - C:\inetpub\logs\LogFiles\*\*
  exclude_lines: ['#']

output.elasticsearch:
  hosts: ["stg-elasticsearch-c02-0001:9200"]
  index: "iis_access-%{+yyyy.MM.dd}"
  pipeline: iis_w3c
  template.enabled: false
  template.name: "iis_access"
  • In paths there are all directories (with patterns) with logs to index.
  • Is necessary exclude lines starting with ‘#’. IIS start log files with comments.
  • In ES section the Eslasticsearch host to send log files
  • The pipeline to use for ingest mode files in ES.
  • Not load local (dir) template.
  • Template name to use (iis_access) from ES for “mapping” fields.

Start Filebeat service and check status (started). Now the first is review filebeat logfile.

In “filebeat.log” we can see something as this:

2017-05-17T14:04:32+02:00 INFO States Loaded from registrar: 3
2017-05-17T14:04:32+02:00 INFO Loading Prospectors: 1
2017-05-17T14:04:32+02:00 INFO Prospector with previous states loaded: 0
2017-05-17T14:04:32+02:00 INFO Starting prospector of type: log; id: 12363095216683313965
2017-05-17T14:04:32+02:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017-05-17T14:04:32+02:00 INFO Starting Registrar
2017-05-17T14:04:32+02:00 INFO Start sending events to output
2017-05-17T14:04:32+02:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-05-17T14:04:32+02:00 INFO Harvester started for file: C:\inetpub\logs\LogFiles\....
2017-05-17T14:04:32+02:00 INFO Connected to Elasticsearch version 5.4.0

Check than there is a nex Index “iis_access-*” in Elasticsearch

[root@es5 files]#  curl 'localhost:9200/_cat/indices?v'
health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .kibana               cy6tNq1gTUWkPOs7Lu4FFg   1   1          1            0     12.9kb         12.9kb
yellow open   iis_access-2017.05.17 uHc5ipPJRU--R9cvF6h_IQ   5   1         87            0    180.9kb        180.9kb

Configure Index Pattern in Kibana

In Kibana we have to Configure a new Index pattern for view new IIS data. In Management / Configure an index pattern, field “Index name or pattern” we write “iis_access-*” and tabulate. Kibana find the “Time-field name” as “@timestamp”. Accept with create botton.

Elasticsearch new Index-Pattern IIS

Now, we can go to “Discover” tab and review all fields available for search and create Visualizations.

Elasticsearch Index for IIS

Leave a comment

Your email address will not be published. Required fields are marked *

*