A Native Windows to Redis/Elasticsearch Logstash Agent which runs as a service.

TimberWinR is a native .NET implementation utilizing Microsoft's LogParser. This means no JVM/JRuby is required, and LogParser does all the heavy lifting. TimberWinR collects the data from LogParser and ships it to Logstash via Redis (or can ship direcly to Elasticsearch)

View Version History

TimberWinR uses a configuration file to control how the logs are collected, filtered and shipped off.
These are broken down into:

1. Inputs (Collect data from different sources)
2. Filters (Are applied to all Inputs)
3. Outputs (e.g. Redis, Elasticsearch, Stdout, StatsD)

Please use the TimberWinR Google Group for discussion and support:

https://groups.google.com/forum/#!forum/timberwinr

Latest Build:

The current supported Input format sources are:

1. Logs (Files, a.k.a Tailing a file)
2. Tcp (listens on TCP port for JSON messages)
3. IISW3C(Internet Information Services W3C Format)
4. WindowsEvents (Windows Event Viewer)
5. Stdin (Standard Input for Debugging)
6. W3C(Internet Information Services W3C Advanced/Custom Format)
7. Udp (listens for UDP on port for JSON messages)
8. TailFiles (Tails log files efficiently)
9. Generator (Generate logs for testing New)

The current list of supported codecs are:

1. Multiline

The current list of supported filters are:

1. Grok
2. Mutate
3. Date
4. Json
5. GeoIP

Note that there are now two syntaxes for filters, array and single, if you want more than one filter of the same type then you must use the array tag instead of the singular tag. i.e:

A single Json filter using the single tag (this is only provided as a convienience, the array syntax is preferred)

"Filters": [          
    {
        "json": {
            "type": "Win32-FileLog",
            "condition": "\"[logSource]\" == \"dev\"",
            "source": "Text",
            "add_field": [
                "_index",
                "dev-%{yyyy.MM.dd}"
            ]
        }
    }
]

Multiple Json filters must use the jsonFilters and array syntax, also mutateFilters, grokFilters, dateFilters, geoipFilters.

"Filters": [
    {
        "jsonFilters": [
            {
                "type": "Win32-FileLog",
                "condition": "\"[logSource]\" == \"dev\"",
                "source": "Text",
                "add_field": [
                    "_index",
                    "dev-%{yyyy.MM.dd}"
                ]
            },
            {
                "type": "Win32-FileLog",
                "condition": "\"[logSource]\" == \"sta\"",
                "source": "Text",
                "add_field": [
                    "_index",
                    "sta-%{yyyy.MM.dd}"
                ]
            }
        ]
    }
]

Since TimberWinR only ships to Redis and Elasticsearch, the format generated by TimberWinR is JSON. All fields referenced by TimberWinR can be represented as a JSON Property or Array.

1. Redis
2. Elasticsearch
3. Stdout
4. File
5. StatsD

TimberWinR reads a JSON configuration file, an example file is shown here:

{
"TimberWinR": {
    "Inputs": {
        "WindowsEvents": [
            {
                "source": "System,Application",
                "binaryFormat": "PRINT",
                "resolveSIDS": true
            }
        ]
    },
    "Filters": [          
        {
            "grok": {
                "condition": "\"[type]\" == \"Win32-Eventlog\"",
                "match": [
                    "Message",
                    ""
                ],                   
                "remove_field": [
                    "ComputerName"                   
                ]
            }
        }
    ],
    "Outputs": {
        "Redis": [
            { 
                "_comment": "Shuffle these hosts",
                "host": [
                   "server1.host.com", 
                   "server2.host.com"
                ]
            }
        ]
    }
}

This configuration:

1. Inputs: Events from the Windows Event Logs (System, Application)
2. Filters: Removes the ComputerName field
3. Sends the event to Redis services (server1.host.com, server2.host.com) in a shuffling manner (balanced).

If installing from source, you must first install LogParser, then install TimberWinR. Install LogParser from here:

Install LogParser from Microsoft.

After installing, follow the remaining directions here. If you install from Chocolatey then LogParser will automatically be installed first.

You can run TimberWinR interactively when you are developing your JSON config file, to do so use the following options:

TimberWinR.ServiceHost.exe -configFile:myconfig.json -logLevel:Debug

TimbeWinR Chocolatey

TimberWinR uses TopShelf to install as a service, so all the documentation for installing and configuring the service is show here TopShelf Doc

Specifically the command line options are listed here in Topshelf Command-Line Reference guide.

Install and set to Automatically Start the service:

; Install Service (will autostart on reboot)
TimberWinR.ServiceHost.exe install --autostart
; Start the Service
TimberWinR.ServiceHost.exe start

To Start/Stop the Service from the Command Line

TimberWinR.ServiceHost.exe start
TimberWinR.ServiceHost.exe stop

Alternatively you can use the Services Control Panel.

TimberWinR.ServiceHost.exe [options]

Options:
-logDir:           Specifies the directory where TimberWinR will write its log file TimberWinR.txt
                   Default is -logDir:"C:\logs"
-logLevel:         Specifies the logging level for TimberWinR
                   Legal Values: Trace|Debug|Info|Warn|Error|Fatal|Off
                   Default is -logDir:Info
-configFile:       Specifies the path to the JSON config file, or directory which contains .json file(s).
                   Default is -configFile:default.json
-diagnosticPort:   Specifies the diagnostic port which can be used to get a health check of the service.
                   Default Port is 5141, A value of 0 will disable it.  Open a browser
                      http://localhost:5141

This may be a single .json file or a directory containing .json file(s). If it is a directory, all files will be read and processed, the order in which the files will be processed will match the alphabetical order on disk.

If you really just want to try it out, grab the binary distribution, extract the .zip file into a directory, e.g. C:\TimberWinR

Grab the JSON example file and place it into C:\TimberWinR\default.json. Edit the default.json file and change the Redis instance to match yours, replace 'tstlexiceapp006.mycompany.svc' with the IP or DNS name of the machine running redis. Fire up the collector, enable the verbose debugging to see some Windows Events.

TimberWinR.ServiceHost.Exe -configFile:default.json -logLevel:Debug

You should see

To run it as a service

TimberWinR.ServiceHost.exe install --autostart
TimberWinR.ServiceHost.exe start

TimberWinR is distributed as an installable package via Chocolatey, and it is dependent on

TimbeWinR Chocolatey

After you install it via Chocolatey, you will have a folder:

C:\Program Files (x86)\TimberWinR

TimberWinR will be configured to read the file default.json located in the folder above. You can edit this file to customize your installation.

Use these commands to Stop/Start the service.

sc stop TimberWinR ; stop the service
sc start TimberWinR; start the service