services | platforms | author | date |
---|---|---|---|
application-insights |
dotnet |
michthub |
May, 2017 |
This sample shows how to implement an Azure Monitoring and Analytics solution at large scale with Application Insights Client SDK pipeline leveraging Application Insights Analytics with OpenSchema and 3rd party services for real-enough-time dashboards.
The Visual Studio solution provides two sample console applications.
- SampleApp.ConsoleAppCore: .NET Core App
- SampleApp.ConsoleApp: .NET Framework App
Both console apps reference the Sample.OpenSchemas
assembly, which shows how to implement custom schemas with Application Insights OpenSchema and leverage the custom extensions in the Microsoft.AzureCAT
namespaces.
- The solution is built with Visual Studio 2015 and requires the .NET Core SDK. Modify
global.json
for the version you install. - Open the
.sln
solution file in Visual Studio 2015. - Select configuration solution platform of
x64
, notAny CPU
. - Build All.
- If the
SampleApp.ConsoleApp
build with errors, just Build again to get it to resolve assemblies. This is a known issue with combined use of.xproj
and.csproj
. If references need to be updated, the.csproj
will have to be manually edited when referencing any.xproj
. - Copy the
appsettings.json.clean
toappsettings.json
, then replace each{{Secret-Key}}
with your values. Specifying these values assumes you have created an Application Insights application which gives you an Instrumentation Key. - See below for further Configuration details.
Logging uses the Microsoft logging extensions to provide the base logging functionality, we add additional custom logging extensions to populate our event telemetry objects, and then pass them into the custom pipeline built with the Application Insights Client SDK.
See either Program.cs
in SampleApp.ConsoleAppCore
or SampleApp.ConsoleApp
project to create and use an ILogger
.
The sample implements a custom pipeline that extends the standard Applications Insights Client SDK. The custom pipeline includes enrichment properties to the event object and adds the following processor sinks to support the following target services:
- Application Insights for Aggregated Metrics
- Application Insights OpenSchema custom schemas (LogOpenSchema, ExceptionsOpenSchema, TimedOperationOpenSchema)
- Elasticsearch uses the custom schemas (LogOpenSchema, ExceptionsOpenSchema but not TimedOperationOpenSchema)
- Graphite for Aggregated Metrics
To view the events written to the above target services use:
- Application Insights Analytics to view
- Aggregated Metrics (customEvents schema which is one of the default schemas in Application Insights)
- OpenSchema data sources (custom schemas created by you, 3 are provided in the sample: LogOpenSchema, ExceptionsOpenSchema, TimedOperationOpenSchema)
- Elasticsearch uses Kibana to view events
- Graphite uses Grafana to view events
An appsettings.json
file is required for each application. Replace these values with yours in the sections for ApplicationInsights and TelemetryProcessorSinks.
"ApplicationInsights": {
"InstrumentationKey": "{{Secret:AIInstrumentationKey}}",
"TelemetryServiceEndpoint": "https://dc.services.visualstudio.com/v2/track",
"InMemoryPublishingChannel": {
"maxBacklog": 1000,
"maxWindowCount": 500,
"windowSize": "00:00:15"
},
"LogLevel": {
"Default": "Information",
"SampleAppCore": "Information"
},
Field | Values/Types | Required | Description |
---|---|---|---|
InstrumentationKey |
"{{Secret:AIInstrumentationKey}}" | Yes | Application Insights application instrumentation key. |
TelemetryServiceEndpoint |
url | Yes | Default Application Insights endpoint. |
InMemoryPublishingChannel |
JSON | No | Task DataFlow buffer block max sizes and timers. |
LogLevel |
JSON | No | Category names and their logging level. |
InMemoryPublishingChannel
Field | Values/Types | Required | Description |
---|---|---|---|
maxBacklog |
Integer | No | Buffer size of events for Application Insights. |
maxWindowCount |
Integer | No | Batch size of events before events are processed. |
windowSize |
Integer | No | Timer window size in HH:MM:SS format before events are processed. |
LogLevel
One or more CategoryNames with their respective logging level.
Field | Values/Types | Required | Description |
---|---|---|---|
Category Name | Trace,Debug,Information,Warning,Error,Critical,None | No | Default is the default category name with default of level Error. |
AggMetrics, Graphite, ElasticSearch, OpenSchema
{
"name": "AggMetrics",
"classAssembly": "Microsoft.AzureCAT.AppInsights.AppInsightAggMetricSinkFactory, Microsoft.AzureCAT.AppInsights.Sink.AggregatedMetric, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
"maxBacklog": 10000,
"maxWindowCount": 1000,
"windowSize": "00:00:15"
}
Field | Values/Types | Required | Description |
---|---|---|---|
name |
"AggMetrics" | Yes | Aggregated Metrics from ILogger.LogMetrics extension. |
classAssembly |
Implementation | Yes | ITelemetryProcessorFactory returns ITelemetryProcessor. |
maxBacklog |
Integer | No | Buffer size of metric events to be aggregated. |
maxWindowCount |
Integer | No | Batch size of metric events to be aggregated before events are processed. |
windowSize |
Integer | No | Timer window size in HH:MM:SS format before events are processed. |
{
"name": "graphite",
"classAssembly": "Microsoft.AzureCAT.AppInsights.AppInsightGraphiteSinkFactory, Microsoft.AzureCAT.AppInsights.Sink.Graphite, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
"server": "{{Secret:GraphiteServer}}",
"port": 2003,
"maxBacklog": 1000,
"maxWindowCount": 1000,
"windowSize": "00:00:05",
"EventTelemetryProperties": [
"Environment",
"MachineRole",
"MachineName" ]
}
Field | Values/Types | Required | Description |
---|---|---|---|
name |
"graphite" | Yes | Aggregated Metrics from ILogger.LogMetrics extension. |
server |
"{{Secret:GraphiteServer}}"" | Yes | Replace with server dns name, "mygraphite.eastus.cloudapp.azure.com". |
port |
Integer | No | Port to write metrics to Graphite (default 2003). |
maxBacklog |
Integer | No | Buffer size of metric events to be aggregated. |
maxWindowCount |
Integer | No | Batch size of metric events to be aggregated before events are processed. |
windowSize |
Integer | No | Timer window size in HH:MM:SS format before events are processed. |
EventTelemetryProperties |
JSON array | No | Prefix properties to build the Graphite metric name. |
{
"name": "elasticsearch",
"classAssembly": "Microsoft.AzureCAT.AppInsights.AppInsightElasticSearchSinkFactory, Microsoft.AzureCAT.AppInsights.Sink.ElasticSearch, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
"indexnamebase": "{{Secret:ElasticSearchIndexBase}}",
"elasticsearchurl": "{{Secret:ElasticSearchUrl}}",
"port": 9201,
"username": "{{Secret:ElasticSearchUsername}}",
"password": "{{Secret:ElasticSearchPassword}}",
"maxBacklog": 1000,
"maxWindowCount": 1000,
"windowSize": "00:00:15",
}
Field | Values/Types | Required | Description |
---|---|---|---|
name |
"elasticsearch" | Yes | Elasticsearch telemetry processor sink. |
classAssembly |
Implementation | Yes | ITelemetryProcessorFactory returns ITelemetryProcessor. |
indexnamebase |
"{{Secret:ElasticSearchIndexBase}}" | Yes | Elasticsearch indexbase, date will be appended to the name (_index). |
typename |
"{{Secret:ElasticSearchTypeName}}" | No | Elasticsearch type name (_type). |
elasticsearchurl |
"{{Secret:ElasticSearchUrl}}" | Yes | Elasticsearch server, "http://myelasticsearch.eastus.cloudapp.azure.com". |
port |
Integer | No | Port to batch load data into ElasticSearch (9202). |
username |
"{{Secret:ElasticSearchUsername}}" | No | Elasticsearch user name, if not provided user id and password won't be used. |
password |
"{{Secret:ElasticSearchPassword}}" | No | Elasticsearch password, default is none. |
maxBacklog |
Integer | No | Buffer size of events to be sent elasticsearch. |
maxWindowCount |
Integer | No | Batch size of events before events are processed. |
windowSize |
Integer | No | Timer window size in HH:MM:SS format before events are processed. |
{
"name": "OpenSchema",
"classAssembly": "Microsoft.AzureCAT.AppInsights.AppInsightBlobSinkFactory, Microsoft.AzureCAT.AppInsights.Sink.Blob, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
"InstrumentationKey": "{{Secret:AIInstrumentationKey}}",
"BlobPublisher": {
"StorageAccounts": [
"DefaultEndpointsProtocol=https;AccountName={{Secret:BlobPublisherStorageAccountName1}};AccountKey={{Secret:BlobPublisherStorageAccountkey1}}"
],
"BaseContainerName": "{{Secret:BaseContainerName}}"
},
"BatchBuffer": {
"maxBacklog": 10000,
"maxWindowCount": 1000,
"windowSize": "00:00:15"
},
"BlobSinkBuffer": {
"memoryBufferBytes": 20485760,
"bufferFlush": "00:00:05",
"useGzip": true
},
"SchemaIdList": {
"LogOpenSchema": {
"Id": "{{Secret:LogOpenSchemaId}}",
"TransformOutput": {
"name": "LogOpenSchema",
"classAssembly": "Sample.OpenSchemas.BlobSinkToOpenSchema, Sample.OpenSchemas"
}
},
"TimedOperationOpenSchema": {
"Id": "{{Secret:TimedOperationOpenSchemaId}}",
"TransformOutput": {
"name": "TimedOperationOpenSchema",
"classAssembly": "Sample.OpenSchemas.BlobSinkToOpenSchema, Sample.OpenSchemas"
}
},
"ExceptionsOpenSchema": {
"Id": "{{Secret:ExceptionsOpenSchemaId}}",
"TransformOutput": {
"name": "ExceptionsOpenSchema",
"classAssembly": "Sample.OpenSchemas.BlobSinkToOpenSchema, Sample.OpenSchemas"
}
}
}
}
Field | Values/Types | Required | Description |
---|---|---|---|
name |
"OpenSchema" | Yes | Application Insights OpenSchema telemetry processor sink. |
classAssembly |
Implementation | Yes | ITelemetryProcessorFactory returns ITelemetryProcessor. |
InstrumentationKey |
"{{Secret:AIInstrumentationKey}}" | Yes | Application Insights application instrumentation key. |
BlobPublisher
Field | Values/Types | Required | Description |
---|---|---|---|
StorageAccounts |
"Array of Azure Storage Accounts" | Yes | Array of Azure Storage Accounts. |
BaseContainerName |
"{{Secret:BaseContainerName}}" | Yes | Blob container name, unique per hour. |
BlobPublisher-StorageAccounts
1 or more Azure Storage Accounts
Field | Values/Types | Required | Description |
---|---|---|---|
AccountName |
"{{Secret:BlobPublisherStorageAccountName1}}" | Yes | Azure Storage Account Name. |
AccountKey |
{{Secret:BlobPublisherStorageAccountkey1}} | Yes | Azure Storage Account Key. |
BatchBuffer
Field | Values/Types | Required | Description |
---|---|---|---|
maxBacklog |
Integer | No | Buffer size of events to be written to memory buffer (BlobSinkBuffer). |
maxWindowCount |
Integer | No | Batch size of events written to memory buffer. |
windowSize |
Integer | No | Timer window size in HH:MM:SS format before events are processed. |
BlobSinkBuffer
Field | Values/Types | Required | Description |
---|---|---|---|
memoryBufferBytes |
Integer | No | Max memory buffer size in bytes of blob to be written and ingested (imported) into Application Insights OpenSchema. |
bufferFlush |
Integer | No | Timer window size in HH:MM:SS format before memory buffer is flushed. |
useGzip |
Boolean | No | Zip the blob, default is false. |
SchemaIdList
For each custom OpenSchema (LogOpenSchema, TimedOperationOpenSchema, ExceptionsOpenSchema), enter the GUID as the Id from the Azure portal for the Application Insights OpenSchema Other Data Sources:
LogOpenSchema
Field | Values/Types | Required | Description |
---|---|---|---|
Id |
"{{Secret:LogOpenSchemaId}}" | Yes | Schema Id GUID from the OpenSchema Other Data Sources. |
TransformOutput |
JSON | Yes | Transform (ITransformOutput) implementation to convert EventTelemetry to OpenSchema. |
TimedOperationOpenSchema
Field | Values/Types | Required | Description |
---|---|---|---|
Id |
"{{Secret:TimedOperationOpenSchemaId}}" | Yes | Schema Id GUID from the OpenSchema Other Data Sources. |
TransformOutput |
JSON | Yes | Transform (ITransformOutput) implementation to convert EventTelemetry to OpenSchema. |
ExceptionsOpenSchema
Field | Values/Types | Required | Description |
---|---|---|---|
Id |
"{{Secret:ExceptionsOpenSchemaId}}" | Yes | Schema Id GUID from the OpenSchema Other Data Sources. |
TransformOutput |
JSON | Yes | Transform (ITransformOutput) implementation to convert EventTelemetry to OpenSchema. |
TransformOutput
Field | Values/Types | Required | Description |
---|---|---|---|
name |
String | Yes | Schema class name matching LogTypes (LogOpenSchema, TimedOperationOpenSchema, ExceptionsOpenSchema). |
classAssembly |
Implementation | Yes | Transform (ITransformOutput) implementation to convert EventTelemetry to OpenSchema. |
Using this samples assumes you are familiar with the general topics of monitoring, diagnostics, and analytics for Azure applications:
- Application Insights
- Analytics in Application Insights
- Application Insights Analytics is notified to import
- Patterns and practices: Monitoring and diagnostics guidance
- Blog post with details of this sample
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.