This sample shows how to implement an Azure Monitoring and Analytics solution at large scale with Application Insights Client SDK pipeline leveraging Application Insights Analytics with OpenSchema and 3rd party services for real-enough-time dashboards.

The Visual Studio solution provides two sample console applications.

1. SampleApp.ConsoleAppCore: .NET Core App
2. SampleApp.ConsoleApp: .NET Framework App

Both console apps reference the Sample.OpenSchemas assembly, which shows how to implement custom schemas with Application Insights OpenSchema and leverage the custom extensions in the Microsoft.AzureCAT namespaces.

• The solution is built with Visual Studio 2015 and requires the .NET Core SDK. Modify global.json for the version you install.
• Open the .sln solution file in Visual Studio 2015.
• Select configuration solution platform of x64, not Any CPU.
• Build All.
• If the SampleApp.ConsoleApp build with errors, just Build again to get it to resolve assemblies. This is a known issue with combined use of .xproj and .csproj. If references need to be updated, the .csproj will have to be manually edited when referencing any .xproj.
• Copy the appsettings.json.clean to appsettings.json, then replace each {{Secret-Key}} with your values. Specifying these values assumes you have created an Application Insights application which gives you an Instrumentation Key.
• See below for further Configuration details.

Logging uses the Microsoft logging extensions to provide the base logging functionality, we add additional custom logging extensions to populate our event telemetry objects, and then pass them into the custom pipeline built with the Application Insights Client SDK. See either Program.cs in SampleApp.ConsoleAppCore or SampleApp.ConsoleApp project to create and use an ILogger.

The sample implements a custom pipeline that extends the standard Applications Insights Client SDK. The custom pipeline includes enrichment properties to the event object and adds the following processor sinks to support the following target services:

• Application Insights for Aggregated Metrics
• Application Insights OpenSchema custom schemas (LogOpenSchema, ExceptionsOpenSchema, TimedOperationOpenSchema)
• Elasticsearch uses the custom schemas (LogOpenSchema, ExceptionsOpenSchema but not TimedOperationOpenSchema)
• Graphite for Aggregated Metrics

To view the events written to the above target services use:

• Application Insights Analytics to view
  • Aggregated Metrics (customEvents schema which is one of the default schemas in Application Insights)
  • OpenSchema data sources (custom schemas created by you, 3 are provided in the sample: LogOpenSchema, ExceptionsOpenSchema, TimedOperationOpenSchema)
• Elasticsearch uses Kibana to view events
• Graphite uses Grafana to view events

An appsettings.json file is required for each application. Replace these values with yours in the sections for ApplicationInsights and TelemetryProcessorSinks.

"ApplicationInsights": {
    "InstrumentationKey": "{{Secret:AIInstrumentationKey}}",
    "TelemetryServiceEndpoint": "https://dc.services.visualstudio.com/v2/track",

    "InMemoryPublishingChannel": {
      "maxBacklog": 1000,
      "maxWindowCount": 500,
      "windowSize": "00:00:15"
    },

    "LogLevel": {
      "Default": "Information",
      "SampleAppCore": "Information"
    },

InMemoryPublishingChannel

LogLevel

One or more CategoryNames with their respective logging level.

AggMetrics, Graphite, ElasticSearch, OpenSchema

{
	"name": "AggMetrics",
	"classAssembly": "Microsoft.AzureCAT.AppInsights.AppInsightAggMetricSinkFactory, Microsoft.AzureCAT.AppInsights.Sink.AggregatedMetric, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
	"maxBacklog": 10000,
	"maxWindowCount": 1000,
	"windowSize": "00:00:15"
}

{
	"name": "graphite",
	"classAssembly": "Microsoft.AzureCAT.AppInsights.AppInsightGraphiteSinkFactory, Microsoft.AzureCAT.AppInsights.Sink.Graphite, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
	"server": "{{Secret:GraphiteServer}}",
	"port": 2003,
	"maxBacklog": 1000,
	"maxWindowCount": 1000,
	"windowSize": "00:00:05",
	"EventTelemetryProperties": [
	  "Environment",
	  "MachineRole",
	  "MachineName"	]
}

{
	"name": "elasticsearch",
	"classAssembly": "Microsoft.AzureCAT.AppInsights.AppInsightElasticSearchSinkFactory, Microsoft.AzureCAT.AppInsights.Sink.ElasticSearch, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
	"indexnamebase": "{{Secret:ElasticSearchIndexBase}}",
	"elasticsearchurl": "{{Secret:ElasticSearchUrl}}",
	"port": 9201,
	"username": "{{Secret:ElasticSearchUsername}}",
	"password": "{{Secret:ElasticSearchPassword}}",
	"maxBacklog": 1000,
	"maxWindowCount": 1000,
	"windowSize": "00:00:15",
}

{
	"name": "OpenSchema",
	"classAssembly": "Microsoft.AzureCAT.AppInsights.AppInsightBlobSinkFactory, Microsoft.AzureCAT.AppInsights.Sink.Blob, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
	"InstrumentationKey": "{{Secret:AIInstrumentationKey}}",

	"BlobPublisher": {
	  "StorageAccounts": [
		"DefaultEndpointsProtocol=https;AccountName={{Secret:BlobPublisherStorageAccountName1}};AccountKey={{Secret:BlobPublisherStorageAccountkey1}}"
	  ],
	  "BaseContainerName": "{{Secret:BaseContainerName}}"
	},

	"BatchBuffer": {
	  "maxBacklog": 10000,
	  "maxWindowCount": 1000,
	  "windowSize": "00:00:15"
	},

	"BlobSinkBuffer": {
	  "memoryBufferBytes": 20485760,
	  "bufferFlush": "00:00:05",
	  "useGzip": true
	},

	"SchemaIdList": {
	  "LogOpenSchema": {
		"Id": "{{Secret:LogOpenSchemaId}}",
		"TransformOutput": {
		  "name": "LogOpenSchema",
		  "classAssembly": "Sample.OpenSchemas.BlobSinkToOpenSchema, Sample.OpenSchemas"
		}
	  },
	  "TimedOperationOpenSchema": {
		"Id": "{{Secret:TimedOperationOpenSchemaId}}",
		"TransformOutput": {
		  "name": "TimedOperationOpenSchema",
		  "classAssembly": "Sample.OpenSchemas.BlobSinkToOpenSchema, Sample.OpenSchemas"
		}
	  },
	  "ExceptionsOpenSchema": {
		"Id": "{{Secret:ExceptionsOpenSchemaId}}",
		"TransformOutput": {
		  "name": "ExceptionsOpenSchema",
		  "classAssembly": "Sample.OpenSchemas.BlobSinkToOpenSchema, Sample.OpenSchemas"
		}
	  }
	}
}

BlobPublisher

BlobPublisher-StorageAccounts

1 or more Azure Storage Accounts

BatchBuffer

BlobSinkBuffer

SchemaIdList

For each custom OpenSchema (LogOpenSchema, TimedOperationOpenSchema, ExceptionsOpenSchema), enter the GUID as the Id from the Azure portal for the Application Insights OpenSchema Other Data Sources:

LogOpenSchema

TimedOperationOpenSchema

ExceptionsOpenSchema

TransformOutput

Using this samples assumes you are familiar with the general topics of monitoring, diagnostics, and analytics for Azure applications:

• Application Insights
• Analytics in Application Insights
• Application Insights Analytics is notified to import
• Patterns and practices: Monitoring and diagnostics guidance
• Blog post with details of this sample

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.