This is a POC of a scheduler service using Hangfire and System.Composition.

This POC is implemented as a Windows Service (that can also be run as a console app) that hosts:

• An instance of Hangfire to schedule and execute background tasks.
• The Hangfire Web Dashboard.
• A WebAPI instance to administer the scheduled recurring jobs.

All of the scheduled tasks are packaged as plug-ins that are dynamically loaded and invoked by the scheduler. This implies there are NO static, compile-time dependencies between the scheduler and the tasks it executes.

Hangfire is an OpenSource .NET Core library similar to Celery for python that can add support for background processing to .NET applications.

In this POC we are leveraging it support recurring (scheduled) jobs.

Note: This POC used Hangfire's InMemory Storage option. In real-world scenarios this should be replaced with a persistent datastore (ex. MSSQL Server)

Note: Hangfire is horizontally scaleable by running additional Windows Service instances. Typically these would be executed on different physical or virtual servers to avoid processer thread contention with this instance.

Here are some useful link when running the POC locally

Note: The HTTP port is set in the hosting exe's appsettings.json file

The hosting process (EXE) appsettings.json file contains configration information for Hangfire and for Kafka. Kafka can be configured centrally and is passed into each plug-in.

{
  "kafkaConfig": {
    "bootstrapServers": "localhost:9092",
    "schemaRegistry": "localhost:8081"
  },
  "hangfireConfig": {
    "isUseSSL": true,
    "dashboardPortNumber": "5000",
    "isDashboardRemoteAccessEnabled": true,
    "workerCount": -1,
    "pollIntervalInSecs": 15
  },
  "AllowedHosts": "*"
}

Scheduled jobs all implement an interface (located in a shared assy) and are marked-up with a set of attributes that allow the runtime to find them.

Note: The string associated with the ExportMetadata attribute must be globally unique and is used to identify a plug-in

Note: The plug-in assembly and its dependencies would typically be copied to a subfolder of the hosting EXE.

Each plug-in supports its own private configuration that can be loaded from a local appsettings.json file

Plug-ins can write log messages directly to the Hangfire Console using standard methods on the ILogger interface that is passed in on the Execute method.

The scheduled jobs can be fully administered (add, remove, list) using a webAPI that includes a Swagger Website.

Using the POST endpoint you can pass the data required to schedule a reoccurring task.

{
  "job_id": "JOB001",
  "job_description": "Check for Invoices every minute",
  "job_plugin_type": "EventTypeA",
  "cron_schedule": "0-59 * * * MON,TUE,WED,THU,FRI",
  "schedule_time_zone": "EST"
}

Note: This plug-in is uniquely identified as EventTypeA and will execute every minute of every hour Mon-Fri. Schedules are defined using CRON syntax.

The scheduled jobs can be viewed on the Hangfire Dashboard on the Recurring Jobs tab

The scheduled jobs execution history can be viewed on the Hangfire Dashboard on the Jobs tab