Get started quickly with writing new connectors for Records365 using this .NET library.
To jump straight in to building a Connector, see the Quick Start Guide.
The Records365 Connectors SDK has the following features:
- Separate and invididually scalable responsiblities
- Clearly defined interfaces for implementing record ingestion logic
- Modularised and easily extendable
- Utilises Dependecy Injection
- Submits content to the Records365 Connector API with Back-Off & Retry logic
- Receive notifications from Records365 (via webhook or by polling the Connector API)
The Connectors SDK is consists of numerous packages that provide various functionality as required by a connector. It includes multiple resource provider implementations, giving you different options for building and deploying your connector.
This project uses semantic versioning which is automated through the build process.
Semantic Versioning: [MAJOR].[MINOR].[PATCH]
To increment various version segments, include "+semver: xxxx" in your commit message:
- MAJOR: use "+semver: major" or "+semver: breaking"
- MINOR: use "+semver: minor" or "+semver: feature"
- PATCH: use "+semver: patch" or "+semver: fix"
To explicitly set a version, tag the commit with the desired semantic version.
- RecordPoint.Connectors.SDK
- RecordPoint.Connectors.SDK.Abstractions
- RecordPoint.Connectors.SDK.Caching
- RecordPoint.Connectors.SDK.Client
- RecordPoint.Connectors.SDK.Client.Abstractions
- RecordPoint.Connectors.SDK.Configuration.AzurekeyVault
- RecordPoint.Connectors.SDK.ContentReport
- RecordPoint.Connectors.SDK.Databases
- RecordPoint.Connectors.SDK.Databases.AzureSql
- RecordPoint.Connectors.SDK.Databases.Cosmos
- RecordPoint.Connectors.SDK.Databases.LocalDb
- RecordPoint.Connectors.SDK.Databases.PostgreSql
- RecordPoint.Connectors.SDK.Databases.Sqlite
- RecordPoint.Connectors.SDK.Health.Windows
- RecordPoint.Connectors.SDK.Logging.Serilog
- RecordPoint.Connectors.SDK.Notifications
- RecordPoint.Connectors.SDK.Toggles.LaunchDarkly
- RecordPoint.Connectors.SDK.WebHost
- RecordPoint.Connectors.SDK.WorkQueue.AzureServiceBus
Records 365 is a Data Intelligent platform that analyses content from external content sources and produces various meta data about the content that can then be used within an organisation for classification.
A Connector is a set of services that integrate the content source to Records 365. These services a broken up in to functional domains that implement a given responsibility covering various aspects such as content discovery and submission.
The Connectors SDK consists of various BackgroundService implementations that provide the main execution layer for the connector features. These services can be included in your connector via dependency injction extentions available within the relevant SDK packages.
Each connector service typically implements logic that executes "Work Operations".
The Connectors SDK is based on a work system. There are different types of work that are responsible for fulfilling the logic required by the varying aspects of a Connector. Work typically contains a context for which the work is being executed within.
Each implementation for a type of work is a Work Operation. Depending on the logic of a Work Operation, it can be derived from one of the following base Work classes:
WorkBase<TParameter>: Single execution Work that can be run immediately from within the calling service instanceQueueableWorkBase<TParameter>: Work that is submitted to a queue for one-time execution by any consuming service instanceManagedQueueableWorkBase<TConfiguration, TState>: Work that is submitted to a queue for repeated executions over a long period of time
The Connectors SDK contains various 'Work Operation' implementations that provide the operational logic for the connector features. These operations can be included in your connector via dependency injection extensions available within the relevant SDK packages.
For a detailed explanation of every Work Operation within the SDK, see The Record Ingestion Process.
Queued Work is a type of work that is submitted to a queue for future execution on any available instance of the relevant consuming service. Each different type of work has its own queue on the underlying queueing resource so each connector responsbility can be self contained and independently scalable.
The Work Queue is implemented via the RecordPoint.Connectors.SDK.WorkQueue.AzureServiceBus package and uses Azure Service Bus as the underlying queueing resource.
A future package should be developed to provide RabbitMq as an underlying queueing resource for On-Premise connector deployments.
See Managed Work Queue
The work operations that derive from ManagedQueueableWorkBase are operations that run within a defined context and share state between subsequent executions.
Executions should be quick and process a limited amount of information. An instance of a work operation can be requeued within the same unmodifiable context, with modifiable state information being shared between executions. When an operation is "continued", the work is resubmitted to the queue for execution at a later time based on configurable delay settings, including exponential back-off.
Some work operations have a related work action that is required to be implemented within your connector. The work action is where you implement the logic to retrieve content/data from the content source for the given connector function, and return it to the SDK for processing.
There are various configurable options within the Connector SDK, with each functional aspect of the Connector SDK having its own configuration options.
There are some global configuration settings (default values shown):
"Connector": {
"BinariesEnabled": true,
"BinarySkipThreshold": 500,
"SubmissionEnabled": true,
"RetryOnFailure": true,
"MaxRetries": 5,
"RetryDelay": 30,
"ExponentialRetryDelay": true
}See ConnectorOptions
A database is used for storing state information such as connector configurations, work status, and channel information.
Some state information is also stored within work queue messages.
The SDK includes several different database providers allowing flexibility when implementing your Connector:
- RecordPoint.Connectors.SDK.Databases.AzureSql
- RecordPoint.Connectors.SDK.Databases.Cosmos
- RecordPoint.Connectors.SDK.Databases.LocalDb
- RecordPoint.Connectors.SDK.Databases.PostgreSql
- RecordPoint.Connectors.SDK.Databases.Sqlite
See Record.Connectors.SDK.Notifications
Records365 generates notifications for connectors that can require the connector to perform some spcified logic. For example, connector configuration chnages (add,edit,delete) are received via notifications from Records365, and instruct the connector to apply the configuration change.
There are two ways to receive notifications from Records365. You can instruct your connector to use either method by calling the relevant dependency injection extension from within the 'RecordPoint.Connectors.SDK.Notifications' package.
Push notifications are implemented via a web-hook to the Connector. Records365 will push notificatons directly to the Connector as they are generated.
Poll based notifications are implemented via a service that periodically calls the Records365 Connector Api. The current implementation of poll based notifications is functionaly limited in comparison to push notifications and should not be used until the Records365 functionality is improved.