Daisy is a framework for building data integrations in a structured manner with minimal overhead. Integrations are implemented as series of steps where the result of each step is the input of the next. These steps are composed into strongly-typed chains using a fluent interface.
Example of a chain:
public class ExampleChain : ChainBuilder<Signal>
{
public override string Name { get; } = "ExampleChain";
public override void ConfigureSources(SourceConnectorCollection<Signal> sources)
{
sources.Add<NCrontabSource>("Cron");
}
public override void ConfigureRootConnector(IConnectorLinker<Signal> root)
{
root.Link<GetDataRecords, DataRecord[]>()
.Map(MapRecords)
.Each(each => each
.Link<PersistRecord, RecordModel>()
.Link<SendEmail, RecordModel>()
);
}
private static RecordModel[] MapRecords(DataRecord[] input)
{
// …
}
}- Install package
dotnet add package DaisyFx- Define links
public class GenerateGuid : StatelessLink<Signal, Guid>
{
protected override ValueTask<Guid> ExecuteAsync(Signal input, ChainContext context)
{
return new(Guid.NewGuid());
}
}public class WriteToConsole : StatelessLink<Guid, Signal>
{
protected override ValueTask<Signal> ExecuteAsync(Guid input, ChainContext context)
{
Console.WriteLine(input);
return Signal.Static;
}
}- Define chain
public class ExampleChain : ChainBuilder<Signal>
{
public override string Name { get; } = "ExampleChain";
public override void ConfigureSources(SourceConnectorCollection<Signal> sources)
{
sources.Add<TriggerOnStartupSource>("TriggerOnStartup");
}
public override void ConfigureRootConnector(IConnectorLinker<Signal> root)
{
root.Link<GenerateGuid, Guid>()
.Link<WriteToConsole, Signal>();
}
}- Register in ServiceCollection
services.AddDaisy(Configuration, d =>
{
d.AddChain<ExampleChain>();
});A chain is an integration defined as a data flow. It is built using connectors which defines the control flow and processing steps for handling data. It supports registering multiple sources of execution. A LockStrategy can be specified to cap the number of simultaneous executions.
It's defined by inheriting ChainBuilder<T> where T is the expected input from sources that want to trigger the chain.
A source is a starting point for a chain. It will implement logic for when to trigger a chain and can pass a payload along with each run. One example is the NCronTabSource which starts the chain based on a defined cron schedule. If the payload of the source doesn't match the input of the chain, a mapping function can be provided.
public class ExampleSource : Source<DateTime>
{
public override async Task ExecuteAsync(SourceNextDelegate<DateTime> next, CancellationToken cancellationToken)
{
// Directly executes the chain once on startup with the current date as payload
await next(DateTime.Now);
}
}// Ordinary registration when the source output matches the ChainBuilder
sources.Add<ExampleSource>("ConfigurationName");
// Registration with a mapping function (DateTime -> int) if used in a ChainBuilder<int>
Sources.Add<ExampleSource,int>("ConfigurationName", dt => dt.Year);Connectors are the steps that form a chain. Created using a fluent builder, they describe how data will flow and be processed.
The ChainContext is a control interface that provides access to a per-execution scoped data collection, utilities and manipulation of the execution itself. The data collection allows steps to communicate without having to flow the data as input / output through the chain. The collection items are discarded and disposed after the execution is done.
Adds or updates an item with a specified key and a value.
context.Set("Foo", "Bar");Gets the value associated with the specified key.
if(context.TryGet("Foo", out string fooValue))
{
context.Logger.LogInformation(fooValue);
}Access an ILogger to perform logging.
context.Logger.LogInformation("Example log message");Used to throttle simultaneous execution of a chain, preventing it from becoming overloaded. Without an explicit lock strategy set, there is no defined limit on concurrent executions.
Callers share a lock with a fixed number of allowed concurrent executions.
// This allows 1 concurrent execution of the chain
public override ILockStrategy CreateLockStrategy()
{
return new SharedLockStrategy(concurrency: 1);
}Used to represent a message without any data flowing through a chain.
A link is re-usable asynchronous step, its ExecuteAsync method is called with the input from the previous step. The link then returns the value that should be passed to the next step in the chain.
Links can be either stateful or stateless:
StatefulLinkis instantiated once and the same instance is used for each execution, therefore they can hold state between executions but has to be thread-safe- More performant if no scoped services are used
- Only singleton and transient services can be requested in the constructor
StatelessLinkis instantiated in each execution scope and can't hold state between executions- Can request singleton, transient and scoped services in ctor
public class IntToString : StatefulLink<int, string>
{
private long _totalValueSeen = 0;
protected override ValueTask ExecuteAsync(int input, ChainContext context)
{
var newTotal = Interlocked.Add(ref _totalValueSeen, input);
context.Logger.LogInformation($"I've seen a total value of {newTotal}");
return input.ToString();
}
}root.Link<IntToString, string>();Run a Func<TInput, TOutput> and use the returned value as input for continuation.
root.Map(input =>
{
return input * 2;
})Allows you to branch the chain to a separate data flow. SubChains are executed sequentially, in other words, the parent chain will wait for the sub-chain to be completed before continuing.
root.SubChain(subChain => subChain
.Map(DivideByTwo)
.Link<IntToString, string>()
.Link<ConsoleWriteLine, string>();
)
.Map(MultiplyByTwo)
.Link<IntToString, string>()
.Link<ConsoleWriteLine, string>();SubChains also provides a way to move isolated parts of a chain into separate static classes or methods.
root.SubChain(DivisionSubChain.Build)
.SubChain(FactorSubChain.Build)
// DivisionSubChain.cs
public static class DivisionSubChain
{
public static void Build(IChainBuilder<int> builder)
{
builder
.Map(DivideByTwo)
.Action(WriteToConsole);
}
}A control flow which executes the sub-chain if the predicate evaluates to true.
root.If(IsDivisibleBy3, then => then
.Map(_ => "Value was divisible by 3")
.Link<ConsoleWriteLine, string>());Like SubChain but takes a collection as input and executes the sub-chain for each value.
root.Map(FilterNegativeNumbers)
.Each(each => each
.Map(MultiplyByTwo)
.Link<ConsoleWriteLine, string>());