public HealthCheckHostedService(IClusterClient client, IMembershipOracle oracle, IOptions <HealthCheckHostedServiceOptions> myOptions) { host = new WebHostBuilder() .UseKestrel(options => options.ListenAnyIP(myOptions.Value.Port)) .ConfigureServices(services => { services.AddHealthChecks() .AddCheck <GrainHealthCheck>("GrainHealth") .AddCheck <SiloHealthCheck>("SiloHealth") .AddCheck <StorageHealthCheck>("StorageHealth") .AddCheck <ClusterHealthCheck>("ClusterHealth"); services.AddSingleton <IHealthCheckPublisher, LoggingHealthCheckPublisher>() .Configure <HealthCheckPublisherOptions>(options => { options.Period = TimeSpan.FromSeconds(1); }); services.AddSingleton(client); services.AddSingleton(Enumerable.AsEnumerable(new IHealthCheckParticipant[] { oracle })); }) .ConfigureLogging(builder => { builder.AddConsole(); }) .Configure(app => { app.UseHealthChecks(myOptions.Value.PathString); }) .Build(); }
public DummySilo( ILifecycleObservable lifecycle, ILoggerFactory loggerFactory, IMembershipOracle membershipOracle, IReminderService reminderService, IMessageCenter messageCenter, IRuntime runtime) { logger = loggerFactory.CreateLogger <DummySilo>(); lifecycle.Subscribe(Initialize, Start, Stop); }
public TestHooksSystemTarget( ISiloHost host, ILocalSiloDetails siloDetails, ILoggerFactory loggerFactory, IMembershipOracle clusteringOracle, TestHooksHostEnvironmentStatistics hostEnvironmentStatistics, IOptions <LoadSheddingOptions> loadSheddingOptions) : base(Constants.TestHooksSystemTargetId, siloDetails.SiloAddress, loggerFactory) { this.host = host; this.clusteringOracle = clusteringOracle; this.hostEnvironmentStatistics = hostEnvironmentStatistics; this.loadSheddingOptions = loadSheddingOptions.Value; this.consistentRingProvider = this.host.Services.GetRequiredService <IConsistentRingProvider>(); }
public ApiHostedService( IOptions <ApiHostedServiceOptions> options, IClusterClient client, IMembershipOracle oracle, IConfiguration configuration, IExportLocatorScope exportScope, IAppInfo appInfo, ILogger <ApiHostedService> logger ) { _appInfo = appInfo; _logger = logger; logger.LogInformation("Initializing api {appName} ({version}) [{env}] on port {apiPort}...", appInfo.Name, appInfo.Version, appInfo.Environment, options.Value.Port); _host = WebHost.CreateDefaultBuilder() .UseGrace(new InjectionScopeConfiguration { Behaviors = { AllowInstanceAndFactoryToReturnNull = true } }) .UseSerilog() .UseConfiguration(configuration) .ConfigureAppConfiguration(cfg => { cfg.Sources.Clear(); cfg.AddConfiguration(configuration); }) .ConfigureServices(services => { services.AddSingleton(appInfo); services.AddSingleton(client); services.AddSingleton(exportScope); }) .UseStartup <ApiStartup>() .UseUrls($"http://*:{options.Value.Port}") .Build(); }
internal Silo(SiloInitializationParameters initializationParams, IServiceProvider services) { string name = initializationParams.Name; ClusterConfiguration config = initializationParams.ClusterConfig; this.initializationParams = initializationParams; this.SystemStatus = SystemStatus.Creating; AsynchAgent.IsStarting = true; var startTime = DateTime.UtcNow; services?.GetService <TelemetryManager>()?.AddFromConfiguration(services, LocalConfig.TelemetryConfiguration); StatisticsCollector.Initialize(LocalConfig); initTimeout = GlobalConfig.MaxJoinAttemptTime; if (Debugger.IsAttached) { initTimeout = StandardExtensions.Max(TimeSpan.FromMinutes(10), GlobalConfig.MaxJoinAttemptTime); stopTimeout = initTimeout; } var localEndpoint = this.initializationParams.SiloAddress.Endpoint; // Configure DI using Startup type if (services == null) { var serviceCollection = new ServiceCollection(); serviceCollection.AddSingleton <Silo>(this); serviceCollection.AddSingleton(initializationParams); serviceCollection.AddLegacyClusterConfigurationSupport(config); serviceCollection.Configure <SiloIdentityOptions>(options => options.SiloName = name); var hostContext = new HostBuilderContext(new Dictionary <object, object>()); DefaultSiloServices.AddDefaultServices(hostContext, serviceCollection); var applicationPartManager = hostContext.GetApplicationPartManager(); applicationPartManager.AddApplicationPartsFromAppDomain(); applicationPartManager.AddApplicationPartsFromBasePath(); services = StartupBuilder.ConfigureStartup(this.LocalConfig.StartupTypeName, serviceCollection); services.GetService <TelemetryManager>()?.AddFromConfiguration(services, LocalConfig.TelemetryConfiguration); } services.GetService <SerializationManager>().RegisterSerializers(services.GetService <ApplicationPartManager>()); this.Services = services; this.Services.InitializeSiloUnobservedExceptionsHandler(); //set PropagateActivityId flag from node cofnig RequestContext.PropagateActivityId = this.initializationParams.NodeConfig.PropagateActivityId; this.loggerFactory = this.Services.GetRequiredService <ILoggerFactory>(); logger = this.loggerFactory.CreateLogger <Silo>(); logger.Info(ErrorCode.SiloGcSetting, "Silo starting with GC settings: ServerGC={0} GCLatencyMode={1}", GCSettings.IsServerGC, Enum.GetName(typeof(GCLatencyMode), GCSettings.LatencyMode)); if (!GCSettings.IsServerGC) { logger.Warn(ErrorCode.SiloGcWarning, "Note: Silo not running with ServerGC turned on - recommend checking app config : <configuration>-<runtime>-<gcServer enabled=\"true\">"); logger.Warn(ErrorCode.SiloGcWarning, "Note: ServerGC only kicks in on multi-core systems (settings enabling ServerGC have no effect on single-core machines)."); } logger.Info(ErrorCode.SiloInitializing, "-------------- Initializing {0} silo on host {1} MachineName {2} at {3}, gen {4} --------------", this.initializationParams.Type, LocalConfig.DNSHostName, Environment.MachineName, localEndpoint, this.initializationParams.SiloAddress.Generation); logger.Info(ErrorCode.SiloInitConfig, "Starting silo {0} with the following configuration= " + Environment.NewLine + "{1}", name, config.ToString(name)); var siloMessagingOptions = this.Services.GetRequiredService <IOptions <SiloMessagingOptions> >(); BufferPool.InitGlobalBufferPool(siloMessagingOptions); try { grainFactory = Services.GetRequiredService <GrainFactory>(); } catch (InvalidOperationException exc) { logger.Error(ErrorCode.SiloStartError, "Exception during Silo.Start, GrainFactory was not registered in Dependency Injection container", exc); throw; } // Performance metrics siloStatistics = Services.GetRequiredService <SiloStatisticsManager>(); // The scheduler scheduler = Services.GetRequiredService <OrleansTaskScheduler>(); healthCheckParticipants.Add(scheduler); runtimeClient = Services.GetRequiredService <InsideRuntimeClient>(); // Initialize the message center messageCenter = Services.GetRequiredService <MessageCenter>(); var dispatcher = this.Services.GetRequiredService <Dispatcher>(); messageCenter.RerouteHandler = dispatcher.RerouteMessage; messageCenter.SniffIncomingMessage = runtimeClient.SniffIncomingMessage; // GrainRuntime can be created only here, after messageCenter was created. grainRuntime = Services.GetRequiredService <IGrainRuntime>(); StreamProviderManager = Services.GetRequiredService <IStreamProviderManager>(); // Now the router/directory service // This has to come after the message center //; note that it then gets injected back into the message center.; localGrainDirectory = Services.GetRequiredService <LocalGrainDirectory>(); // Now the activation directory. activationDirectory = Services.GetRequiredService <ActivationDirectory>(); // Now the consistent ring provider RingProvider = Services.GetRequiredService <IConsistentRingProvider>(); catalog = Services.GetRequiredService <Catalog>(); siloStatistics.MetricsTable.Scheduler = scheduler; siloStatistics.MetricsTable.ActivationDirectory = activationDirectory; siloStatistics.MetricsTable.ActivationCollector = catalog.ActivationCollector; siloStatistics.MetricsTable.MessageCenter = messageCenter; executorService = Services.GetRequiredService <ExecutorService>(); // Now the incoming message agents var messageFactory = this.Services.GetRequiredService <MessageFactory>(); incomingSystemAgent = new IncomingMessageAgent(Message.Categories.System, messageCenter, activationDirectory, scheduler, catalog.Dispatcher, messageFactory, executorService, this.loggerFactory); incomingPingAgent = new IncomingMessageAgent(Message.Categories.Ping, messageCenter, activationDirectory, scheduler, catalog.Dispatcher, messageFactory, executorService, this.loggerFactory); incomingAgent = new IncomingMessageAgent(Message.Categories.Application, messageCenter, activationDirectory, scheduler, catalog.Dispatcher, messageFactory, executorService, this.loggerFactory); membershipOracle = Services.GetRequiredService <IMembershipOracle>(); if (!this.GlobalConfig.HasMultiClusterNetwork) { logger.Info("Skip multicluster oracle creation (no multicluster network configured)"); } else { multiClusterOracle = Services.GetRequiredService <IMultiClusterOracle>(); } this.SystemStatus = SystemStatus.Created; AsynchAgent.IsStarting = false; StringValueStatistic.FindOrCreate(StatisticNames.SILO_START_TIME, () => LogFormatter.PrintDate(startTime)); // this will help troubleshoot production deployment when looking at MDS logs. var fullSiloLifecycle = this.Services.GetRequiredService <SiloLifecycle>(); this.siloLifecycle = fullSiloLifecycle; IEnumerable <ILifecycleParticipant <ISiloLifecycle> > lifecyleParticipants = this.Services.GetServices <ILifecycleParticipant <ISiloLifecycle> >(); foreach (ILifecycleParticipant <ISiloLifecycle> participant in lifecyleParticipants) { participant.Participate(fullSiloLifecycle); } this.Participate(fullSiloLifecycle); logger.Info(ErrorCode.SiloInitializingFinished, "-------------- Started silo {0}, ConsistentHashCode {1:X} --------------", SiloAddress.ToLongString(), SiloAddress.GetConsistentHashCode()); }
private void DoStart() { lock (lockable) { if (SystemStatus.Current != SystemStatus.Created) { throw new InvalidOperationException(String.Format("Calling Silo.Start() on a silo which is not in the Start state. This silo is in the {0} state.", SystemStatus.Current)); } SystemStatus.Current = SystemStatus.Starting; } logger.Info(ErrorCode.SiloStarting, "Silo Start()"); // Hook up to receive notification of process exit / Ctrl-C events AppDomain.CurrentDomain.ProcessExit += HandleProcessExit; Console.CancelKeyPress += HandleProcessExit; ConfigureThreadPoolAndServicePointSettings(); // This has to start first so that the directory system target factory gets loaded before we start the router. typeManager.Start(); InsideRuntimeClient.Current.Start(); // The order of these 4 is pretty much arbitrary. scheduler.Start(); messageCenter.Start(); incomingPingAgent.Start(); incomingSystemAgent.Start(); incomingAgent.Start(); LocalGrainDirectory.Start(); // Set up an execution context for this thread so that the target creation steps can use asynch values. RuntimeContext.InitializeMainThread(); SiloProviderRuntime.Initialize(GlobalConfig); statisticsProviderManager = new StatisticsProviderManager("Statistics", SiloProviderRuntime.Instance); string statsProviderName = statisticsProviderManager.LoadProvider(GlobalConfig.ProviderConfigurations) .WaitForResultWithThrow(initTimeout); if (statsProviderName != null) { LocalConfig.StatisticsProviderName = statsProviderName; } // can call SetSiloMetricsTableDataManager only after MessageCenter is created (dependency on this.SiloAddress). siloStatistics.SetSiloStatsTableDataManager(this, nodeConfig).WaitWithThrow(initTimeout); siloStatistics.SetSiloMetricsTableDataManager(this, nodeConfig).WaitWithThrow(initTimeout); membershipOracle = membershipFactory.CreateMembershipOracle(this).WaitForResultWithThrow(initTimeout); // This has to follow the above steps that start the runtime components CreateSystemTargets(); InjectDependencies(); // ensure this runs in the grain context, wait for it to complete scheduler.QueueTask(CreateSystemGrains, catalog.SchedulingContext) .WaitWithThrow(initTimeout); // Initialize storage providers once we have a basic silo runtime environment operating storageProviderManager = new StorageProviderManager(); scheduler.QueueTask( () => storageProviderManager.LoadStorageProviders(GlobalConfig.ProviderConfigurations), providerManagerSystemTarget.SchedulingContext) .WaitWithThrow(initTimeout); catalog.SetStorageManager(storageProviderManager); // Load and init stream providers before silo becomes active var siloStreamProviderManager = new Orleans.Streams.StreamProviderManager(); scheduler.QueueTask( () => siloStreamProviderManager.LoadStreamProviders(this.GlobalConfig.ProviderConfigurations, SiloProviderRuntime.Instance), providerManagerSystemTarget.SchedulingContext) .WaitWithThrow(initTimeout); InsideRuntimeClient.Current.CurrentStreamProviderManager = siloStreamProviderManager; ISchedulingContext statusOracleContext = ((SystemTarget)LocalSiloStatusOracle).SchedulingContext; bool waitForPrimaryToStart = globalConfig.PrimaryNodeIsRequired && siloType != SiloType.Primary; scheduler.QueueTask(() => LocalSiloStatusOracle.Start(waitForPrimaryToStart), statusOracleContext) .WaitWithThrow(initTimeout); scheduler.QueueTask(LocalSiloStatusOracle.BecomeActive, statusOracleContext) .WaitWithThrow(initTimeout); try { siloStatistics.Start(LocalConfig); // Finally, initialize the deployment load collector, for grains with load-based placement scheduler.QueueTask(DeploymentLoadPublisher.Instance.Start, DeploymentLoadPublisher.Instance.SchedulingContext) .WaitWithThrow(initTimeout); // Start background timer tick to watch for platform execution stalls, such as when GC kicks in platformWatchdog = new Watchdog(nodeConfig.StatisticsLogWriteInterval, healthCheckParticipants); platformWatchdog.Start(); // so, we have the view of the membership in the consistentRingProvider. We can start the reminder service scheduler.QueueTask(reminderService.Start, ((SystemTarget)reminderService).SchedulingContext) .WaitWithThrow(initTimeout); // Start stream providers after silo is active (so the pulling agents don't start sending messages before silo is active). scheduler.QueueTask(siloStreamProviderManager.StartStreamProviders, providerManagerSystemTarget.SchedulingContext) .WaitWithThrow(initTimeout); var bootstrapProviderManager = new BootstrapProviderManager(); scheduler.QueueTask( () => bootstrapProviderManager.LoadAppBootstrapProviders(GlobalConfig.ProviderConfigurations), providerManagerSystemTarget.SchedulingContext) .WaitWithThrow(initTimeout); BootstrapProviders = bootstrapProviderManager.GetProviders(); // Data hook for testing & diagnotics // Now that we're active, we can start the gateway var mc = messageCenter as MessageCenter; if (mc != null) { mc.StartGateway(); } SystemStatus.Current = SystemStatus.Running; } catch (Exception exc) { SafeExecute(() => logger.Error(ErrorCode.Runtime_Error_100330, String.Format("Error starting silo {0}. Going to FastKill().", SiloAddress), exc)); FastKill(); // if failed after Membership became active, mark itself as dead in Membership abale. throw; } }
public Silo(ILocalSiloDetails siloDetails, IServiceProvider services) { string name = siloDetails.Name; // Temporarily still require this. Hopefuly gone when 2.0 is released. this.siloDetails = siloDetails; this.SystemStatus = SystemStatus.Creating; AsynchAgent.IsStarting = true; // todo. use ISiloLifecycle instead? var startTime = DateTime.UtcNow; IOptions <ClusterMembershipOptions> clusterMembershipOptions = services.GetRequiredService <IOptions <ClusterMembershipOptions> >(); initTimeout = clusterMembershipOptions.Value.MaxJoinAttemptTime; if (Debugger.IsAttached) { initTimeout = StandardExtensions.Max(TimeSpan.FromMinutes(10), clusterMembershipOptions.Value.MaxJoinAttemptTime); stopTimeout = initTimeout; } var localEndpoint = this.siloDetails.SiloAddress.Endpoint; services.GetService <SerializationManager>().RegisterSerializers(services.GetService <IApplicationPartManager>()); this.Services = services; this.Services.InitializeSiloUnobservedExceptionsHandler(); //set PropagateActivityId flag from node config IOptions <SiloMessagingOptions> messagingOptions = services.GetRequiredService <IOptions <SiloMessagingOptions> >(); RequestContext.PropagateActivityId = messagingOptions.Value.PropagateActivityId; this.loggerFactory = this.Services.GetRequiredService <ILoggerFactory>(); logger = this.loggerFactory.CreateLogger <Silo>(); logger.Info(ErrorCode.SiloGcSetting, "Silo starting with GC settings: ServerGC={0} GCLatencyMode={1}", GCSettings.IsServerGC, Enum.GetName(typeof(GCLatencyMode), GCSettings.LatencyMode)); if (!GCSettings.IsServerGC) { logger.Warn(ErrorCode.SiloGcWarning, "Note: Silo not running with ServerGC turned on - recommend checking app config : <configuration>-<runtime>-<gcServer enabled=\"true\">"); logger.Warn(ErrorCode.SiloGcWarning, "Note: ServerGC only kicks in on multi-core systems (settings enabling ServerGC have no effect on single-core machines)."); } logger.Info(ErrorCode.SiloInitializing, "-------------- Initializing silo on host {0} MachineName {1} at {2}, gen {3} --------------", this.siloDetails.DnsHostName, Environment.MachineName, localEndpoint, this.siloDetails.SiloAddress.Generation); logger.Info(ErrorCode.SiloInitConfig, "Starting silo {0}", name); var siloMessagingOptions = this.Services.GetRequiredService <IOptions <SiloMessagingOptions> >(); BufferPool.InitGlobalBufferPool(siloMessagingOptions.Value); try { grainFactory = Services.GetRequiredService <GrainFactory>(); } catch (InvalidOperationException exc) { logger.Error(ErrorCode.SiloStartError, "Exception during Silo.Start, GrainFactory was not registered in Dependency Injection container", exc); throw; } // Performance metrics siloStatistics = Services.GetRequiredService <SiloStatisticsManager>(); // The scheduler scheduler = Services.GetRequiredService <OrleansTaskScheduler>(); healthCheckParticipants.Add(scheduler); runtimeClient = Services.GetRequiredService <InsideRuntimeClient>(); // Initialize the message center messageCenter = Services.GetRequiredService <MessageCenter>(); var dispatcher = this.Services.GetRequiredService <Dispatcher>(); messageCenter.RerouteHandler = dispatcher.RerouteMessage; messageCenter.SniffIncomingMessage = runtimeClient.SniffIncomingMessage; // Now the router/directory service // This has to come after the message center //; note that it then gets injected back into the message center.; localGrainDirectory = Services.GetRequiredService <LocalGrainDirectory>(); // Now the activation directory. activationDirectory = Services.GetRequiredService <ActivationDirectory>(); // Now the consistent ring provider RingProvider = Services.GetRequiredService <IConsistentRingProvider>(); catalog = Services.GetRequiredService <Catalog>(); executorService = Services.GetRequiredService <ExecutorService>(); // Now the incoming message agents var messageFactory = this.Services.GetRequiredService <MessageFactory>(); incomingSystemAgent = new IncomingMessageAgent(Message.Categories.System, messageCenter, activationDirectory, scheduler, catalog.Dispatcher, messageFactory, executorService, this.loggerFactory); incomingPingAgent = new IncomingMessageAgent(Message.Categories.Ping, messageCenter, activationDirectory, scheduler, catalog.Dispatcher, messageFactory, executorService, this.loggerFactory); incomingAgent = new IncomingMessageAgent(Message.Categories.Application, messageCenter, activationDirectory, scheduler, catalog.Dispatcher, messageFactory, executorService, this.loggerFactory); membershipOracle = Services.GetRequiredService <IMembershipOracle>(); this.clusterOptions = Services.GetRequiredService <IOptions <ClusterOptions> >().Value; var multiClusterOptions = Services.GetRequiredService <IOptions <MultiClusterOptions> >().Value; if (!multiClusterOptions.HasMultiClusterNetwork) { logger.Info("Skip multicluster oracle creation (no multicluster network configured)"); } else { multiClusterOracle = Services.GetRequiredService <IMultiClusterOracle>(); } this.SystemStatus = SystemStatus.Created; AsynchAgent.IsStarting = false; StringValueStatistic.FindOrCreate(StatisticNames.SILO_START_TIME, () => LogFormatter.PrintDate(startTime)); // this will help troubleshoot production deployment when looking at MDS logs. this.siloLifecycle = this.Services.GetRequiredService <ISiloLifecycleSubject>(); // register all lifecycle participants IEnumerable <ILifecycleParticipant <ISiloLifecycle> > lifecycleParticipants = this.Services.GetServices <ILifecycleParticipant <ISiloLifecycle> >(); foreach (ILifecycleParticipant <ISiloLifecycle> participant in lifecycleParticipants) { participant?.Participate(this.siloLifecycle); } // register all named lifecycle participants IKeyedServiceCollection <string, ILifecycleParticipant <ISiloLifecycle> > namedLifecycleParticipantCollection = this.Services.GetService <IKeyedServiceCollection <string, ILifecycleParticipant <ISiloLifecycle> > >(); foreach (ILifecycleParticipant <ISiloLifecycle> participant in namedLifecycleParticipantCollection ?.GetServices(this.Services) ?.Select(s => s.GetService(this.Services))) { participant?.Participate(this.siloLifecycle); } // add self to lifecycle this.Participate(this.siloLifecycle); logger.Info(ErrorCode.SiloInitializingFinished, "-------------- Started silo {0}, ConsistentHashCode {1:X} --------------", SiloAddress.ToLongString(), SiloAddress.GetConsistentHashCode()); }
internal Silo(SiloInitializationParameters initializationParams) { string name = initializationParams.Name; ClusterConfiguration config = initializationParams.ClusterConfig; this.initializationParams = initializationParams; SystemStatus.Current = SystemStatus.Creating; CurrentSilo = this; var startTime = DateTime.UtcNow; siloTerminatedEvent = new ManualResetEvent(false); if (!LogManager.IsInitialized) LogManager.Initialize(LocalConfig); config.OnConfigChange("Defaults/Tracing", () => LogManager.Initialize(LocalConfig, true), false); MultiClusterRegistrationStrategy.Initialize(config.Globals); StatisticsCollector.Initialize(LocalConfig); SerializationManager.Initialize(GlobalConfig.SerializationProviders, this.GlobalConfig.FallbackSerializationProvider); initTimeout = GlobalConfig.MaxJoinAttemptTime; if (Debugger.IsAttached) { initTimeout = StandardExtensions.Max(TimeSpan.FromMinutes(10), GlobalConfig.MaxJoinAttemptTime); stopTimeout = initTimeout; } var localEndpoint = this.initializationParams.SiloAddress.Endpoint; LogManager.MyIPEndPoint = localEndpoint; logger = LogManager.GetLogger("Silo", LoggerType.Runtime); logger.Info(ErrorCode.SiloGcSetting, "Silo starting with GC settings: ServerGC={0} GCLatencyMode={1}", GCSettings.IsServerGC, Enum.GetName(typeof(GCLatencyMode), GCSettings.LatencyMode)); if (!GCSettings.IsServerGC || !GCSettings.LatencyMode.Equals(GCLatencyMode.Batch)) { logger.Warn(ErrorCode.SiloGcWarning, "Note: Silo not running with ServerGC turned on or with GCLatencyMode.Batch enabled - recommend checking app config : <configuration>-<runtime>-<gcServer enabled=\"true\"> and <configuration>-<runtime>-<gcConcurrent enabled=\"false\"/>"); logger.Warn(ErrorCode.SiloGcWarning, "Note: ServerGC only kicks in on multi-core systems (settings enabling ServerGC have no effect on single-core machines)."); } logger.Info(ErrorCode.SiloInitializing, "-------------- Initializing {0} silo on host {1} MachineName {2} at {3}, gen {4} --------------", this.initializationParams.Type, LocalConfig.DNSHostName, Environment.MachineName, localEndpoint, this.initializationParams.SiloAddress.Generation); logger.Info(ErrorCode.SiloInitConfig, "Starting silo {0} with the following configuration= " + Environment.NewLine + "{1}", name, config.ToString(name)); // Configure DI using Startup type this.Services = StartupBuilder.ConfigureStartup( this.LocalConfig.StartupTypeName, (services, usingCustomServiceProvider) => { // add system types // Note: you can replace IGrainFactory with your own implementation, but // we don't recommend it, in the aspect of performance and usability services.TryAddSingleton(sp => sp); services.TryAddSingleton(this); services.TryAddSingleton(initializationParams); services.TryAddSingleton<ILocalSiloDetails>(initializationParams); services.TryAddSingleton(initializationParams.ClusterConfig); services.TryAddSingleton(initializationParams.GlobalConfig); services.TryAddTransient(sp => initializationParams.NodeConfig); services.TryAddSingleton<ITimerRegistry, TimerRegistry>(); services.TryAddSingleton<IReminderRegistry, ReminderRegistry>(); services.TryAddSingleton<IStreamProviderManager, StreamProviderManager>(); services.TryAddSingleton<GrainRuntime>(); services.TryAddSingleton<IGrainRuntime, GrainRuntime>(); services.TryAddSingleton<OrleansTaskScheduler>(); services.TryAddSingleton<GrainFactory>(sp => sp.GetService<InsideRuntimeClient>().ConcreteGrainFactory); services.TryAddExisting<IGrainFactory, GrainFactory>(); services.TryAddExisting<IInternalGrainFactory, GrainFactory>(); services.TryAddSingleton<TypeMetadataCache>(); services.TryAddSingleton<AssemblyProcessor>(); services.TryAddSingleton<ActivationDirectory>(); services.TryAddSingleton<LocalGrainDirectory>(); services.TryAddExisting<ILocalGrainDirectory, LocalGrainDirectory>(); services.TryAddSingleton<SiloStatisticsManager>(); services.TryAddSingleton<ISiloPerformanceMetrics>(sp => sp.GetRequiredService<SiloStatisticsManager>().MetricsTable); services.TryAddSingleton<SiloAssemblyLoader>(); services.TryAddSingleton<GrainTypeManager>(); services.TryAddExisting<IMessagingConfiguration, GlobalConfiguration>(); services.TryAddSingleton<MessageCenter>(); services.TryAddExisting<IMessageCenter, MessageCenter>(); services.TryAddExisting<ISiloMessageCenter, MessageCenter>(); services.TryAddSingleton<Catalog>(); services.TryAddSingleton(sp => sp.GetRequiredService<Catalog>().Dispatcher); services.TryAddSingleton<InsideRuntimeClient>(); services.TryAddExisting<IRuntimeClient, InsideRuntimeClient>(); services.TryAddExisting<ISiloRuntimeClient, InsideRuntimeClient>(); services.TryAddSingleton<MembershipFactory>(); services.TryAddSingleton<MultiClusterOracleFactory>(); services.TryAddSingleton<LocalReminderServiceFactory>(); services.TryAddSingleton<DeploymentLoadPublisher>(); services.TryAddSingleton<IMembershipTable>( sp => sp.GetRequiredService<MembershipFactory>().GetMembershipTable(sp.GetRequiredService<GlobalConfiguration>())); services.TryAddSingleton<MembershipOracle>( sp => sp.GetRequiredService<MembershipFactory>() .CreateMembershipOracle(sp.GetRequiredService<Silo>(), sp.GetRequiredService<IMembershipTable>())); services.TryAddExisting<IMembershipOracle, MembershipOracle>(); services.TryAddExisting<ISiloStatusOracle, MembershipOracle>(); services.TryAddSingleton<Func<ISiloStatusOracle>>(sp => () => sp.GetRequiredService<ISiloStatusOracle>()); services.TryAddSingleton<MultiClusterOracleFactory>(); services.TryAddSingleton<LocalReminderServiceFactory>(); services.TryAddSingleton<ClientObserverRegistrar>(); services.TryAddSingleton<SiloProviderRuntime>(); services.TryAddExisting<IStreamProviderRuntime, SiloProviderRuntime>(); services.TryAddSingleton<ImplicitStreamSubscriberTable>(); // Placement services.TryAddSingleton<PlacementDirectorsManager>(); services.TryAddSingleton<IPlacementDirector<RandomPlacement>, RandomPlacementDirector>(); services.TryAddSingleton<IPlacementDirector<PreferLocalPlacement>, PreferLocalPlacementDirector>(); services.TryAddSingleton<IPlacementDirector<StatelessWorkerPlacement>, StatelessWorkerDirector>(); services.TryAddSingleton<IPlacementDirector<ActivationCountBasedPlacement>, ActivationCountPlacementDirector>(); services.TryAddSingleton<DefaultPlacementStrategy>(); services.TryAddSingleton<ClientObserversPlacementDirector>(); services.TryAddSingleton<Func<IGrainRuntime>>(sp => () => sp.GetRequiredService<IGrainRuntime>()); services.TryAddSingleton<GrainCreator>(); if (initializationParams.GlobalConfig.UseVirtualBucketsConsistentRing) { services.TryAddSingleton<IConsistentRingProvider>( sp => new VirtualBucketsRingProvider( this.initializationParams.SiloAddress, this.initializationParams.GlobalConfig.NumVirtualBucketsConsistentRing)); } else { services.TryAddSingleton<IConsistentRingProvider>( sp => new ConsistentRingProvider(this.initializationParams.SiloAddress)); } }); this.assemblyProcessor = this.Services.GetRequiredService<AssemblyProcessor>(); this.assemblyProcessor.Initialize(); BufferPool.InitGlobalBufferPool(GlobalConfig); UnobservedExceptionsHandlerClass.SetUnobservedExceptionHandler(UnobservedExceptionHandler); AppDomain.CurrentDomain.UnhandledException += this.DomainUnobservedExceptionHandler; try { grainFactory = Services.GetRequiredService<GrainFactory>(); } catch (InvalidOperationException exc) { logger.Error(ErrorCode.SiloStartError, "Exception during Silo.Start, GrainFactory was not registered in Dependency Injection container", exc); throw; } grainTypeManager = Services.GetRequiredService<GrainTypeManager>(); // Performance metrics siloStatistics = Services.GetRequiredService<SiloStatisticsManager>(); // The scheduler scheduler = Services.GetRequiredService<OrleansTaskScheduler>(); healthCheckParticipants.Add(scheduler); runtimeClient = Services.GetRequiredService<InsideRuntimeClient>(); // Initialize the message center messageCenter = Services.GetRequiredService<MessageCenter>(); messageCenter.RerouteHandler = runtimeClient.RerouteMessage; messageCenter.SniffIncomingMessage = runtimeClient.SniffIncomingMessage; // GrainRuntime can be created only here, after messageCenter was created. grainRuntime = Services.GetRequiredService<IGrainRuntime>(); // Now the router/directory service // This has to come after the message center //; note that it then gets injected back into the message center.; localGrainDirectory = Services.GetRequiredService<LocalGrainDirectory>(); // Now the activation directory. activationDirectory = Services.GetRequiredService<ActivationDirectory>(); // Now the consistent ring provider RingProvider = Services.GetRequiredService<IConsistentRingProvider>(); // to preserve backwards compatibility, only use the service provider to inject grain dependencies if the user supplied his own // service provider, meaning that he is explicitly opting into it. catalog = Services.GetRequiredService<Catalog>(); siloStatistics.MetricsTable.Scheduler = scheduler; siloStatistics.MetricsTable.ActivationDirectory = activationDirectory; siloStatistics.MetricsTable.ActivationCollector = catalog.ActivationCollector; siloStatistics.MetricsTable.MessageCenter = messageCenter; // Now the incoming message agents incomingSystemAgent = new IncomingMessageAgent(Message.Categories.System, messageCenter, activationDirectory, scheduler, catalog.Dispatcher); incomingPingAgent = new IncomingMessageAgent(Message.Categories.Ping, messageCenter, activationDirectory, scheduler, catalog.Dispatcher); incomingAgent = new IncomingMessageAgent(Message.Categories.Application, messageCenter, activationDirectory, scheduler, catalog.Dispatcher); membershipFactory = Services.GetRequiredService<MembershipFactory>(); membershipOracle = Services.GetRequiredService<IMembershipOracle>(); SystemStatus.Current = SystemStatus.Created; StringValueStatistic.FindOrCreate(StatisticNames.SILO_START_TIME, () => LogFormatter.PrintDate(startTime)); // this will help troubleshoot production deployment when looking at MDS logs. logger.Info(ErrorCode.SiloInitializingFinished, "-------------- Started silo {0}, ConsistentHashCode {1:X} --------------", SiloAddress.ToLongString(), SiloAddress.GetConsistentHashCode()); }
internal Silo(SiloInitializationParameters initializationParams, IServiceProvider services) { string name = initializationParams.Name; ClusterConfiguration config = initializationParams.ClusterConfig; this.initializationParams = initializationParams; this.SystemStatus = SystemStatus.Creating; AsynchAgent.IsStarting = true; var startTime = DateTime.UtcNow; if (!LogManager.IsInitialized) { LogManager.Initialize(LocalConfig); } services?.GetService <TelemetryManager>()?.AddFromConfiguration(services, LocalConfig.TelemetryConfiguration); config.OnConfigChange("Defaults/Tracing", () => LogManager.Initialize(LocalConfig, true), false); StatisticsCollector.Initialize(LocalConfig); initTimeout = GlobalConfig.MaxJoinAttemptTime; if (Debugger.IsAttached) { initTimeout = StandardExtensions.Max(TimeSpan.FromMinutes(10), GlobalConfig.MaxJoinAttemptTime); stopTimeout = initTimeout; } var localEndpoint = this.initializationParams.SiloAddress.Endpoint; LogManager.MyIPEndPoint = localEndpoint; logger = LogManager.GetLogger("Silo", LoggerType.Runtime); logger.Info(ErrorCode.SiloGcSetting, "Silo starting with GC settings: ServerGC={0} GCLatencyMode={1}", GCSettings.IsServerGC, Enum.GetName(typeof(GCLatencyMode), GCSettings.LatencyMode)); if (!GCSettings.IsServerGC) { logger.Warn(ErrorCode.SiloGcWarning, "Note: Silo not running with ServerGC turned on - recommend checking app config : <configuration>-<runtime>-<gcServer enabled=\"true\">"); logger.Warn(ErrorCode.SiloGcWarning, "Note: ServerGC only kicks in on multi-core systems (settings enabling ServerGC have no effect on single-core machines)."); } logger.Info(ErrorCode.SiloInitializing, "-------------- Initializing {0} silo on host {1} MachineName {2} at {3}, gen {4} --------------", this.initializationParams.Type, LocalConfig.DNSHostName, Environment.MachineName, localEndpoint, this.initializationParams.SiloAddress.Generation); logger.Info(ErrorCode.SiloInitConfig, "Starting silo {0} with the following configuration= " + Environment.NewLine + "{1}", name, config.ToString(name)); // Configure DI using Startup type if (services == null) { var serviceCollection = new ServiceCollection(); serviceCollection.AddSingleton <Silo>(this); serviceCollection.AddSingleton(initializationParams); DefaultSiloServices.AddDefaultServices(serviceCollection); services = StartupBuilder.ConfigureStartup(this.LocalConfig.StartupTypeName, serviceCollection); services.GetService <TelemetryManager>()?.AddFromConfiguration(services, LocalConfig.TelemetryConfiguration); } this.Services = services; this.assemblyProcessor = this.Services.GetRequiredService <AssemblyProcessor>(); this.assemblyProcessor.Initialize(); BufferPool.InitGlobalBufferPool(GlobalConfig); if (!UnobservedExceptionsHandlerClass.TrySetUnobservedExceptionHandler(UnobservedExceptionHandler)) { logger.Warn(ErrorCode.Runtime_Error_100153, "Unable to set unobserved exception handler because it was already set."); } AppDomain.CurrentDomain.UnhandledException += this.DomainUnobservedExceptionHandler; try { grainFactory = Services.GetRequiredService <GrainFactory>(); } catch (InvalidOperationException exc) { logger.Error(ErrorCode.SiloStartError, "Exception during Silo.Start, GrainFactory was not registered in Dependency Injection container", exc); throw; } grainTypeManager = Services.GetRequiredService <GrainTypeManager>(); // Performance metrics siloStatistics = Services.GetRequiredService <SiloStatisticsManager>(); // The scheduler scheduler = Services.GetRequiredService <OrleansTaskScheduler>(); healthCheckParticipants.Add(scheduler); runtimeClient = Services.GetRequiredService <InsideRuntimeClient>(); // Initialize the message center messageCenter = Services.GetRequiredService <MessageCenter>(); var dispatcher = this.Services.GetRequiredService <Dispatcher>(); messageCenter.RerouteHandler = dispatcher.RerouteMessage; messageCenter.SniffIncomingMessage = runtimeClient.SniffIncomingMessage; // GrainRuntime can be created only here, after messageCenter was created. grainRuntime = Services.GetRequiredService <IGrainRuntime>(); // Now the router/directory service // This has to come after the message center //; note that it then gets injected back into the message center.; localGrainDirectory = Services.GetRequiredService <LocalGrainDirectory>(); // Now the activation directory. activationDirectory = Services.GetRequiredService <ActivationDirectory>(); // Now the consistent ring provider RingProvider = Services.GetRequiredService <IConsistentRingProvider>(); catalog = Services.GetRequiredService <Catalog>(); siloStatistics.MetricsTable.Scheduler = scheduler; siloStatistics.MetricsTable.ActivationDirectory = activationDirectory; siloStatistics.MetricsTable.ActivationCollector = catalog.ActivationCollector; siloStatistics.MetricsTable.MessageCenter = messageCenter; // Now the incoming message agents var messageFactory = this.Services.GetRequiredService <MessageFactory>(); incomingSystemAgent = new IncomingMessageAgent(Message.Categories.System, messageCenter, activationDirectory, scheduler, catalog.Dispatcher, messageFactory); incomingPingAgent = new IncomingMessageAgent(Message.Categories.Ping, messageCenter, activationDirectory, scheduler, catalog.Dispatcher, messageFactory); incomingAgent = new IncomingMessageAgent(Message.Categories.Application, messageCenter, activationDirectory, scheduler, catalog.Dispatcher, messageFactory); membershipOracle = Services.GetRequiredService <IMembershipOracle>(); if (!this.GlobalConfig.HasMultiClusterNetwork) { logger.Info("Skip multicluster oracle creation (no multicluster network configured)"); } else { multiClusterOracle = Services.GetRequiredService <IMultiClusterOracle>(); } this.SystemStatus = SystemStatus.Created; AsynchAgent.IsStarting = false; StringValueStatistic.FindOrCreate(StatisticNames.SILO_START_TIME, () => LogFormatter.PrintDate(startTime)); // this will help troubleshoot production deployment when looking at MDS logs. logger.Info(ErrorCode.SiloInitializingFinished, "-------------- Started silo {0}, ConsistentHashCode {1:X} --------------", SiloAddress.ToLongString(), SiloAddress.GetConsistentHashCode()); }
private void DoStart() { lock (lockable) { if (!SystemStatus.Current.Equals(SystemStatus.Created)) { throw new InvalidOperationException(String.Format("Calling Silo.Start() on a silo which is not in the Created state. This silo is in the {0} state.", SystemStatus.Current)); } SystemStatus.Current = SystemStatus.Starting; } logger.Info(ErrorCode.SiloStarting, "Silo Start()"); // Hook up to receive notification of process exit / Ctrl-C events AppDomain.CurrentDomain.ProcessExit += HandleProcessExit; Console.CancelKeyPress += HandleProcessExit; ConfigureThreadPoolAndServicePointSettings(); // This has to start first so that the directory system target factory gets loaded before we start the router. typeManager.Start(); InsideRuntimeClient.Current.Start(); // The order of these 4 is pretty much arbitrary. scheduler.Start(); messageCenter.Start(); incomingPingAgent.Start(); incomingSystemAgent.Start(); incomingAgent.Start(); LocalGrainDirectory.Start(); // Set up an execution context for this thread so that the target creation steps can use asynch values. RuntimeContext.InitializeMainThread(); SiloProviderRuntime.Initialize(GlobalConfig, SiloIdentity, grainFactory, Services); InsideRuntimeClient.Current.CurrentStreamProviderRuntime = SiloProviderRuntime.Instance; statisticsProviderManager = new StatisticsProviderManager("Statistics", SiloProviderRuntime.Instance); string statsProviderName = statisticsProviderManager.LoadProvider(GlobalConfig.ProviderConfigurations) .WaitForResultWithThrow(initTimeout); if (statsProviderName != null) { LocalConfig.StatisticsProviderName = statsProviderName; } allSiloProviders.AddRange(statisticsProviderManager.GetProviders()); // can call SetSiloMetricsTableDataManager only after MessageCenter is created (dependency on this.SiloAddress). siloStatistics.SetSiloStatsTableDataManager(this, nodeConfig).WaitWithThrow(initTimeout); siloStatistics.SetSiloMetricsTableDataManager(this, nodeConfig).WaitWithThrow(initTimeout); IMembershipTable membershipTable = membershipFactory.GetMembershipTable(GlobalConfig.LivenessType, GlobalConfig.MembershipTableAssembly); membershipOracle = membershipFactory.CreateMembershipOracle(this, membershipTable); multiClusterOracle = multiClusterFactory.CreateGossipOracle(this).WaitForResultWithThrow(initTimeout); // This has to follow the above steps that start the runtime components CreateSystemTargets(); InjectDependencies(); // Validate the configuration. GlobalConfig.Application.ValidateConfiguration(logger); // ensure this runs in the grain context, wait for it to complete scheduler.QueueTask(CreateSystemGrains, catalog.SchedulingContext) .WaitWithThrow(initTimeout); if (logger.IsVerbose) { logger.Verbose("System grains created successfully."); } // Initialize storage providers once we have a basic silo runtime environment operating storageProviderManager = new StorageProviderManager(grainFactory, Services); scheduler.QueueTask( () => storageProviderManager.LoadStorageProviders(GlobalConfig.ProviderConfigurations), providerManagerSystemTarget.SchedulingContext) .WaitWithThrow(initTimeout); catalog.SetStorageManager(storageProviderManager); allSiloProviders.AddRange(storageProviderManager.GetProviders()); if (logger.IsVerbose) { logger.Verbose("Storage provider manager created successfully."); } // Load and init stream providers before silo becomes active var siloStreamProviderManager = (StreamProviderManager)grainRuntime.StreamProviderManager; scheduler.QueueTask( () => siloStreamProviderManager.LoadStreamProviders(GlobalConfig.ProviderConfigurations, SiloProviderRuntime.Instance), providerManagerSystemTarget.SchedulingContext) .WaitWithThrow(initTimeout); InsideRuntimeClient.Current.CurrentStreamProviderManager = siloStreamProviderManager; allSiloProviders.AddRange(siloStreamProviderManager.GetProviders()); if (logger.IsVerbose) { logger.Verbose("Stream provider manager created successfully."); } ISchedulingContext statusOracleContext = ((SystemTarget)LocalSiloStatusOracle).SchedulingContext; bool waitForPrimaryToStart = globalConfig.PrimaryNodeIsRequired && siloType != SiloType.Primary; if (waitForPrimaryToStart) // only in MembershipTableGrain case. { scheduler.QueueTask(() => membershipFactory.WaitForTableToInit(membershipTable), statusOracleContext) .WaitWithThrow(initTimeout); } scheduler.QueueTask(() => membershipTable.InitializeMembershipTable(GlobalConfig, true, LogManager.GetLogger(membershipTable.GetType().Name)), statusOracleContext) .WaitWithThrow(initTimeout); scheduler.QueueTask(() => LocalSiloStatusOracle.Start(), statusOracleContext) .WaitWithThrow(initTimeout); if (logger.IsVerbose) { logger.Verbose("Local silo status oracle created successfully."); } scheduler.QueueTask(LocalSiloStatusOracle.BecomeActive, statusOracleContext) .WaitWithThrow(initTimeout); if (logger.IsVerbose) { logger.Verbose("Local silo status oracle became active successfully."); } //if running in multi cluster scenario, start the MultiClusterNetwork Oracle if (GlobalConfig.HasMultiClusterNetwork) { logger.Info("Creating multicluster oracle with my ServiceId={0} and ClusterId={1}.", GlobalConfig.ServiceId, GlobalConfig.ClusterId); ISchedulingContext clusterStatusContext = ((SystemTarget)multiClusterOracle).SchedulingContext; scheduler.QueueTask(() => multiClusterOracle.Start(LocalSiloStatusOracle), clusterStatusContext) .WaitWithThrow(initTimeout); if (logger.IsVerbose) { logger.Verbose("multicluster oracle created successfully."); } } try { siloStatistics.Start(LocalConfig); if (logger.IsVerbose) { logger.Verbose("Silo statistics manager started successfully."); } // Finally, initialize the deployment load collector, for grains with load-based placement scheduler.QueueTask(DeploymentLoadPublisher.Instance.Start, DeploymentLoadPublisher.Instance.SchedulingContext) .WaitWithThrow(initTimeout); if (logger.IsVerbose) { logger.Verbose("Silo deployment load publisher started successfully."); } // Start background timer tick to watch for platform execution stalls, such as when GC kicks in platformWatchdog = new Watchdog(nodeConfig.StatisticsLogWriteInterval, healthCheckParticipants); platformWatchdog.Start(); if (logger.IsVerbose) { logger.Verbose("Silo platform watchdog started successfully."); } if (reminderService != null) { // so, we have the view of the membership in the consistentRingProvider. We can start the reminder service scheduler.QueueTask(reminderService.Start, ((SystemTarget)reminderService).SchedulingContext) .WaitWithThrow(initTimeout); if (logger.IsVerbose) { logger.Verbose("Reminder service started successfully."); } } bootstrapProviderManager = new BootstrapProviderManager(); scheduler.QueueTask( () => bootstrapProviderManager.LoadAppBootstrapProviders(GlobalConfig.ProviderConfigurations), providerManagerSystemTarget.SchedulingContext) .WaitWithThrow(initTimeout); BootstrapProviders = bootstrapProviderManager.GetProviders(); // Data hook for testing & diagnotics allSiloProviders.AddRange(BootstrapProviders); if (logger.IsVerbose) { logger.Verbose("App bootstrap calls done successfully."); } // Start stream providers after silo is active (so the pulling agents don't start sending messages before silo is active). // also after bootstrap provider started so bootstrap provider can initialize everything stream before events from this silo arrive. scheduler.QueueTask(siloStreamProviderManager.StartStreamProviders, providerManagerSystemTarget.SchedulingContext) .WaitWithThrow(initTimeout); if (logger.IsVerbose) { logger.Verbose("Stream providers started successfully."); } // Now that we're active, we can start the gateway var mc = messageCenter as MessageCenter; if (mc != null) { mc.StartGateway(clientRegistrar); } if (logger.IsVerbose) { logger.Verbose("Message gateway service started successfully."); } SystemStatus.Current = SystemStatus.Running; } catch (Exception exc) { SafeExecute(() => logger.Error(ErrorCode.Runtime_Error_100330, String.Format("Error starting silo {0}. Going to FastKill().", SiloAddress), exc)); FastKill(); // if failed after Membership became active, mark itself as dead in Membership abale. throw; } if (logger.IsVerbose) { logger.Verbose("Silo.Start complete: System status = {0}", SystemStatus.Current); } }