private Task <IReadOnlyCollection <KeyValuePair <CellFeed, IAggregateProgress> > > MapFeedBatchesAsync(List <List <SheetCellInfo> > cellGroups, string cellsFeedUrl, string batchUrl, IAggregateProgress progress, CancellationToken cancellationToken)
{
return(Task.Run <IReadOnlyCollection <KeyValuePair <CellFeed, IAggregateProgress> > >(() =>
{
var progresses = progress.CreateParallelProgresses(cellGroups.Select(g => (double)g.Count).ToList());
var batches = cellGroups.Zip(progresses, (cells, p) =>
{
var mappingFeed = new CellFeed(new Uri(cellsFeedUrl), Service)
{
Batch = batchUrl
};
foreach (var cellEntry in cells)
{
mappingFeed.Entries.Add(cellEntry.BatchCreateEntry(cellsFeedUrl));
}
cancellationToken.ThrowIfCancellationRequested();
return new KeyValuePair <CellFeed, IAggregateProgress>(mappingFeed, p);
});
return batches.ToList();
}, cancellationToken));
}
private async Task <ResGroupModel <TModel> > GetGroupAsync <TModel>(WorksheetEntry worksheet, SemaphoreSlim semaphore, IAggregateProgress progress, CancellationToken cancellationToken)
where TModel : IRowModel, new()
{
uint rowsPerBatch = MaxReadBatchSize / worksheet.Cols;
IEnumerable <uint> batchStartIndexes = Enumerable.Range(0, (int)(worksheet.Rows / rowsPerBatch + 1)).Select(batchIndex => (uint)(batchIndex * rowsPerBatch + 1));
var cellsQueries = batchStartIndexes.Select(rowIndex =>
{
var cellsQuery = worksheet.GetCellsQuery();
cellsQuery.MinimumRow = rowIndex;
cellsQuery.MaximumRow = rowIndex + rowsPerBatch;
if (cellsQuery.MaximumRow > worksheet.Rows)
{
cellsQuery.MaximumRow = worksheet.Rows;
}
return(cellsQuery);
}).ToList();
var progresses = progress.CreateParallelProgresses(cellsQueries.Select(cq => (double)cq.MaximumRow - cq.MinimumRow).ToList());
var cellsTasks = cellsQueries.Zip(progresses, async(cellsQuery, p) =>
{
await semaphore.WaitAsync(cancellationToken);
try
{
var cellsFeed = await Service.GetFeedAsync(cellsQuery, p, cancellationToken);
return(cellsFeed.Entries.Cast <CellEntry>());
}
finally
{
semaphore.Release();
}
});
var cells = (await Task.WhenAll(cellsTasks)).SelectMany(cs => cs).ToList();
return(await ProcessGroupAsync <TModel>(worksheet, cells, cancellationToken));
}
private Task <IReadOnlyCollection <KeyValuePair <CellFeed, IAggregateProgress> > > UpdateFeedBatchesAsync(
List <List <SheetCellInfo> > cellGroups,
IEnumerable <CellFeed> mappingFeeds,
string cellsFeedUrl,
string batchUrl,
IAggregateProgress progress,
CancellationToken cancellationToken)
{
return(Task.Run <IReadOnlyCollection <KeyValuePair <CellFeed, IAggregateProgress> > >(
() =>
{
var cellsJoin = mappingFeeds.Zip(cellGroups, (mappingFeed, cellsGroup) =>
from feedCell in mappingFeed.Entries.Cast <CellEntry>()
from cell in cellsGroup
where feedCell.Row == cell.Row && feedCell.Column == cell.Column
select new { Cell = cell, FeedCell = feedCell });
var updatingProgresses = progress.CreateParallelProgresses(cellGroups.Select(g => (double)g.Count).ToList());
var batches = cellsJoin.Zip(updatingProgresses, (cells, p) =>
{
var updateFeed = new CellFeed(new Uri(cellsFeedUrl), Service)
{
Batch = batchUrl
};
foreach (var pair in cells)
{
updateFeed.Entries.Add(pair.Cell.BatchUpdateEntry(pair.FeedCell));
}
cancellationToken.ThrowIfCancellationRequested();
return new KeyValuePair <CellFeed, IAggregateProgress>(updateFeed, p);
});
return batches.ToList();
}, cancellationToken));
}
private async Task PushCellsAsync <TModel>(WorksheetEntry worksheet, ResGroupModel <TModel> group, SemaphoreSlim semaphore, IAggregateProgress progress, CancellationToken cancellationToken)
where TModel : IRowModel
{
var progresses = progress.CreateParallelProgresses(0.2, 0.8);
var mappingProgress = progresses[0];
var updatingProgress = progresses[1];
var cellFeed = await Service.GetFeedAsync(worksheet.GetCellsQuery(), cancellationToken : cancellationToken);
string cellsFeedUrl = cellFeed.Self;
string batchUrl = cellFeed.Batch;
var resCells = await CreateSheetCellInfos(group, cancellationToken);
var cellGroups = Devide(resCells, MaxBatchSize);
var mappingTasks = (await MapFeedBatchesAsync(cellGroups, cellsFeedUrl, batchUrl, mappingProgress, cancellationToken))
.Select(async kvp =>
{
var mappingFeed = kvp.Key;
var p = kvp.Value;
await semaphore.WaitAsync(cancellationToken);
try
{
return(await Service.BatchFeedAsync(mappingFeed, TimeOutAttemptsCount, p, cancellationToken));
}
finally
{
semaphore.Release();
}
});
//Mapping cells.
List <CellFeed> mappingFeeds = new List <CellFeed>(cellGroups.Count);
foreach (var mappingTask in mappingTasks)
{
mappingFeeds.Add(await mappingTask);
}
//Updating cells.
var updatingTasks = (await UpdateFeedBatchesAsync(cellGroups, mappingFeeds, cellsFeedUrl, batchUrl, updatingProgress, cancellationToken))
.Select(async kvp =>
{
var updateFeed = kvp.Key;
var p = kvp.Value;
await semaphore.WaitAsync(cancellationToken);
try
{
return(await Service.BatchFeedAsync(updateFeed, TimeOutAttemptsCount, p, cancellationToken));
}
finally
{
semaphore.Release();
}
});
foreach (var updatingTask in updatingTasks)
{
await updatingTask;
}
}