public void AbstractTransformerStateTests() { var mockAbstractTransformer = new Mock <AbstractTransformer>(); mockAbstractTransformer.Protected().Setup <StringBuilder>("ConvertToWord").Returns(new StringBuilder("1")); AbstractTransformer abstractTransformer = mockAbstractTransformer.Object; Assert.AreEqual("1", abstractTransformer.TransformToWords()); }
public void AbstractTransformerBehaviourTests() { var mockAbstractTransformer = new Mock <AbstractTransformer>(); mockAbstractTransformer.Protected().Setup <StringBuilder>("ConvertToWord").Returns(new StringBuilder("1")); AbstractTransformer abstractTransformer = mockAbstractTransformer.Object; abstractTransformer.TransformToWords(); mockAbstractTransformer.Protected() .Verify <StringBuilder>("ConvertToWord", Times.Exactly(1)); }
private JsonDocument TransformDocumentIfNeeded(JsonDocument document, AbstractTransformer storedTransformer, DocumentRetriever documentRetriever) { if (storedTransformer == null) { return(document); } using (new CurrentTransformationScope(Database, documentRetriever)) { var transformed = storedTransformer .TransformResultsDefinition(new[] { new DynamicJsonObject(document.ToJson()) }) .Select <dynamic, dynamic>(x => JsonExtensions.ToJObject((object)x)) .ToArray(); RavenJObject ravenJObject; switch (transformed.Length) { case 0: throw new InvalidOperationException("The transform results function failed on a document: " + document.Key); case 1: ravenJObject = transformed[0]; break; default: ravenJObject = new RavenJObject { { "$values", new RavenJArray(transformed) } }; break; } return(new JsonDocument { Etag = document.Etag.HashWith(storedTransformer.GetHashCodeBytes()).HashWith(documentRetriever.Etag), NonAuthoritativeInformation = document.NonAuthoritativeInformation, LastModified = document.LastModified, DataAsJson = ravenJObject }); } }
public void GetDocumentsWithIdStartingWith(string idPrefix, string matches, string exclude, int start, int pageSize, CancellationToken token, ref int nextStart, Action <RavenJObject> addDoc, string transformer = null, Dictionary <string, RavenJToken> transformerParameters = null, string skipAfter = null) { if (idPrefix == null) { throw new ArgumentNullException("idPrefix"); } idPrefix = idPrefix.Trim(); var canPerformRapidPagination = nextStart > 0 && start == nextStart; var actualStart = canPerformRapidPagination ? start : 0; var addedDocs = 0; var matchedDocs = 0; TransactionalStorage.Batch( actions => { var docsToSkip = canPerformRapidPagination ? 0 : start; int docCount; AbstractTransformer storedTransformer = null; if (transformer != null) { storedTransformer = IndexDefinitionStorage.GetTransformer(transformer); if (storedTransformer == null) { throw new InvalidOperationException("No transformer with the name: " + transformer); } } do { docCount = 0; var docs = actions.Documents.GetDocumentsWithIdStartingWith(idPrefix, actualStart, pageSize, string.IsNullOrEmpty(skipAfter) ? null : skipAfter); var documentRetriever = new DocumentRetriever(actions, Database.ReadTriggers, Database.InFlightTransactionalState, transformerParameters); foreach (var doc in docs) { token.ThrowIfCancellationRequested(); docCount++; var keyTest = doc.Key.Substring(idPrefix.Length); if (!WildcardMatcher.Matches(matches, keyTest) || WildcardMatcher.MatchesExclusion(exclude, keyTest)) { continue; } DocumentRetriever.EnsureIdInMetadata(doc); var nonAuthoritativeInformationBehavior = Database.InFlightTransactionalState.GetNonAuthoritativeInformationBehavior <JsonDocument>(null, doc.Key); var document = nonAuthoritativeInformationBehavior != null ? nonAuthoritativeInformationBehavior(doc) : doc; document = documentRetriever.ExecuteReadTriggers(document, null, ReadOperation.Load); if (document == null) { continue; } matchedDocs++; if (matchedDocs <= docsToSkip) { continue; } token.ThrowIfCancellationRequested(); if (storedTransformer != null) { using (new CurrentTransformationScope(Database, documentRetriever)) { var transformed = storedTransformer.TransformResultsDefinition(new[] { new DynamicJsonObject(document.ToJson()) }) .Select(x => JsonExtensions.ToJObject(x)) .ToArray(); if (transformed.Length == 0) { throw new InvalidOperationException("The transform results function failed on a document: " + document.Key); } var transformedJsonDocument = new JsonDocument { Etag = document.Etag.HashWith(storedTransformer.GetHashCodeBytes()).HashWith(documentRetriever.Etag), NonAuthoritativeInformation = document.NonAuthoritativeInformation, LastModified = document.LastModified, DataAsJson = new RavenJObject { { "$values", new RavenJArray(transformed) } }, }; addDoc(transformedJsonDocument.ToJson()); } } else { addDoc(document.ToJson()); } addedDocs++; if (addedDocs >= pageSize) { break; } } actualStart += pageSize; }while (docCount > 0 && addedDocs < pageSize && actualStart > 0 && actualStart < int.MaxValue); }); if (addedDocs != pageSize) { nextStart = start; // will mark as last page } else if (canPerformRapidPagination) { nextStart = start + matchedDocs; } else { nextStart = actualStart; } }
public Etag GetDocuments(int start, int pageSize, Etag etag, CancellationToken token, Func <JsonDocument, bool> addDocument, string transformer = null, Dictionary <string, RavenJToken> transformerParameters = null, long?maxSize = null, TimeSpan?timeout = null) { Etag lastDocumentReadEtag = null; using (DocumentCacher.SkipSetDocumentsInDocumentCache()) TransactionalStorage.Batch(actions => { AbstractTransformer storedTransformer = null; if (transformer != null) { storedTransformer = IndexDefinitionStorage.GetTransformer(transformer); if (storedTransformer == null) { throw new InvalidOperationException("No transformer with the name: " + transformer); } } var returnedDocs = false; while (true) { var documents = etag == null ? actions.Documents.GetDocumentsByReverseUpdateOrder(start, pageSize) : actions.Documents.GetDocumentsAfter(etag, pageSize, token, maxSize: maxSize, timeout: timeout); var documentRetriever = new DocumentRetriever(Database.Configuration, actions, Database.ReadTriggers, transformerParameters); var docCount = 0; var docCountOnLastAdd = 0; foreach (var doc in documents) { docCount++; token.ThrowIfCancellationRequested(); if (docCount - docCountOnLastAdd > 1000) { addDocument(null); // heartbeat } if (etag != null) { etag = doc.Etag; } JsonDocument.EnsureIdInMetadata(doc); var nonAuthoritativeInformationBehavior = actions.InFlightStateSnapshot.GetNonAuthoritativeInformationBehavior <JsonDocument>(null, doc.Key); var document = nonAuthoritativeInformationBehavior == null ? doc : nonAuthoritativeInformationBehavior(doc); document = documentRetriever.ExecuteReadTriggers(document, null, ReadOperation.Load); if (document == null) { continue; } returnedDocs = true; Database.WorkContext.UpdateFoundWork(); document = TransformDocumentIfNeeded(document, storedTransformer, documentRetriever); var canContinue = addDocument(document); if (!canContinue) { break; } lastDocumentReadEtag = etag; docCountOnLastAdd = docCount; } if (returnedDocs || docCount == 0) { break; } // No document was found that matches the requested criteria // If we had a failure happen, we update the etag as we don't need to process those documents again (no matches there anyways). if (lastDocumentReadEtag != null) { etag = lastDocumentReadEtag; } start += docCount; } }); return(lastDocumentReadEtag); }
public void GetDocumentsWithIdStartingWith(string idPrefix, string matches, string exclude, int start, int pageSize, CancellationToken token, ref int nextStart, Action <JsonDocument> addDoc, string transformer = null, Dictionary <string, RavenJToken> transformerParameters = null, string skipAfter = null) { if (idPrefix == null) { throw new ArgumentNullException("idPrefix"); } idPrefix = idPrefix.Trim(); var canPerformRapidPagination = nextStart > 0 && start == nextStart; var actualStart = canPerformRapidPagination ? start : 0; var addedDocs = 0; var docCountOnLastAdd = 0; var matchedDocs = 0; TransactionalStorage.Batch( actions => { var docsToSkip = canPerformRapidPagination ? 0 : start; int docCount; AbstractTransformer storedTransformer = null; if (transformer != null) { storedTransformer = IndexDefinitionStorage.GetTransformer(transformer); if (storedTransformer == null) { throw new InvalidOperationException("No transformer with the name: " + transformer); } } do { Database.WorkContext.UpdateFoundWork(); docCount = 0; var docs = actions.Documents.GetDocumentsWithIdStartingWith(idPrefix, actualStart, pageSize, string.IsNullOrEmpty(skipAfter) ? null : skipAfter); var documentRetriever = new DocumentRetriever(Database.Configuration, actions, Database.ReadTriggers, transformerParameters); foreach (var doc in docs) { token.ThrowIfCancellationRequested(); docCount++; if (docCount - docCountOnLastAdd > 1000) { addDoc(null); // heartbeat } var keyTest = doc.Key.Substring(idPrefix.Length); if (!WildcardMatcher.Matches(matches, keyTest) || WildcardMatcher.MatchesExclusion(exclude, keyTest)) { continue; } JsonDocument.EnsureIdInMetadata(doc); var nonAuthoritativeInformationBehavior = actions.InFlightStateSnapshot.GetNonAuthoritativeInformationBehavior <JsonDocument>(null, doc.Key); var document = nonAuthoritativeInformationBehavior != null ? nonAuthoritativeInformationBehavior(doc) : doc; document = documentRetriever.ExecuteReadTriggers(document, null, ReadOperation.Load); if (document == null) { continue; } matchedDocs++; if (matchedDocs <= docsToSkip) { continue; } token.ThrowIfCancellationRequested(); document = TransformDocumentIfNeeded(document, storedTransformer, documentRetriever); addDoc(document); addedDocs++; docCountOnLastAdd = docCount; if (addedDocs >= pageSize) { break; } } actualStart += pageSize; } while (docCount > 0 && addedDocs < pageSize && actualStart > 0 && actualStart < int.MaxValue); }); if (addedDocs != pageSize) { nextStart = start; // will mark as last page } else if (canPerformRapidPagination) { nextStart = start + matchedDocs; } else { nextStart = actualStart; } }
public string CreateAndPersistTransform(TransformerDefinition transformerDefinition, AbstractTransformer transformer) { transformCache.AddOrUpdate(transformerDefinition.TransfomerId, transformer, (s, viewGenerator) => transformer); if (configuration.RunInMemory == false) { WriteTransformerDefinition(transformerDefinition); } return(transformer.Name); }
public string CreateTransform(TransformerDefinition transformerDefinition, AbstractTransformer transformer) { transformCache.AddOrUpdate(transformerDefinition.TransfomerId, transformer, (s, viewGenerator) => transformer); return(transformer.Name); }