Skip to content

After using Azure Data Factory Copy Activity to copy the files (e.g. from S3 to Azure Blob), you can use this tool to verify all files are copied as expected.

brijrajsingh/Validate-Copied-Files

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Inventory audit tool (.NET Core based) for AWS S3 and Azure Blob store

This tool is made up of 3 utilities which can be used seperately or together and can be useful to audit the files copied from AWS S3 to Azure Storage (Blob) and also returns the list of files which are not copied successfully from AWS S3 to Azure Storage and the log about what may be wrong with those files, in case of missing files one can always look at the log of both utilities and figure out which files may be missing. Based on .NET Core works for both Windows and Linux.

Repository strcuture

  • /Source: includes the source codes.
    • FetchBlobFileInfoCore - .net core (2.2) based app to download list of objects in a given Azure Storage Container.
    • FetchS3FileInfoCore - .net core (2.2) based app to download list of objects in a given AWS S3 bucket
    • ValidateCopiedFileLengthCore - Looks at the log of both above mentioned utilities, gives a count and also logs information about files missing in destination Azure storage container.
  • /Tool: includes the executable files for both linux and Windows platforms and shell/batch scripts to run the tool.

.NET Core Installation

Follow steps from here to install .NET Core on your machine https://dotnet.microsoft.com/download. For running the tool you may install the runtime only and for compiling the code opt for SDK installation. The tool is compiled to run on .NET Core 2.2.

About

After using Azure Data Factory Copy Activity to copy the files (e.g. from S3 to Azure Blob), you can use this tool to verify all files are copied as expected.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • C# 98.9%
  • Other 1.1%