private UploadModResult UploadMod(Action <double> progressCallback = null, Action <TaskbarItemProgressState> setTaskbarProgressState = null) { #region online fetch //Fetch current production manifest for mod (it may not exist) setTaskbarProgressState?.Invoke(TaskbarItemProgressState.Indeterminate); using var wc = new System.Net.WebClient(); try { CurrentActionText = M3L.GetString(M3L.string_checkingIfUpdaterServiceIsConfiguredForMod); string validationUrl = $@"{UpdaterServiceCodeValidationEndpoint}?updatecode={mod.ModClassicUpdateCode}&updatexmlname={mod.UpdaterServiceServerFolderShortname}.xml"; string isBeingServed = wc.DownloadStringAwareOfEncoding(validationUrl); if (string.IsNullOrWhiteSpace(isBeingServed) || isBeingServed != @"true") //we don't parse for bool because it might have a different text that is not specifically true or false. It might // have an error for example { //Not being served Log.Error(@"This mod is not configured for serving on the Updater Service. Please contact Mgamerz."); CurrentActionText = M3L.GetString(M3L.string_serverNotConfiguredForModContactMgamerz); HideChangelogArea(); return(UploadModResult.NOT_BEING_SERVED); } } catch (Exception ex) { Log.Error(@"Error validating mod is configured on updater service: " + ex.Message); CurrentActionText = M3L.GetString(M3L.string_interp_errorCheckingUpdaterServiceConfiguration, ex.Message); HideChangelogArea(); return(UploadModResult.ERROR_VALIDATING_MOD_IS_CONFIGURED); } #endregion #region get current production version to see if we should prompt user var latestVersionOnServer = OnlineContent.GetLatestVersionOfModOnUpdaterService(mod.ModClassicUpdateCode); if (latestVersionOnServer != null) { if (latestVersionOnServer >= mod.ParsedModVersion) { bool cancel = false; setTaskbarProgressState?.Invoke(TaskbarItemProgressState.Paused); Application.Current.Dispatcher.Invoke(delegate { // server is newer or same as version we are pushing var response = M3L.ShowDialog(mainwindow, M3L.GetString(M3L.string_interp_dialog_serverVersionSameOrNewerThanLocal, mod.ParsedModVersion, latestVersionOnServer), M3L.GetString(M3L.string_serverVersionSameOrNewerThanLocal), MessageBoxButton.YesNo, MessageBoxImage.Warning); if (response == MessageBoxResult.No) { CurrentActionText = M3L.GetString(M3L.string_uploadAbortedModOnServerIsSameOrNewerThanLocalOneBeingUploaded); HideChangelogArea(); cancel = true; return; } }); setTaskbarProgressState?.Invoke(TaskbarItemProgressState.Indeterminate); if (cancel) { return(UploadModResult.ABORTED_BY_USER_SAME_VERSION_UPLOADED); } } } #endregion #region mod variables //get refs var files = mod.GetAllRelativeReferences(true); files = files.OrderByDescending(x => new FileInfo(Path.Combine(mod.ModPath, x)).Length).ToList(); long totalModSizeUncompressed = files.Sum(x => new FileInfo(Path.Combine(mod.ModPath, x)).Length); #endregion #region compress and stage mod void updateCurrentTextCallback(string newText) { CurrentActionText = newText; } bool?canceledCheckCallback() => CancelOperations; CurrentActionText = M3L.GetString(M3L.string_compressingModForUpdaterService); progressCallback?.Invoke(0); setTaskbarProgressState?.Invoke(TaskbarItemProgressState.Normal); var lzmaStagingPath = OnlineContent.StageModForUploadToUpdaterService(mod, files, totalModSizeUncompressed, canceledCheckCallback, updateCurrentTextCallback, progressCallback); #endregion if (CancelOperations) { AbortUpload(); return(UploadModResult.ABORTED_BY_USER); } setTaskbarProgressState?.Invoke(TaskbarItemProgressState.Indeterminate); #region hash mod and build server manifest CurrentActionText = M3L.GetString(M3L.string_buildingServerManifest); long amountHashed = 0; ConcurrentDictionary <string, SourceFile> manifestFiles = new ConcurrentDictionary <string, SourceFile>(); Parallel.ForEach(files, new ParallelOptions() { MaxDegreeOfParallelism = Math.Max(1, Environment.ProcessorCount - 1) }, x => { if (CancelOperations) { return; } SourceFile sf = new SourceFile(); var sFile = Path.Combine(mod.ModPath, x); var lFile = Path.Combine(lzmaStagingPath, x + @".lzma"); sf.hash = Utilities.CalculateMD5(sFile); sf.lzmahash = Utilities.CalculateMD5(lFile); var fileInfo = new FileInfo(sFile); sf.size = fileInfo.Length; sf.timestamp = fileInfo.LastWriteTimeUtc.Ticks; sf.relativefilepath = x; sf.lzmasize = new FileInfo(lFile).Length; manifestFiles.TryAdd(x, sf); var done = Interlocked.Add(ref amountHashed, sf.size); CurrentActionText = M3L.GetString(M3L.string_buildingServerManifest) + $@" {Math.Round(done * 100.0 / totalModSizeUncompressed)}%"; }); if (CancelOperations) { AbortUpload(); return(UploadModResult.ABORTED_BY_USER); } //Build document XmlDocument xmlDoc = new XmlDocument(); XmlNode rootNode = xmlDoc.CreateElement(@"mod"); xmlDoc.AppendChild(rootNode); foreach (var mf in manifestFiles) { if (CancelOperations) { AbortUpload(); return(UploadModResult.ABORTED_BY_USER); } XmlNode sourceNode = xmlDoc.CreateElement(@"sourcefile"); var size = xmlDoc.CreateAttribute(@"size"); size.InnerText = mf.Value.size.ToString(); var hash = xmlDoc.CreateAttribute(@"hash"); hash.InnerText = mf.Value.hash; var lzmasize = xmlDoc.CreateAttribute(@"lzmasize"); lzmasize.InnerText = mf.Value.lzmasize.ToString(); var lzmahash = xmlDoc.CreateAttribute(@"lzmahash"); lzmahash.InnerText = mf.Value.lzmahash; var timestamp = xmlDoc.CreateAttribute(@"timestamp"); timestamp.InnerText = mf.Value.timestamp.ToString(); sourceNode.InnerText = mf.Key; sourceNode.Attributes.Append(size); sourceNode.Attributes.Append(hash); sourceNode.Attributes.Append(lzmasize); sourceNode.Attributes.Append(lzmahash); sourceNode.Attributes.Append(timestamp); rootNode.AppendChild(sourceNode); } if (CancelOperations) { AbortUpload(); return(UploadModResult.ABORTED_BY_USER); } foreach (var bf in mod.UpdaterServiceBlacklistedFiles) { if (CancelOperations) { AbortUpload(); return(UploadModResult.ABORTED_BY_USER); } var bfn = xmlDoc.CreateElement(@"blacklistedfile"); bfn.InnerText = bf; rootNode.AppendChild(bfn); } if (CancelOperations) { AbortUpload(); return(UploadModResult.ABORTED_BY_USER); } var updatecode = xmlDoc.CreateAttribute(@"updatecode"); updatecode.InnerText = mod.ModClassicUpdateCode.ToString(); rootNode.Attributes.Append(updatecode); var version = xmlDoc.CreateAttribute(@"version"); version.InnerText = mod.ParsedModVersion.ToString(); rootNode.Attributes.Append(version); var serverfolder = xmlDoc.CreateAttribute(@"folder"); serverfolder.InnerText = mod.UpdaterServiceServerFolder; rootNode.Attributes.Append(serverfolder); setTaskbarProgressState?.Invoke(TaskbarItemProgressState.Indeterminate); #endregion //wait to ensure changelog is set. while (ChangelogNotYetSet) { setTaskbarProgressState?.Invoke(TaskbarItemProgressState.Paused); if (CancelOperations) { AbortUpload(); return(UploadModResult.ABORTED_BY_USER); } CurrentActionText = M3L.GetString(M3L.string_waitingForChangelogToBeSet); Thread.Sleep(250); //wait for changelog to be set. } setTaskbarProgressState?.Invoke(TaskbarItemProgressState.Indeterminate); #region Finish building manifest var changelog = xmlDoc.CreateAttribute(@"changelog"); changelog.InnerText = ChangelogText; rootNode.Attributes.Append(changelog); using var stringWriter = new StringWriterWithEncoding(Encoding.UTF8); XmlWriterSettings settings = new XmlWriterSettings(); settings.Indent = true; settings.IndentChars = @" "; settings.Encoding = Encoding.UTF8; using var xmlTextWriter = XmlWriter.Create(stringWriter, settings); xmlDoc.WriteTo(xmlTextWriter); xmlTextWriter.Flush(); #endregion var finalManifestText = stringWriter.GetStringBuilder().ToString(); #region Connect to ME3Tweaks CurrentActionText = M3L.GetString(M3L.string_connectingToME3TweaksUpdaterService); Log.Information(@"Connecting to ME3Tweaks as " + Username); string host = @"ftp.me3tweaks.com"; string username = Username; string password = Settings.DecryptUpdaterServicePassword(); using SftpClient sftp = new SftpClient(host, username, password); sftp.Connect(); Log.Information(@"Connected to ME3Tweaks over SSH (SFTP)"); CurrentActionText = M3L.GetString(M3L.string_connectedToME3TweaksUpdaterService); var serverFolderName = mod.UpdaterServiceServerFolderShortname; //sftp.ChangeDirectory(LZMAStoragePath); //Log.Information(@"Listing files/folders for " + LZMAStoragePath); //var lzmaStorageDirectoryItems = sftp.ListDirectory(LZMAStoragePath); var serverModPath = LZMAStoragePath + @"/" + serverFolderName; bool justMadeFolder = false; if (!sftp.Exists(serverModPath)) { CurrentActionText = M3L.GetString(M3L.string_creatingServerFolderForMod); Log.Information(@"Creating server folder for mod: " + serverModPath); sftp.CreateDirectory(serverModPath); justMadeFolder = true; } var dirContents = sftp.ListDirectory(serverModPath).ToList(); Dictionary <string, string> serverHashes = new Dictionary <string, string>(); //Open SSH connection as we will need to hash files out afterwards. Log.Information(@"Connecting to ME3Tweaks Updater Service over SSH (SSH Shell)"); using SshClient sshClient = new SshClient(host, username, password); sshClient.Connect(); Log.Information(@"Connected to ME3Tweaks Updater Service over SSH (SSH Shell)"); if (!justMadeFolder && dirContents.Any(x => x.Name != @"." && x.Name != @"..")) { CurrentActionText = M3L.GetString(M3L.string_hashingFilesOnServerForDelta); Log.Information(@"Hashing existing files on server to compare for delta"); serverHashes = getServerHashes(sshClient, serverFolderName, serverModPath); } //Calculate what needs to be updated or removed from server List <string> filesToUploadToServer = new List <string>(); List <string> filesToDeleteOffServer = new List <string>(); //Files to upload foreach (var sourceFile in manifestFiles) { //find matching server file if (serverHashes.TryGetValue(sourceFile.Key.Replace('\\', '/') + @".lzma", out var matchingHash)) { //exists on server, compare hash if (matchingHash != sourceFile.Value.lzmahash) { //server hash is different! Upload new file. Log.Information(@"Server version of file is different from local: " + sourceFile.Key); filesToUploadToServer.Add(sourceFile.Key); } else { Log.Information(@"Server version of file is same as local: " + sourceFile.Key); } } else { Log.Information(@"Server does not have file: " + sourceFile.Key); filesToUploadToServer.Add(sourceFile.Key); } } //Files to remove from server foreach (var serverfile in serverHashes.Keys) { if (!manifestFiles.Any(x => (x.Key + @".lzma") == serverfile.Replace('/', '\\'))) { Log.Information(@"File exists on server but not locally: " + serverfile); filesToDeleteOffServer.Add(serverfile); } } #endregion long amountUploaded = 0, amountToUpload = 1; //Confirm changes if (filesToDeleteOffServer.Any() || filesToUploadToServer.Any()) { var text = M3L.GetString(M3L.string_interp_updaterServiceDeltaConfirmationHeader, mod.ModName); if (filesToUploadToServer.Any()) { text += M3L.GetString(M3L.string_nnFilesToUploadToServern) + @" " + string.Join('\n' + @" - ", filesToUploadToServer); //weird stuff to deal with localizer } if (filesToDeleteOffServer.Any()) { text += M3L.GetString(M3L.string_nnFilesToDeleteOffServern) + @" " + string.Join('\n' + @" - ", filesToDeleteOffServer); //weird stuff to deal with localizer } text += M3L.GetString(M3L.string_interp_updaterServiceDeltaConfirmationFooter); bool performUpload = false; Log.Information(@"Prompting user to accept server delta"); setTaskbarProgressState?.Invoke(TaskbarItemProgressState.Paused); Application.Current.Dispatcher.Invoke(delegate { performUpload = M3L.ShowDialog(mainwindow, text, M3L.GetString(M3L.string_confirmChanges), MessageBoxButton.OKCancel, MessageBoxImage.Exclamation) == MessageBoxResult.OK; }); setTaskbarProgressState?.Invoke(TaskbarItemProgressState.Indeterminate); if (performUpload) { Log.Information(@"User has accepted the delta, applying delta to server"); #region upload files //Create directories SortedSet <string> directoriesToCreate = new SortedSet <string>(); foreach (var f in filesToUploadToServer) { string foldername = f; var lastIndex = foldername.LastIndexOf(@"\"); while (lastIndex > 0) { foldername = foldername.Substring(0, lastIndex); directoriesToCreate.Add(foldername.Replace('\\', '/')); lastIndex = foldername.LastIndexOf(@"\"); } } #endregion //UploadDirectory(sftp, lzmaStagingPath, serverModPath, (ucb) => Debug.WriteLine("UCB: " + ucb)); var dirsToCreateOnServerSorted = directoriesToCreate.ToList(); dirsToCreateOnServerSorted.Sort((a, b) => a.Length.CompareTo(b.Length)); //short to longest so we create top levels first! int numFoldersToCreate = dirsToCreateOnServerSorted.Count(); int numDone = 0; if (dirsToCreateOnServerSorted.Count > 0) { CurrentActionText = M3L.GetString(M3L.string_creatingModDirectoriesOnServer); foreach (var f in dirsToCreateOnServerSorted) { var serverFolderStr = serverModPath + @"/" + f; if (!sftp.Exists(serverFolderStr)) { Log.Information(@"Creating directory on server: " + serverFolderStr); sftp.CreateDirectory(serverFolderStr); } else { Log.Information(@"Server folder already exists, skipping: " + serverFolderStr); } numDone++; CurrentActionText = M3L.GetString(M3L.string_creatingModDirectoriesOnServer) + @" " + Math.Round(numDone * 100.0 / numFoldersToCreate) + @"%"; } } //Upload files progressCallback?.Invoke(0); setTaskbarProgressState?.Invoke(TaskbarItemProgressState.Normal); amountToUpload = filesToUploadToServer.Sum(x => new FileInfo(Path.Combine(lzmaStagingPath, x + @".lzma")).Length); foreach (var file in filesToUploadToServer) { if (CancelOperations) { AbortUpload(); return(UploadModResult.ABORTED_BY_USER); } var fullPath = Path.Combine(lzmaStagingPath, file + @".lzma"); var serverFilePath = serverModPath + @"/" + file.Replace(@"\", @"/") + @".lzma"; Log.Information(@"Uploading file " + fullPath + @" to " + serverFilePath); long amountUploadedBeforeChunk = amountUploaded; using Stream fileStream = new FileStream(fullPath, FileMode.Open); sftp.UploadFile(fileStream, serverFilePath, true, (x) => { if (CancelOperations) { CurrentActionText = M3L.GetString(M3L.string_abortingUpload); return; } amountUploaded = amountUploadedBeforeChunk + (long)x; var uploadedHR = ByteSize.FromBytes(amountUploaded).ToString(@"0.00"); var totalUploadHR = ByteSize.FromBytes(amountToUpload).ToString(@"0.00"); if (amountToUpload > 0) { progressCallback?.Invoke(amountUploaded * 1.0 / amountToUpload); } CurrentActionText = M3L.GetString(M3L.string_interp_uploadingFilesToServerXY, uploadedHR, totalUploadHR); }); } setTaskbarProgressState?.Invoke(TaskbarItemProgressState.Indeterminate); if (CancelOperations) { AbortUpload(); return(UploadModResult.ABORTED_BY_USER); } //delete extra files int numdone = 0; foreach (var file in filesToDeleteOffServer) { CurrentActionText = M3L.GetString(M3L.string_interp_deletingObsoleteFiles, numdone, filesToDeleteOffServer.Count); var fullPath = $@"{LZMAStoragePath}/{serverFolderName}/{file}"; Log.Information(@"Deleting unused file off server: " + fullPath); sftp.DeleteFile(fullPath); numdone++; } //Upload manifest using var manifestStream = finalManifestText.ToStream(); var serverManifestPath = $@"{ManifestStoragePath}/{serverFolderName}.xml"; Log.Information(@"Uploading manifest to server: " + serverManifestPath); sftp.UploadFile(manifestStream, serverManifestPath, true, (x) => { var uploadedAmountHR = ByteSize.FromBytes(amountUploaded).ToString(@"0.00"); var uploadAmountTotalHR = ByteSize.FromBytes(amountToUpload).ToString(@"0.00"); CurrentActionText = M3L.GetString(M3L.string_uploadingUpdateManifestToServer) + $@"{uploadedAmountHR}/{uploadAmountTotalHR}"; }); } else { Log.Warning(@"User has declined uploading the delta. We will not change anything on the server."); CancelOperations = true; AbortUpload(); return(UploadModResult.ABORTED_BY_USER); } CurrentActionText = M3L.GetString(M3L.string_validatingModOnServer); Log.Information(@"Verifying hashes on server for new files"); var newServerhashes = getServerHashes(sshClient, serverFolderName, serverModPath); var badHashes = verifyHashes(manifestFiles, newServerhashes); if (badHashes.Any()) { CurrentActionText = M3L.GetString(M3L.string_someHashesOnServerAreIncorrectContactMgamerz); return(UploadModResult.BAD_SERVER_HASHES_AFTER_VALIDATION); } else { CurrentActionText = M3L.GetString(M3L.string_modUploadedToUpdaterService); return(UploadModResult.UPLOAD_OK); } } return(UploadModResult.ABORTED_BY_USER); }