To use Language Forge go to languageforge.org.

To use Scripture Forge go to beta.scriptureforge.org.

To report an issue with the Language Forge application, email "languageforgeissues @ sil dot org".

To report an issue with the Scripture Forge application, email "scriptureforgeissues @ sil dot org".

The rest of this document discusses the development of the underlying software.

• Users
  • Report problems
• Sites
• Special Thanks To
• Developers
  • Builds
  • Gitflow
  • Style Guides
  • Layout
  • Development Environment
    • Vagrant Development Machine
    • Local Linux Development Setup
    • Manual Setup
  • Development Process
  • Reference
• Testing
  • .NET Unit Testing
  • Node Unit Testing
    • Debugging Unit Tests
  • Angular Linting and Prettiering
  • Angular Unit Testing
    • Debugging Unit Tests
    • Filtering Unit Tests
  • Angular End-To-End (E2E) Testing
    • Debugging E2E Tests
  • PWA Testing
  • Physical Device Testing
• Backend Development
  • Model Changes
• Debugging
• Database
• USX Validation

This repository is for Scripture Forge version 2. See also the repository for Language Forge and Scripture Forge version 1.

For end-to-end test automation:

For error reporting:

Status of builds from our continuous integration (CI) server:

Site	Master Unit	Master E2E	QA	Live
Scripture Forge v2	Angular test coverage not reported here but available in TC	not yet operational

Successful builds from our CI server deploy to:

We use Gitflow with some modifications:

• Our master branch is the Gitflow develop development branch. All pull requests go against master.
• Our qa branch is similar to a Gitflow release branch.
• Our live branch is the the Gitflow master production branch.

If you are working on a site Beta then it looks like normal Gitflow and pull requests go against the relevant site development branch.

We merge from master to the QA testing branch, then ship from the QA branch to the live production branch.

TypeScript follows the Angular Style Guide. This is opinionated not only about things like file name conventions but also file and folder structure.

We use Prettier with a pre-commit hook.

Microsoft .NET library design guidelines.

We use Angular Flex-Layout with Angular MDC including the Material Design Icons.

To generate src/SIL.XForge.Scripture/wwwroot/css/sf.min.css, install VS Code extension "ritwickdey.live-sass", which can watch and re-generate it upon changes to wwwroot/scss files, or manually with "Compile Sass" from the VS Code Command Palette.

• Vagrant GUI Setup. A Vagrant box with xForge already installed is downloaded and set up on your machine. This is the easiest and cleanest to setup.
• Local Linux Development Setup. Everything is installed directly on your machine, which needs to be running Ubuntu 16.04. This is the fastest method because development is not done in a virtual machine.
• Manual Setup This setup is specifically written for Windows but the steps could be used for any OS.

Install VirtualBox, Vagrant, and git. To do this in Linux, run

sudo apt install vagrant virtualbox virtualbox-guest-additions-iso git

Setup git. At least name and email is needed in .gitconfig. You can do this from a terminal in your host machine by running

git config --global user.name "My Name"
git config --global user.email "me@example.com"

Hardware-assisted virtualization (VT-x or AMD-V) needs to be enabled in your BIOS.

Clone the xforge git repository to access (and later receive updates to) the vagrant development machine configuration file:

git clone https://github.com/sillsdev/web-xforge
cd web-xforge/deploy/vagrant/sfdev

Run vagrant up. This will download, initialize, and run the development machine. The machine is about 7GB, so expect the download to take a while.

In the guest development machine, take note of the machine-instructions.txt file on the desktop. Set local server secrets. Then do the following:

sudo apt update
sudo apt upgrade
cd ~/src/web-xforge/src/SIL.XForge.Scripture
dotnet run

In the guest development machine, browse to http://localhost:5000/projects and log in.

Start by installing Git and Ansible:

sudo add-apt-repository ppa:ansible/ansible
sudo apt update
sudo apt install -y git ansible

Now create a directory for installation and clone the repo:

git clone --recurse-submodules https://github.com/sillsdev/web-xforge

The --recurse-submodules is used to fetch many of the Ansible roles used by the Ansible playbooks in the deploy folder. If you've already cloned the repo without --recurse-submodules, run git submodule update --init --recursive to pull and initialize them.

Change the variable mongo_path: /var/lib/mongodb in deploy/vars/config_palaso.yml. Set it to a location where MongoDB should store its databases.

• Vagrant VM Setup: uncomment line 5 and comment line 4
• Local Linux Development Setup: uncomment line 4 and comment line 5 (or whatever is appropriate on your system, its best to have Mongo store databases on your HDD rather than SSD). Make sure the mongodb user has permission to read and write to the path you specify.

Run the following Ansible playbook (works on Ubuntu 16.04 thru 20.04 (focal)).

cd web-xforge/deploy
ansible-playbook playbook_focal.yml --limit localhost -K

Add developer secrets. Ask another developer how to get these.

In src/SIL.XForge.Scripture/, run dotnet run. Browse to http://localhost:5000.

Although this setup is specifically written for Windows, the steps could be used for any OS and only step 3 is a Windows specific link. The order below is not particulalry important.

1. Install git, e.g. Git Kraken

2. Clone the repo from the command line including recursing submodules (feel free to clone with SSH instead of HTTPS):

   git clone --recurse-submodules https://github.com/sillsdev/web-xforge.

3. Install MongoDB v4 as a service

4. Install .Net Core SDK-3.1

5. Install Node v12

6. Install FFmpeg v4 executable into the C:\usr\bin\ directory.

7. Install a developer editor, VS Code is recommended (the repo includes VS Code settings)

8. Install Mercurial v4.8 (python 2) and copy contents into the C:\usr\local\bin\ directory.

9. Create folders owned by you. Check in the Ansible deploy/dependencies.yml for the valid list of folders. As of writing they were:

• /var/lib/scriptureforge/sync/
• /var/lib/scriptureforge/audio/
• /var/lib/xforge/avatars/

On Windows, just put these off your root drive, e.g. C:\var\lib\...

10. Add developer secrets. Ask another developer how to get these.
11. In src/SIL.XForge.Scripture/, run dotnet run. Browse to http://localhost:5000.

The first task on a job is to create a feature branch. Branch off of the master branch.

git checkout master
git pull
git checkout -b feature/<featureName>

Do some useful work and commit it.

Upload your work:

git push origin feature/<featureName>

Rebase often (at least at the start of the day, and before making a PR). Force pushing to your own branch is fine (even during review).

Make PR's against the master branch. If the master branch has moved on since the feature branch was made, rebase your changes on top of the master branch before making your PR.

Ensure all tests are passing before submitting a PR.

We use Reviewable for GitHub Pull Requests (PRs). When submitting a PR, a This change is Reviewable link is added to the PR description. Remember to click the Publish button after adding comments in Reviewable.

If the person reviewing feels comfortable to approve it they can. However if they want other eyes on it, mention it in a comment on the PR. If you have minor changes to request on a PR you can say 'Make change X and then LGTM'. This means the person making the PR can merge it themselves after the requested change. People merging PRs can and should rebase the completed PR change (default to squash and rebase unless commits have good reason to stay separate).

Delete the PR branch after merge.

• Angular Tutorial https://angular.io/tutorial
• Angular https://angular.io/api
• Angular MDC https://trimox.github.io/angular-mdc-web/#/angular-mdc-web/button-demo/api
• Angular Material https://material.angular.io/components/categories
• TypeScript https://www.typescriptlang.org/docs/home.html
• JavaScript https://developer.mozilla.org/en-US/docs/Web/JavaScript
• ts-mockito https://github.com/NagRock/ts-mockito#ts-mockito--
• Mockito (for Java Mockito, but helps know how to use ts-mockito) http://static.javadoc.io/org.mockito/mockito-core/2.23.0/org/mockito/Mockito.html

To run .NET backend unit tests, from the repo (repository) root

dotnet test

.NET backend unit tests can also be run using the .NET Test Explorer area of the Test sidebar, or by clicking "Run Test" above a test in a code window. Clicking "Debug Test" will allow debugging.

See documentation for running tests and writing tests.

To run Node backend unit tests, from the repo root

cd src/RealtimeServer
npm test

Jest is the test framework for the Node backend.

Unit tests can be debugged easily in VS Code using the Jest extension. After installing the exension, start the Jest test runner by executing the Jest: Start Runner command in the VS Code command palette. The runner will automatically run all of the Node unit tests and display a green or red circle next to each unit test indicating whether the unit test passed. If the unit test failed, a Debug code lens command will appear above the test. Set a breakpoint and click on the command.

To check TypeScript for readability, maintainability, and functionality errors, and to check a few other files for proper formatting. From the repo root

cd src/SIL.XForge.Scripture/ClientApp/
npm run prettier
ng lint

Or just use VS Code with this project's recommended extensions.

Tests are run by Karma in a browser. Help Karma find Chromium by setting CHROME_BIN. Set it persistently with

tee -a ~/.pam_environment <<< "CHROME_BIN=chromium-browser"

and then log back in to your desktop, or set it temporarily with

export CHROME_BIN=chromium-browser

CHROME_BIN is already set in the vagrant.

Run all the front-end unit tests, automatically re-running when you change a file:

cd src/SIL.XForge.Scripture/ClientApp
npm test

Run tests only in a specific spec file, or only in all spec files in a directory, with the following, where PATH is a relative path to a file or directory. PATH must be relative to ClientApp/src; for example app/sync/sync.component.spec.ts or app/sync.

cd src/SIL.XForge.Scripture/ClientApp/src
npm test -- --include PATH

npm test will monitor and run tests in a Chromium browser window. You can also monitor and run tests headlessly from the command line by running

src/SIL.XForge.Scripture/ClientApp/monitor-test-headless.sh

Or just run tests once without monitoring with

src/SIL.XForge.Scripture/ClientApp/test-headless.sh

You can filter the tests to compile and run by passing spec file names as arguments. For example,

src/SIL.XForge.Scripture/ClientApp/monitor-test-headless.sh some.component.spec.ts another.component.spec.ts

The best way to debug Angular unit tests is with Chrome/Chromium.

• Run npm test (which will include source maps, ng test does not)
• When the Chrome/Chromium window appears, press F12
• Click the Sources tab
• Files might show up under webpack:// or context/localhost:dddd/src or elsewhere, but you can always press CTRL-P and type the name of a file to get there faster.

This video has a live demo of the process.

It is also possible to debug Angular unit tests in VS Code.

• Open the spec file that you want to debug in VS Code.
• Set a breakpoint.
• Navigate to the Debug view.
• Select Karma active spec from the debug dropdown.
• Click the Start Debugging button.

This will run ng test on the active spec file, open Chrome, and attach the VS Code debugger. You can refresh the page by clicking the Restart button in the Debug toolbar.

To run (or not to run) specific tests or fixtures, you can use the prefixes focus and exclude, as in fdescribe or fit to run only the specified functions, or xdescribe and xit to skip running the specified functions (but all functions will still be built). To skip building extra tests, use the --include option when running ng test. See the Angular CLI docs for more info.

See documentation for running tests and writing tests.

To run E2E tests, make sure you are serving the app. From the repo root

cd src/SIL.XForge.Scripture/
dotnet run --environment "Testing"

In another terminal, from the repo root

cd src/SIL.XForge.Scripture/ClientApp/
./rune2e.sh

To debug E2E tests, from the repo root

cd src/SIL.XForge.Scripture/
dotnet run --environment "Testing"

In another terminal, from the repo root

cd src/SIL.XForge.Scripture/ClientApp/
ng serve

Add a new line of debugger; to the *.e2e-spec.ts where you want it to break.

In another terminal, from the repo root

cd src/SIL.XForge.Scripture/ClientApp/
./rune2e.sh debug

Open chrome://inspect/#devices in Chromium and click inspect. This opens an instance of DevTools and immediately breaks the code at the top of the ng module. Click the continue button (or press F8) in your debugger to run your e2e tests, and hit any debugger statements in your code. Close the DevTools window to finish the tests.

To test the PWA (Progressive Web App), build the app for PWA testing and run the server without ng serve. From the repo root

cd src/SIL.XForge.Scripture/ClientApp/
ng build --configuration=pwaTest

In another terminal, from the repo root

cd src/SIL.XForge.Scripture/
dotnet run --start-ng-serve=no

!!! IMPORTANT !!! When you have finished testing, remove the built app dist folder. From the repo root

rm -rf src/SIL.XForge.Scripture/ClientApp/dist

To test localhost on a physical device you need to ensure you have the following:

• An Android device
• The latest version of Chrome on your desktop
• The latest version of Chrome on your device
• USB debugging enabled on your device

Enabling USB debugging first involves switching on developer mode on your device. This usually involves going to your System->About screen and tapping repeatedly on your build number. Doing a Google search for, "How to enable USB Debugging on {device name}" should help get it enabled. Once the device is developer mode there will be a new developer link on your device - likely under system - and in there you'll find an option to toggle on USB debugging.

With USB debugging turned on navigate in Chrome on your desktop to chrome://inspect/#devices. Your device may at this point present a prompt to allow your Chrome browser access to debug your device - you must allow this. Once allowed your device name will appear in the list of available devices. Before you can run the app you first need to update the Port forwarding settings to reflect our app. Click the button on the inspection screen and set as follows:

• 5000    |    localhost:5000
• 5003    |    localhost:5003

You're now ready to go. Build the app as normal on your desktop using dotnet run or ng serve or whichever variation you need. Once the build is complete, open up Chrome on your device and navigate to localhost:5000. On your desktop the inspection page will show all the open tabs you have in Chrome on the device. Locate the tab running http://localhost:5000/ and click the inspect link. A new window will open with the Chrome DevTools giving you full access to the console, DOM, network, screen sharing, etc.

Please note that if your device goes to sleep or switches to the lock screen then the USB debugging will no longer have permission.

Normally when you run dotnet run it starts ng serve for you. This works great if you are developing on the front end as it watches for file changes and reloads your browser once it has compiled.

If you are developing on the backend this works better

cd src/SIL.XForge.Scripture/
dotnet watch run --start-ng-serve=listen

In another terminal

cd src/SIL.XForge.Scripture/ClientApp/
ng serve

When files change on the backend it will compile the changes automatically and now ng serve won't re-start every time.

See the Debugging section below for how to do this in VS Code.

The Angular app has a dependency on the Node backend NPM package so that it has access to the model types. If the models are changed, the Angular app will not see the changes until the backend package is rebuilt. You can rebuild the backend by executing the following commands:

cd src/RealtimeServer/
npm run build

If a model change is made, then a corresponding data migration should be implemented in the Node backend. A data migration is implemented by following these steps:

1. Create a class that extends the Migration base class with the name <collection>Migration<version> in the appropriate collection migrations file. For example, if you are adding a user migration for schema version 10, then you would add the class UserMigration10 to the src/RealtimeServer/common/services/user-migrations.ts file.
2. Implement the migrateDoc method. The submitMigrationOp function MUST be used to submit any migration changes to the doc.
3. Implement the migrateOp method.
4. Add the class to the migrations array in the migrations file.

In Visual Studio Code, in the debug sidebar, choose Full App (SF) to debug the front-end and back-end at the same time, or Launch Chrome (SF) or .NET Core (SF) to just debug the front-end or back-end.

The VS Code extension Azure Cosmos DB can be used to inspect our Mongo DB.

USX data from the Paratext Data Access API is validated in order to ensure that Scripture Forge does not corrupt Scripture data that it does not know how to properly round-trip. If the DeltaUsxMapper class is updated to support new USX data, then Scripture Forge's USX schema should be updated to reflect the corresponding change. The schema is based on the USX Relax NG schema and can be found at src/SIL.XForge.Scripture/usx-sf.rnc. Once the schema is updated, it must be converted to an XML Schema file. The schema can be converted using Trang. On Linux, install Trang using

sudo apt install trang

Convert the schema to XML Schema format by running

cd src/SIL.XForge.Scripture
trang usx-sf.rnc usx-sf.xsd