Skip to content

Kystverket/GravityVectorToolkit

 
 

Repository files navigation

Gravity Vector Toolkit

Tools to import gravity vectors, normal routes and deviation cells from CSV files into a PostGIS database. Also includes tools for combining AIS data with weather data from thredds.met.no.

Requirements:

  • Visual Studio 2019 Community Edition (only required to compile from source)
  • .NET Core 3.1 SDK (or newer)

How to use GravityVectorToolkit.Tools.DatabaseImport

When using a compiled program, DatabaseImport can be executed as follows: $ dotnet run GravityVectorToolkit.Tools.DatabaseImport.exe --connectionstring <str> --gv-merged-path <path> --normal-route-path <path> [--deviation-cells-path <path>]

DatabaseImport can also be run directly from source. Simply cd into the GravityVectorToolkit.Tools.DatabaseImport project folder and run:

$ dotnet build 
$ dotnet run --connectionstring <str> --gv-merged-path <path> --normal-route-path <path> [--deviation-cells-path <path>]

When using Visual Studio, GVTK can be executed by running the project GravityVectorToolkit.Tools.DatabaseImport. However, the command line arguments must be entered into launchSettings.json or added to the application arguments text box in the project properties view.

Available options:

Name Required Description
--connectionstring <str> Yes A full connection string to a PostgreSQL database with PostGIS enabled.
--gv-merged-path <path> Yes The path to a merged gravity vector file (output from MADART).
--normal-route-path <path> Yes The path to a merged normal routes file (output from MADART).
--deviation-cells-path <path> No The path to a deviation cell file (output from MADART).

How to use GravityVectorToolKit.Tools.AisCombine

When using a compiled program, AisCombine can be executed as follows: $ dotnet run GravityVectorToolkit.Tools.AisCombine.exe

AisCombine reads its settings from a configuration file called settings.json. Here is an example settings.json file:

{
   "AisSourceDirectory":"/path/to/inputdir",
   "AisDestinationDirectory":"/path/to/outputdir",
   "WeatherDbPath":"/path/to/weather-database.sqlite",
   "MaxParallellism":8,
   "AssumeNoEpochCollissions":true,
   "TimestampColumnName":"date_time_utc",
   "LatitudeColumnName":"lat",
   "LongitudeColumnName":"lon",
   "CalculateClosest":true,
   "GeohashMatchPrecision":5,
   "PrecisionSearchLimit":3
}

Available settings:

Name Required Default value Description
AisSourceDirectory Yes n/a AIS source file directory containing AIS files in CSV format.
AisDestinationDirectory Yes n/a Destination directory (requires approx 150% capacity compared to the total input file sizes).
AssumeNoEpochCollissions Yes true Set this to true if the files containing AIS data is split on days. If runtime concurrency errors occur, set this to false. Medium to high negative performance impact.
CalculateClosest Yes true Higher precision when matching positions with weather data. Very low negative performance impact, but with greatly improved end result.
Delimiter No ";" Which delimiter to use to parse the input CSV files.
GeohashMatchPrecision Yes 5 The initial geohash precision to search for weather data (includes neighbours). Should be roughly equivalent to the resolution of the weather data set.
LatitudeColumnName Yes "lat" The name of the latitude column in the source AIS files.
LongitudeColumnName Yes "lon" The name of the longitude column in the source AIS files.
MaximumEpochAgeMinutes No 15 The maximum age of cached weather epochs before being vacuumed.
MaxParallellism Yes maximum available threads The maximum amount of threads to use. Set this to a lower value if you run out of memory.
PrecisionSearchLimit Yes 3 The lowest geohash precision to search for weather data (includes neighbours). Values higher than 3 is not recommended.
SortBy No "path" Which sort order to use when processing files. Valid settings are "path" and "size".
StatusMessageCycleSeconds No 10 How often the status messages is printed to the console during processing.
TimestampColumnName Yes "date_time_utc" The name of the timestamp column in the source AIS files.
UseDirectSQL No false Set this to true to query the database directly and cache the results, which is faster for CSV-files grouped by MMSI. For CSV files grouped by day, set this to false.
WeatherDbPath Yes n/a The path to the SQLite file containing weather data.