Skip to content

rasberry/HugeStructures

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

25 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

HugeStructures

Disk-backed structures that can be larger than available RAM

Overview

TitanicArray classes

These classes are designed to emulate a regular array and have the following features:

  • they are a fixed length
  • data can be accessed in a random manner
  • the arrays are Read/Write
  • all extend the ITitanicArray<T> interface

TitanicFileArray

A basic disk-backed array using just a file and and LRU based cache.

TitanicMMFArray

This array is backed by a MemoryMapped file. Pretty much just a wrapper around the MemoryMappedFile class

TitanicSQLiteArray

This array is backed by a SQLite database.

TitanicLiteDBArray

This array is backed by a LiteDB database.

TitanicRaptorDBArray

This array is backed by a RaptorDb database.

TitanicIMArray

This array is backed by Magick.NET
Note: TitanicIMArray sometimes has a problem converting data to float values (the internal storage type). So this array might be unreliable.

Performance

Testing on my machine using 2^20 doubles (4MiB of data) results in these times:

  • TitanicMMFArray
    • Random Sequence: 3s
    • Linear Sequence: 2s
  • TitanicIMArray
    • Random Sequence: 10s
    • Linear Sequence: 9s
  • TitanicRaptorDBArray
    • Random Sequence: 13s
    • Linear Sequence: 12s
  • TitanicFileArray
    • Random Sequence: 25s
    • Linear Sequence: 12s
  • TitanicSQLiteArray
    • Random Sequence: 345s
    • Linear Sequence: 294s
  • TitanicLiteDBArray
    • Random Sequence: ?s (took too long)
    • Linear Sequence: 570s

About

Disk-backed structures that can be larger than available RAM

Topics

Resources

Stars

Watchers

Forks

Packages

No packages published

Languages