This is a tool providing multiple forms of compression (Huffman, Arithmetic, a LZ77 variant, and my own LZ format, LZCL) suitable for low resource environments. A C# compression tool does the compression, provides decompression, and a single file decompressor written in C makes it easy to embed decompression in a C environment.

The usefulness of this is it allows very tunable decompression resource requirements for constrained environments.

The compression tools side provides a lot of primitives for developing compression algorithms with many pluggable and advanced pieces.

Neither side is optimized for speed, as both were meant to be simple and clean to read/extend/implement.

From the help in CompressionTester.exe

Usage:
 -i inputfile (can be path to do directory with recurse if no output file)
 -o outputfile (if none, no output files)
 -r recurse directories. No outputs
 -f output formats : C# C Binary
 -c codecs         : Arithmetic Huffman Lz77 Lzcl
    The following codec have parameters that can be selected.
    Select as: -c LZ77:param1=value1:param2=value2
              Lz77:  maxDist - Max distance to look back in buffer
              Lz77:   minLen - Minimum length of a run to be worthwhile
              Lz77:   maxLen - Maximum length of a run to be worthwhile
              Lzcl:  maxDist - Max distance to look back in buffer
              Lzcl:   minLen - Minimum length of a run to be worthwhile
              Lzcl:   maxLen - Maximum length of a run to be worthwhile
 -v verbose
 -t test - run current test set
 -d decompress, else compress

You can take a file, compress in one of 4 methods, output to binary or C code, and specify optional parameters if desired.

The compressors generally compress more in order Huffman, Arithmetic, LZ77, LZCL. For general use LZ77 is very good, but if you really need to fit more data in a small space LZCL will beat it handily.

LZ77 and LZCL allow setting the max lookback window and max length of a run. Lower generally lowers compression, but often the optimal value (exhaustively tested) is not large. The buffer size required for these two for incremental decompression is the max of maxDist and maxLen, plus 1. Setting minDist to other than default 2 has is not generally useful.

To decompress in your project, add Decompressor.c and Decompressor.h. In Decompressor.h, select which compression routines you want by commenting out the ones you don't. LZCL needs all the others and will re-#define them.

#define DECOMPRESSOR_USE_HUFFMAN
#define DECOMPRESSOR_USE_ARITHMETIC
#define DECOMPRESSOR_USE_LZ77
#define DECOMPRESSOR_USE_LZCL

In your file needing decompression, #include "Decompressor.h", then use code like

uint8_t outputBuffer[DECOMPRESSED_SIZE]; 
DecompressHuffman(compressedData, sizeof(compressedData), outputBuffer, DECOMPRESSED_SIZE);

to decompress in one large run. The same interface works with Huffman replaced by Arithmetic, LZ77, and LZCL.

To decompress incrementally, use each like

// incremental decompression
uint32_t symbol, index;

// Huffman
HuffmanState_t huffmanState;
DecompressHuffmanStart(&huffmanState, huffData, sizeof(huffData));
index = 0;
while (1)
{
    symbol = DecompressHuffmanSymbol(&huffmanState);
    if (symbol == CL_COMPRESSOR_END_TOKEN)
        break;
    ... use symbol here...
}

// Arithmetic
ArithmeticState_t arithmeticState;
uint32_t length = DecompressArithmeticStart(&arithmeticState, arithData, sizeof(arithData));
index = 0;
while (index < length) 
{
    symbol = DecompressArithmeticSymbol(&arithmeticState);
    ... use symbol here ...
}

// LZ77
uint8_t localBuffer[<size determined by compression parameters>]; 
uint32_t destIndex = 0, srcIndex = 0;
LZ77State_t lz77State;
DecompressLZ77Start(&lz77State, lz77Data, sizeof(lz77Data),localBuffer,sizeof(localBuffer));
destIndex = srcIndex = 0;
while (1)
{
    uint32_t runLength = DecompressLZ77Block(&lz77State);
    if (runLength == CL_COMPRESSOR_END_TOKEN)
        break;
    // copy out run
    while (runLength-- > 0)
        ... use byte from localBuffer[(srcIndex++)%sizeof(localBuffer)]....
}

LZCLState_t lzclState;
DecompressLZCLStart(&lzclState, lzclData, sizeof(lzclData),localBuffer,sizeof(localBuffer));
destIndex = srcIndex = 0;
while (1)
{
    uint32_t runLength = DecompressLZCLBlock(&lzclState);
    if (runLength == CL_COMPRESSOR_END_TOKEN)
        break;
    // copy out run
    while (runLength-- > 0)
        ... use byte from localBuffer[(srcIndex++)%sizeof(localBuffer)]....
}

And that's it!

Compression is generally not very fast, since the algorithms are designed to be easily extended, munged, and used to create more formats if needed. Decompression code is designed to be small, not fast, for the use case I wrote this code for. Decompressor.c fits all four decompression routines into a self contained 750ish line C file (~100 for Huffman, ~200 for arithmetic, ~75 for LZ77, ~250 LZCL, rest support.).

The PIC32 sample code, running on a PIC32MX150F128B at 40MHZ, compiled with x32-gcc v1.31, -O3 optimized code, obtains the following decompression rates

This was tested on a compressed version of the 26217 byte file Decompressor.c, with both LZ variants set to allow decoding incrementally into a 257 byte buffer.

A common test for smaller files is the Calgary Corpus and Cantebury Corpus

Finally, LZ77 and LZCL can vary drastically with different parameters - the above parameters are not tuned to be optimal, which can be done by exhaustively trying them.

The code includes a Visual Studio 2015 project that contains the C# compressor, a C DLL to test decompression, and also includes a PIC32 sample program to illustrate usage and to measure decompression speed.

My LZCL format is a very bit optimized format, separating the usual LZ77 streams into components

• Decisions: 0/1 to tell next token type
• Literals: single bytes encoded
• Tokens: each is a distance, length pair, encoded in one of two methods

It was designed to solve a use case I had to get the best compression I could, subject to very small code size and very low memory usage for an embedded project. I may finalize the spec carefully (it requires careful specification of multiple other sub-formats... a lot of work) one day.

Some advanced codecs that are small are used here: Golomb coding and Binary Adaptive Sequential Coding (BASC).

Then, decisions is either stored as a bitstream of 0/1, or converted to runs (basic RLE compression). Tokens are tested as both (distance,length) pairs and encoded similarly to length*(maxDistance+1)+distance. All these variants are tested across all supported compression formats (Fixed length, Huffman, Arithmetic, Golomb Coding), and the best combination is chosen.

All formats contribute a decent amount on the corpus tests above.

I plan to write two longer articles on all of this at (or possibly at lomont.org, once I merge sites):

http://clomont.com/programming/compression-notes/ http://clomont.com/software/lzcl-an-embedded-compression-library/

Happy hacking! Chris Lomont