Parquet.Net is developed mostly by myself, and although I'm trying to invest some time in this project, it's not always possible to cut out time from my daily job or evenings. If you feel you'd like to sponsor this project, please click the Sponsor button above and choose one of the available options.💰

Build	Test

Fully managed .NET library to read and write Apache Parquet files. Supports:

• .NET 4.5 and up.
• .NET Standard 1.4 and up (for those who are in a tank that means it supports .NET Core (all versions) implicitly)

Runs on all flavors of Windows, Linux, MacOSX, mobile devices (iOS, Android) via Xamarin, gaming consoles or anywhere .NET Standard runs which is a lot!

Performs integration tests with parquet-mr (original Java parquet implementation) to test for identical behavior. We are planning to add more third-party platforms integration as well.

Parquet library is mostly available for Java, C++ and Python, which somewhat limits .NET/C# platform in big data applications. Whereas C# is a beautiful language (C# is just Java done right) working on all platforms and devices, we still don't have anything good in this area. Note that ParquetSharp provides a P/Invoke wrapper around parquet-cpp library, however it's a windows-only version with plenty of limitations around usability, is generally slower and leaks memory.

Parquet.Net is used by many small and large organisations for production workloads. If you are one of them, please email ivan.gavryliuk@outlook.com to be displayed here.

How do we compare to other parquet implementations? We are fast and getting faster with every release. Parquet.Net is dedicated to low memory footprint, small GC pressure and low CPU usage. In this test we are using a file with 8 columns and 150'000 rows, and the result is:

All the parties in this test were given 10 iteration and time was taken as an average. Parquet-Mr was even given a warm-up time being the slowest candidate, so it can fit on the chart.

• Getting Started
• Reading Data
• Writing Data
• Complex Types
• Row-Based API
• Fast Automatic Serialisation
• Declaring Schema
  • Supported Types
• parq!!!
• Sponsorship

You can track the amount of features we have implemented so far.

Parquet.Net is redistributed as a NuGet package. All the code is managed and doesn't have any native dependencies, therefore you are ready to go after referencing the package. This also means the library works on Windows, Linux and MacOS X.

This intro is covering only basic use cases. Parquet format is more complicated when it comes to complex types like structures, lists, maps and arrays, therefore you should read this page if you are planning to use them.

In order to read a parquet file you need to open a stream first. Due to the fact that Parquet utilises file seeking extensively, the input stream must be readable and seekable. You cannot stream parquet data! This somewhat limits the amount of streaming you can do, for instance you can't read a parquet file from a network stream as we need to jump around it, therefore you have to download it locally to disk and then open.

For instance, to read a file c:\test.parquet you would normally write the following code:

using System.Collections.Generic;
using System.IO;
using System.Linq;
using Parquet.Data;

// open file stream
using (Stream fileStream = System.IO.File.OpenRead("c:\\test.parquet"))
{
   // open parquet file reader
   using (var parquetReader = new ParquetReader(fileStream))
   {
      // get file schema (available straight after opening parquet reader)
      // however, get only data fields as only they contain data values
      DataField[] dataFields = parquetReader.Schema.GetDataFields();

      // enumerate through row groups in this file
      for(int i = 0; i < parquetReader.RowGroupCount; i++)
      {
         // create row group reader
         using (ParquetRowGroupReader groupReader = parquetReader.OpenRowGroupReader(i))
         {
            // read all columns inside each row group (you have an option to read only
            // required columns if you need to.
            DataColumn[] columns = dataFields.Select(groupReader.ReadColumn).ToArray();

            // get first column, for instance
            DataColumn firstColumn = columns[0];

            // .Data member contains a typed array of column data you can cast to the type of the column
            Array data = firstColumn.Data;
            int[] ids = (int[])data;
         }
      }
   }
}

Parquet.Net operates on streams, therefore you need to create it first. The following example shows how to create a file on disk with two columns - id and city.

//create data columns with schema metadata and the data you need
var idColumn = new DataColumn(
   new DataField<int>("id"),
   new int[] { 1, 2 });

var cityColumn = new DataColumn(
   new DataField<string>("city"),
   new string[] { "London", "Derby" });

// create file schema
var schema = new Schema(idColumn.Field, cityColumn.Field);

using (Stream fileStream = System.IO.File.Create("c:\\test.parquet"))
{
   using (var parquetWriter = new ParquetWriter(schema, fileStream))
   {
      // create a new row group in the file
      using (ParquetRowGroupWriter groupWriter = parquetWriter.CreateRowGroup())
      {
         groupWriter.WriteColumn(idColumn);
         groupWriter.WriteColumn(cityColumn);
      }
   }
}

Parquet.Net includes API for row-based access that simplify parquet programming at the expense of memory, speed and flexibility. We recommend using column based approacha when you can (examples above) however if not possible use these API as we constantly optimise for speed and use them internally outselves in certain situations.

Parquet.Net is licensed under the MIT license.

Your privacy is important to us. Full details are specified in the privacy statement.

We are desperately looking for new contributors to this project. It's getting a lot of good use in small to large organisations, however parquet format is complicated and we're out of resources to fix all the issues.

For details on how to start see this guide. If you are a developer who is interested in Parquet development please read this guide

This framework is free and can be used for free, open source and commercial applications. Parquet.Net (all code, NuGets and binaries) are under the MIT License (MIT). It's battle-tested and used by many awesome people and organisations. So hit the magic ⭐️ button, we appreciate it!!! 🙏 Thx!

The core team members, Parquet.Net contributors and contributors in the ecosystem do this open source work in their free time. If you use Parquet.Net, and you'd like us to invest more time on it, please donate by pressing the ❤ Sponsor button on top of this page. This project increases your income/productivity/usability too.

If your company/project is using Parquet.Net we'd be happy to list your logo here on the front page with your kind permission, absolutely for free. Please contact ivan.gavryliuk@outlook.com with details and graphics attached.

• Open-Source Maintainers are Jerks! | Nick Randolph & Geoffrey Huntley
• FOSS is free as in toilet | Geoffroy Couprie
• How to Charge for your Open Source | Mike Perham
• Sustain OSS: The Report
• Open Source Maintainers Owe You Nothing | Mike McQuaid
• Who should fund open source projects? | Jane Elizabeth
• Apply at OSS Inc today | Ryan Chenkie
• The Ethics of Unpaid Labor and the OSS Community | Ashe Dryden