Announcing FFFF

Announcing FFFF

Robbert Haarman

2013-01-06

Posted by inglorion
at 2010-10-16 15:15:50

For a number of years now, I have been walking around with the idea of devising a file format that is flexible enough that it can be used for a variety of purposes, and efficient enough that it doesn't take up a lot of space and can be used for storing and processing of large amounts of data.

I have finally sat down and written up a draft of this idea. It's called FFFF (like Fast FFF, my generic textual format). There is much work to be done yet, but this should be enough for a first draft.

The next step is to write some software to actually use the format, to see how it performs in practice. Using the experience gathered with this experimental software, the format can then be refined and improved.

There are some ideas I have that aren't in the draft yet. In no particular order:

- Adding IEEE floating points as a data type

- An import directive to import a bunch of definitions - like the language directive, but leaving the active encoding and definitions in place

- A minimal subset, just enough to indicate the language being used without all the baggage you don't need if you are going to use a different language anyway

- Compression. Chunks of data can be compressed using a specified compression algorithm. To keep implementation bloat down, I will standardize on a single compression algorithm; probably one of gzip, BWT (bzip2), or LZMA