7.0.0

Welcome to FlatSharp 7! Version 7 is a major release that includes many under-the-hood improvements such as .NET 7 support, performance improvements for several cases, generated code deduplication, experimental object pools, and some big breaking changes. This is a long list of changes, so buckle up!

Runtime-Only Mode Deprecation

Let's start with the bad news: Reflection-only Runtime mode is not making the jump to FlatSharp 7.

The original version of FlatSharp did not include a compiler or support for FBS files. Instead, it used attribute-based reflection like so many other .NET serializers do with attributes like [FlatBufferTable] which allowed generation of code at runtime. FlatSharp.Compiler was introduced in early 2020, and in version 6 it was ported to use flatc as its parser instead of the custom grammar. Today, there is really no reason to use reflection mode any longer. The FlatSharp compiler is mature and enables all of the same semantics, with a slew of additional benefits over runtime mode:

• Support for AOT / Unity
• Cross-platform compatibility with other FBS compilers such as flatc and flatcc.
• No runtime overhead for code generation
• No package dependencies on Roslyn
• Errors caught at build time rather than runtime
• Custom Union types
• gRPC integration
• Copy/clone constructors
• ...and many others

For these reasons, the trend has been that most new features and development of FlatSharp use the compiler rather than the reflection-only mode. So in version 7, FlatSharp is dropping support for runtime code generation. This is a difficult decision, but one that allows the project to keep moving forward and is broadly aligned to most customer's usage of FlatSharp, along with the bigger trends in .NET, such as AOT and source generators. Going forward, only two packages will see new versions published: FlatSharp.Runtime and FlatSharp.Compiler.

Removing Runtime-only mode removes an entire class of human error from FlatSharp and makes AOT easier to reason about. One immediate benefit of this change is that the FlatSharp.Runtime package now has no internal reflection calls any longer, which makes AOT less error-prone. Unfortunately, it does also remove support for some features such as Type Facades.

Array Vector and fs_nonVirtual Deprecation

In the interest of helping engineers make good decisions about how to use FlatSharp, Array vectors (fs_vector:"Array") and non-virtual properties have been deprecated. These behave very unpredictably with FlatSharp, because...

• They must be accessed greedily, which means they don't perform well at all with Lazy deserialization. Accessing an Array property on a Lazy object simply allocates a new array each time. Not good!
• They cannot satisfy FlatSharp's immutability promises in Lazy, Greedy, and Progressive modes
• They cannot support WriteThrough (fs_writeThrough)
• Virtual methods and interfaces, such as IList<T>, provide much of the performance, with an order of magnitude more flexibility.

When serializing Vectors, FlatSharp does still attempt to devirtualize IList<T> and IReadOnlyList<T> to arrays for the performance boost, but this is more an implementation detail of the handling for IList<T> than it is about continuing to support arrays explicitly.

.NET 7 Support

FlatSharp version 7 supports .NET 7. What a happy coincidence! This is not a major change. However, there are a few things worth calling out.

The first is that IFlatBufferSerializable<T> has a few new members when using .NET 7:

public interface IFlatBufferSerializable<T> where T : class
{
      ISerializer<T> Serializer { get; } // this already existed

#if NET7_0_OR_GREATER
     static abstract ISerializer<T> LazySerializer { get; }
     static abstract ISerializer<T> ProgressiveSerializer { get; }
     static abstract ISerializer<T> GreedySerializer { get; }
     static abstract ISerializer<T> GreedyMutableSerializer { get; }
#endif
}

This allows writing code like this:

public static T Parse<T>(byte[] buffer) where T : class, IFlatBufferSerializable<T>
{
    // All serializer types are available now, as well
    return T.LazySerializer.Parse(buffer);
}

Next, FlatSharp 7 supports required members. That is, when you specify the required FBS attribute, the generated C# also contains that annotation:

table RequiredTable
{
     Numbers : [ int ] (required);
}

Now generates:

public class RequiredTable
{
    public required virtual IList<int> Numbers { get; set; }
}

This is, of course, only available for those of you using .NET 7. If you aren't ready for .NET 7 just yet, then no problem; FlatSharp will continue to generate code that works with .NET 6, .NET Standard 2.0, and .NET Standard 2.1, though you may see #if NET7_0_OR_GREATER peppered throughout your generated code now!

Finally, FlatSharp 7 fully supports .NET 7 AOT. Effort has been made to remove the last vestiges of reflection from the FlatSharp.Runtime package.

Code Deduplication

Previous versions of FlatSharp generated separate code for each root type. That is, imagine this schema:

// A big table
table Common
{
    A : string;
    B : string;
    ...
    Z : string;
}

table Outer1 (fs_serializer) { C : Common }
table Outer2 (fs_serializer) { C : Common }

In this case, FlatSharp 6 and below would generate 2 full serializers and parsers for Common, since it is used by both Outer1 and Outer2. This led to cases where commonly-used large tables would have more than one serializer implementation generated. Such code duplication leads to poor cache performance, poor utilization of the branch predictor, and an explosion of code when the type in question is used in multiple places, such as gRPC definitions.

FlatSharp 7 does the expected thing and generates only one serializer/parser for each type. As part of this change, FlatSharp 7 emits all serializer types, which comes with the happy accident of allowing you to specify it at runtime:

ISerializer<T> lazySerializer = Outer1.Serializer.WithSettings(settings => settings.UseLazyDeserialization());

Performance Improvements

FlatSharp 7 improves the performance of IList<T> vectors by enabling devirtualization of internal method calls. Previous versions of FlatSharp defined the base vector along these lines:

public abstract class BaseVector<T> : IList<T>
{
    public T this[int index]
    {
        get => this.ItemAtIndex(index);
    }

    protected abstract T ItemAtIndex(int index);
}

Virtual methods do have a cost, because the assembly must look first to the vtable to then jump to the actual methods. A better way to write the same code is with this technique:

public interface IVectorAccessor<T>
{
    T GetItemAtIndex(int index);
}

public class Vector<T, TVectorAccessor>
    where TVectorAccessor : struct, IVectorAccessor<T>
{
    private readonly TVectorAccessor accessor;
    
    public T this[int index]
    {
        get => this.accessor.ItemAtIndex(index);
    }
}

But wait! Aren't these the same thing? Both are calling a virtual method after all. The trick is subtle, and involves generics and structs. Stephen Cleary writes about it more clearly than I can here.

This technique is not new to FlatSharp. Support for this trick was added in Version 4, and why IInputBuffer implementations have been structs and the entire Parsing stack is templatized. However, the opportunity to do the same for vectors was only recently discovered, and the improvements are impressive. Iterating through a Lazy vector is often 20-50% faster.

Additionally, VTable parsing has been improved by several whole nanoseconds! Joking aside, this is an important thing since every table has a VTable, so this is one of the operations that FlatSharp does at the very core, and the benefit multiplies with the number of tables you read. Only tables with 8 or fewer fields benefit from this optimization. Larger tables fall back to the previous behavior.

Here's a quick teaser of FlatSharp parse/traverse performance of a vector of structs, both reference and value:

Method	Mean	Error	StdDev	P25	P95	Code Size	Gen0	Allocated
Parse_Vector_Of_ReferenceStruct_FlatSharp7	247.95 ns	22.909 ns	8.169 ns	242.53 ns	258.21 ns	291 B	0.0787	1320 B
Parse_Vector_Of_ReferenceStruct_FlatSharp6	324.56 ns	4.396 ns	1.568 ns	323.32 ns	326.39 ns	263 B	0.0787	1320 B
Parse_Vector_Of_ValueStruct_FlatSharp7	93.86 ns	1.254 ns	0.447 ns	93.71 ns	94.29 ns	279 B	0.0072	120 B
Parse_Vector_Of_ValueStruct_FlatSharp6	194.78 ns	8.377 ns	2.987 ns	192.61 ns	197.42 ns	251 B	0.0072	120 B

Object Pooling

Object Pooling is a technique that can be used to reduce allocations by returning them to a pool and re-initializing them later. FlatSharp 7's object pool is experimental. The intent of this release is to get the feature into the wild and how well it works. With Object Pooling enabled, it is possible to use FlatSharp in true zero-allocation mode. The wiki has full details.

Field Name Normalization

In version 6.2.0, FlatSharp introduced an optional switch to normalize snake_case fields into UpperPascalCase. This was off by default in version 6 for compatibil...

6.3.3

6.3.3 is a minor release of FlatSharp with a couple of bugfixes and a minor new feature.

Bug 1:

#307 -- FlatSharp only generated unions for up to 30 types. Some people seem to need more! Version 6.3.3 includes unions up to 50 generic types. Have fun!

Bug 2:

#299 -- Fixed an issue where FlatSharp would encounter a NullReferenceException when parsing a schema with a circular Table --> Union --> Table chain.

Feature: Union Visitors

Unions now support a native visitor pattern. You can refer to the unions sample for more details. The short version is that Union types now include a Accept method that requires a IFlatBufferUnionVistior<TReturn, T1, T2, ..., TN> visitor. The visitor interface must match the union definition, which adds a layer of compiler validation to ensure that all union cases are handled.

6.3.1

6.3.1 is a bugfix release of FlatSharp that resolves a "unreachable code detected" warning message.

What's Changed

• Update samples to 6.3.0 by @jamescourtney in #291
• Fix 'code unreachable' warning by @jamescourtney in #292

Full Changelog: 6.3.0...6.3.1

6.3.0

6.3.0 addresses two issues and includes a very minor breaking change (necessary to fix one of the issues):

• A security issue enabling a denial of service attack affecting schemas that are too deep or have cycles when accepting input from an untrusted source. A canonical example is a Linked List:

table LinkedListNode
{
    Next : LinkedListNode;
    Value : int;
}

In this example, an attacker could send a LinkedList that was long enough to trigger a stack overflow, which would crash the server process. FlatSharp 6.3.0 adds depth tracking when deserializing "deep" schemas (currently defined as cyclical or over 500 items deep). Depth tracking does introduce a performance penalty, but only to those schemas. FlatSharp will throw a System.IO.InvalidDataException at depths over 1000 items, though this is configurable via ISerializer<T>.WithSettings. Cycle-free schemas of reasonable depths will not have depth tracking enabled.

• Fixes a bug where Sorted and Indexed Vectors could not be part of a cyclical relationship:

table A
{
    Item : B;
    Key : string (key);
}

table B
{
   Items : [A] (fs_vector:"IIndexedVector");
}

• Finally, 6.3.0 does introduce a minor breaking change to the IInputBuffer interface, as the DepthLimit parameter needs to be tunneled through.

What's Changed

• Add Object Depth Limit by @jamescourtney in #289
• Fix circular initialization/validation issues. by @jamescourtney in #290

Full Changelog: 6.2.1...6.3.0

6.2.1

6.2.1 is a bugfix release of FlatSharp that contains two fixes:

1. Addressed an issue where FBS includes in the FlatSharp compiler would not be resolved if the path ended with a \. Thanks to @shadowbane1000 for opening the issue and @yak-shaver for providing a fix!
2. Support for declaring NaN, PositiveInfinity, and NegativeInfinity as default values on doubles in FBS files. Thanks to @mattico for identifying and fixing this!

What's Changed

• Update samples to 6.2.0 by @jamescourtney in #281
• Fixed issue with trailing backslash in the IncludePath by @yak-shaver in #283
• Support nan/inf default values by @mattico in #284
• Update to 6.2.1 by @jamescourtney in #285

New Contributors

• @mattico made their first contribution in #284

Full Changelog: 6.2.0...6.2.1

6.2.0

Summary

6.2.0 is a small feature release of FlatSharp that introduces a couple of quality-of-life features:

Field Name Normalization

The standard naming scheme in FBS files is to use snake_case for fields. FlatSharp has traditionally not modified the field names. However, 6.2.0 adds an optional switch to normalize snake_case to UpperPascalCase to fit with C# conventions. Note that this is an opt-in switch:

  <PropertyGroup>
    <FlatSharpNameNormalization>true</FlatSharpNameNormalization>
  </PropertyGroup>

XML Documentation Comments

The FlatSharp Compiler now injects any documentation comments from FBS files into your generated code:

/// This is my table!
table MyTable
{
    /// This is a property!
    MyProperty : int32;
}

/// <summary>
/// This is my table!
/// </summary>
[FlatBufferTable]
public class MyTable
{
    /// <summary>
    /// This is a property!
    /// </summary>
    [FlatBufferItem(0)]
    public virtual int MyProperty { get; set; }
}

XML Summary comments are applied on:

• Enums and enum members
• Tables and table members
• Structs and struct members (including struct vectors)
• RPC services and method calls
• Unions, but not union members

What's Changed

• More global namespaces by @jamescourtney in #278
• Field Names Normalization by @jamescourtney in #279
• XML Documentation by @jamescourtney in #280

Full Changelog: 6.1.1...6.2.0

6.1.1

FlatSharp 6.1.1 is a bugfix release to address #274, which should improve compatibility with .NET Core based MSBuild implementations. Thanks, @yak-shaver!

What's Changed

• Updated Samples to demonstrate the new IncludePath feature by @yak-shaver in #271
• Use RoslynCodeTaskFactory when available (MSBuild 15.8 and newer) by @yak-shaver in #274

Full Changelog: 6.1.0...6.1.1

6.1.0

6.1.0 is a minor release of FlatSharp that introduces a few new features:

• The FlatSharp.Compiler package now supports specifying FBS include search paths. Thanks, @yak-shaver!
• Removed a branch from vtable lookups for small tables (<= 8 fields). This can be a big boost in cases where a table field might-or-might not be present. This will be most acutely observed with Lazy deserialization.
• Moved all classes that are considered "internal" to FlatSharp into the FlatSharp.Internal namespace. There is a slight possibility you will get a build break here, but only if you are doing something nonstandard.
• Added a new IInputBuffer2 interface, which extends IInputBuffer. IInputBuffer2 adds a couple of new methods for getting Span<byte> and Memory<byte> from a IInputBuffer. For some input buffers, this can allow write through to be twice as fast. IInputBuffer2 will be merged into IInputBuffer in next major release.
  • If you have custom IInputBuffer implementations, consider also supporting IInputBuffer2, though FlatSharp includes a safe fallback.

Internally, Flatsharp has been updated to use file-scoped namespaces and global usings.

What's Changed

• File scoped namespaces by @jamescourtney in #264
• VTable Optimizations by @jamescourtney in #265
• Update IInputBuffer by @jamescourtney in #267
• Move internal classes to FlatSharp.Internal and FlatSharp.CodeGen by @jamescourtney in #269
• Optimize vtable initialization by @jamescourtney in #270
• Adding IncludePath metadata for complex include hierarchies by @yak-shaver in #268
• Split sorted vector helpers into FlatSharp and FlatSharp.Internal by @jamescourtney in #272

New Contributors

• @yak-shaver made their first contribution in #268

Full Changelog: 6.0.5...6.1.0

6.0.5

Summary

The main change is that Binary Search has been given an overhaul. This affects all customers using BinarySearchByFlatBufferKey or IIndexedVector<TKey, TValue>. There are three main changes in this area from prior versions:

• Lambdas are no longer used when doing binary search. Instead, we use structs with interfaces and generic constraints. This avoids allocating closures, which cuts down on allocations during binary search in all cases. It also enables a modest performance win as the JIT can devirtualize all of the method calls.

• FlatSharp now includes an optimized binary search algorithm. This algorithm reduces allocations during binary search by 95%! The trick is that it avoids byte[] -> string -> byte[] conversions by using the underlying buffer directly. It is used when the following conditions are met:

  • The deserialization mode is non-greedy; that is it is Lazy or Progressive.
  • The Key is a string

• String comparison has been accelerated slightly by comparing ulongs instead of bytes when possible.

Combined, these changes show big wins compared to older versions:

Version	Method	Option	Length	Mean	Gen 0	Gen 1	Allocated
Old	ParseAndTraverse	Lazy	10000	8.921 ms	1328.1250	-	22,442,928 B
New	ParseAndTraverse	Lazy	10000	3.967 ms	39.0625	-	720,107 B
Old	ParseAndTraverse	GreedyMutable	10000	4.736 ms	296.8750	125.0000	5,040,144 B
New	ParseAndTraverse	GreedyMutable	10000	4.849 ms	78.1250	31.2500	1,360,147 B

There are a few other changes as well:

• The master branch has been renamed to main.

• FlatSharp should now compile and run on MacOS and Linux without any hair-pulling. Thanks @0xced!

• The invocation of flatc should now be more robust. Thanks @atifaziz!

What's Changed

• Give the samples an update by @jamescourtney in #256
• More robust spawning of flatc with output handling by @atifaziz in #257
• Treat warnings as errors by @jamescourtney in #259
• Fix ExperimentalBenchmark and FlatSharpEndToEndTests on Linux and macOS by @0xced in #258
• Rename Workflows Master -> Main by @jamescourtney in #260
• Binary Search Optimizations by @jamescourtney in #261

New Contributors

• @atifaziz made their first contribution in #257
• @0xced made their first contribution in #258

Full Changelog: 6.0.4...6.0.5

6.0.4

6.0.4 is a minor release of FlatSharp that contains a few enhancements.

• The .targets file in the FlatSharp.Compiler package now does a better job of detecting installed .NET SDKs. Previously, it tried to use the same SDK that you were using to build your project. Now it will detect the installed SDKs using dotnet --list-sdks and try to use the latest one.

• Optimized some array allocations for netstandard2.0 and netstandard2.1 builds.

• Added code to lock FlatSharp.Runtime to the same version as FlatSharp.Compiler. The generated code now includes a check to ensure that the assembly versions are the same between the two packages. Previously, it would have been theoretically possible to use FlatSharp.Runtime version N with FlatSharp.Compiler version N+1. While this is probably safe, FlatSharp isn't tested for mix-and-match compatibility, even within a major version, so the behavior is now to fail fast until the versions match.

• Exposed the Deserialization Option on the ISerializer and ISerializer<T> interfaces.

Changelog:

• Interface enhancements by @jamescourtney in #251
• Tweak polyfills and do a better job for netstandard 2.1 by @jamescourtney in #253
• Reduce string allocations for .NET Standard by using Array Pool by @jamescourtney in #254
• Lock FlatSharp versions by @jamescourtney in #255

Full Changelog: 6.0.2...6.0.4