Serialization with FlatBuffers in Java – 在 Java 中使用 FlatBuffers 进行序列化

最后修改: 2024年 3月 10日

中文/混合/英文(键盘快捷键：t)

1. Introduction

1.导言

In this tutorial, we’ll explore FlatBuffers in Java and perform serialization and deserialization using it.

在本教程中，我们将探索 Java 中的 FlatBuffers 并使用它执行序列化和反序列化。

2. Serialization in Java

2.Java 中的序列化

Serialization is the process of converting Java objects into a stream of bytes that can be transmitted over a network or persist in a file. Java provides an inbuilt object serialization mechanism through the java.io.Serializable interface and the java.io.ObjectOutputStream and java.io.ObjectInputStream classes.

序列化是将 Java 对象转换为字节流的过程，该字节流可通过网络传输或保存在文件中。Java 通过 java.io.Serializable 接口和 java.io.ObjectOutputStream 和 java.io.ObjectInputStream 类提供了内置的对象序列化机制。

However, owing to its several downsides, including a complicated approach to dealing with complex object graphs and dependent classes, several libraries are available for serialization and deserialization in Java.

然而，由于该方法有几个缺点，包括处理复杂对象图和依赖类的方法比较复杂，因此有几个库可用于 Java 中的序列化和反序列化。

Some of the widely used Java serialization libraries include Jackson and Gson. A newer standard for object serialization format is Protocol Buffers. Protocol Buffers is a language-agnostic binary serialization format developed by Google. They are used in high-performance environments and distributed systems where efficiency and interoperability are critical.

一些广泛使用的 Java 序列化库包括Jackson和Gson。一种较新的对象序列化格式标准是协议缓冲区。协议缓冲区是一种与语言无关的二进制序列化格式，由 Google 开发。它们用于对效率和互操作性要求极高的高性能环境和分布式系统中。

3. FlatBuffers

3.扁平缓冲区

FlatBuffers is an efficient cross-platform serialization library developed by Google. It supports several languages, such as C, C++, Java, Kotlin, and Go. FlatBuffers were created for game development; therefore, performance and low memory overheads are default considerations in its design.

FlatBuffers 是由 Google 开发的高效跨平台序列化库。它支持多种语言，如 C、C++、Java、Kotlin 和 Go。FlatBuffers 是为游戏开发而创建的；因此，性能和低内存开销是其设计的默认考虑因素。

FlatBuffers and Protocol Buffers are created by Google and are very similar binary-based data formats. Both of these formats support efficient high-speed serialization and deserialization. The primary difference is that FlatBuffers doesn’t need additional data unpacking to an intermediate data structure before access.

FlatBuffers 和 Protocol Buffers 由 Google 创建，是非常相似的基于二进制的数据格式。这两种格式都支持高效的高速序列化和反序列化。它们的主要区别在于，FlatBuffers 在访问前不需要将数据解压缩到中间数据结构中。

3.1. Introduction to the FlatBuffers Library

3.1.FlatBuffers 库简介

A complete FlatBuffers implementation consists of the following components:

一个完整的 FlatBuffers 实现由以下组件组成：

  • A FlatBuffer schema file
  • A flatc compiler
  • Serialization and deserialization code

The FlatBuffer schema file serves as a template for the structure of the data model we’ll use. The syntax for the schema file follows a similar pattern to that of C-type or other interface description language (IDL) formats. We need to define the schema and the flatc compiler, then compile the schema file.

FlatBuffer 模式文件是我们将要使用的数据模型结构的模板。模式文件的语法与 C 类型或其他接口 描述语言（IDL）格式的语法相似。我们需要定义模式和 flatc 编译器，然后编译模式文件。

3.2. Tables and Schemas

3.2.表格和模式

A FlatBuffer is a binary buffer that contains nested objects (such as structs, tables, and vectors) organized using offsets.

FlatBuffer 是一种二进制缓冲区，其中包含使用偏移量组织的嵌套对象（如结构体、表格和向量）。

This arrangement allows data to be traversed in place, similar to traditional pointer-based data structures. However, unlike many in-memory data structures, FlatBuffers strictly adhere to rules of alignment and endianness (always little), ensuring cross-platform compatibility. Moreover, for table objects, FlatBuffers offers both forward and backward compatibility.

这种安排允许就地遍历数据，类似于传统的基于指针的数据结构。不过，与许多内存数据结构不同的是，FlatBuffers 严格遵守对齐规则和字节序（始终为小），确保跨平台兼容性。此外，对于表对象，FlatBuffers 还具有向前和向后兼容性。

Tables in FlatBuffers are the most basic data structures used to represent complex structures with named fields. Tables are similar to classes or structs in some languages and support fields of several types, such as int, short, string, struct, vectors, and even other tables.

FlatBuffers中的表是最基本的数据结构，用于表示具有命名字段的复杂结构。表类似于某些语言中的类或结构体，支持多种类型的字段，如int、short、string、struct、vectors甚至其他表。

3.3. The flatc compiler

3.3.flatc 编译器

The flatc compiler is a crucial tool provided by FlatBuffers that generates code in various programming languages, such as  C++ and Java, to help serialize and deserialize data according to the schema. This compiler inputs the schema definition and generates code in the desired programming language.

flatc编译器是 FlatBuffers 提供的一个重要工具，可生成 C++ 和 Java 等各种编程语言的代码，帮助根据模式对数据进行序列化和反序列化。

In upcoming sections, we’ll compile our schema files to generate code. However, we need to build and set up our compiler first to be able to use it.

在接下来的章节中，我们将编译模式文件以生成代码。不过，我们需要先构建和设置编译器，才能使用它。

We start by cloning the flatbuffers library into our system:

我们首先将 flatbuffers 库克隆到系统中：

$ git clone https://github.com/google/flatbuffers.git

Once the flatbuffers directory is created, we use cmake to build the library into an executable. CMake (Cross-platform Make)  is an open-source, platform-independent build system designed to automate the process of building software projects:

创建 flatbuffers 目录后，我们将使用 cmake 将库构建为可执行文件。CMake（跨平台 Make）是一个开源、独立于平台的构建系统，旨在自动完成软件项目的构建过程：

$ cd flatbuffers
$ mkdir build
$ cd build
$ cmake ..

This completes the flatc compiler build process. We can verify the success of the installation by printing the version:

至此，flatc 编译器的编译过程完成。我们可以通过打印版本来验证安装是否成功：

$ ./flatc --version
flatc version 23.5.26

The compiled files are now stored under the /flatbuffers/build path, and the flatc executable is also available in the same directory. We’ll use this file to build all schema files, and therefore, we can create a shortcut or alias to this path.

编译文件现在存储在/flatbuffers/build路径下，flatc可执行文件也在同一目录下。我们将使用该文件来构建所有模式文件，因此，我们可以为该路径创建快捷方式或别名。

4. Working With FlatBuffers

4.使用 FlatBuffers

In this section, we’ll explore the FlatBuffers library by implementing our use case. Let’s consider that we are developing a game across different terrains such as the sea, mountain, and plain land. Each terrain has its own set of unique properties.

在本节中，我们将通过实现用例来探索 FlatBuffers 库。假设我们正在开发一款跨越不同地形（如海洋、山脉和平原）的游戏。每种地形都有自己独特的属性。

The terrain information is necessary to load the game level and needs to be transmitted across the network to the players. Efficient serialization and deserialization are a must.

地形信息是加载游戏关卡所必需的，并需要通过网络传输给玩家。高效的序列化和反序列化是必须的。

4.1. Schema Definition

4.1 计划定义

The first thing we should start with is defining our terrain schema type. A terrain is a table in our flatbuffer. It can have many attributes, such as a name (Sea, Land, Mountain, Desert, etc.), color, and position (in the form of 3d vector coordinates). The terrain can have an effect applied as well. For example, there might be a sandstorm in a desert or a flood in the land. The effect can be a separate table within the original schema.

我们首先应该定义地形模式类型。地形是flatbuffer中的一个表。它可以有许多属性，例如名称（海洋、陆地、山脉、沙漠等）、颜色和位置（以 3D 矢量坐标的形式）。地形还可以应用效果。例如，沙漠中可能会有沙尘暴，陆地上可能会有洪水。效果可以是原始模式中的一个单独表格。

With this understanding, let’s write our schema as follows:

有了这样的认识，我们就来编写如下模式：

namespace MyGame.baeldung;
enum Color:byte { Brown = 0, Red = 1, Green = 2, Blue = 3, White = 4 }
struct Vec3 {
  x:float;
  y:float;
  z:float;
}
table Terrain {
  pos:Vec3; // Struct.
  name:string;
  color:Color = Blue;
  navigation: string;
  effects: [Effect]
}

table Effect {
  name:string;
  damage:short;
}

root_type Terrain;

We have an enum for identifying the color of the terrain, a struct for the coordinates, and two tables, the Terrain and Effect, with Terrain being the root type.

我们有一个枚举用于识别地形的颜色，一个结构用于识别坐标，还有两个表，即地形表和效果表，其中地形表是根类型。

4.2. Schema Compilation

4.2.模式编译

The flatc compiler is ready, and we use it to compile our schema file terrain.fbs:

flatc编译器已准备就绪，我们可以用它来编译模式文件 terrain.fbs ：

$ cd <path to schema>
$ flatc --java terrain.fbs

We should note that the flatc path might vary from system to system depending on the installation location described in the previous section.

我们需要注意的是，flatc 路径可能因系统而异，这取决于上一节所述的安装位置。

4.3. Creating Objects and Perform Serialization

4.3.创建对象并执行序列化

The schema has already been compiled and is ready to go. We can start creating some terrains for our game using the schema. As part of this example walkthrough, we’ll create a desert terrain and a few effects for our terrain.

模式已经编译完成，随时可以使用。我们可以开始使用模式为我们的游戏创建一些地形。作为本示例演示的一部分，我们将创建一个沙漠地形，并为我们的地形创建一些特效。

To use FlatBuffers in Java, we need to add a Maven dependency:

要在 Java 中使用 FlatBuffers，我们需要添加 Maven 依赖项：

<dependency>
    <groupId>com.google.flatbuffers</groupId>
    <artifactId>flatbuffers-java</artifactId>
    <version>23.5.26</version>
</dependency>

We can now import the flatbuffers library along with the generated files from our schema:

现在，我们可以导入 flatbuffers 库以及从模式中生成的文件：

import MyGame.terrains.*;
import com.google.flatbuffers.FlatBufferBuilder;

The files generated as part of the compilation process go under the same path defined in the schema’s namespace section (MyGame in our case).

编译过程中生成的文件会进入模式的 namespace 部分（在我们的例子中为 MyGame）中定义的相同路径。

An Effect class is available for us to use as a result of the compilation, which provides a createEffect() method. We’ll use that to create our desired effect. We’ll start by creating a builder object with an initial buffer size of 1024 bytes:

通过编译，我们可以使用一个 Effect 类，该类提供了一个 createEffect() 方法。 我们将首先创建一个初始缓冲区大小为 1024 字节的构建器对象：

FlatBufferBuilder builder = new FlatBufferBuilder(INITIAL_BUFFER);

int sandstormOffset = builder.createString("sandstorm");
short damage = 3;
int sandStorm = MyGame.terrains.Effect.createEffect(builder, sandstormOffset, damage);

We can add more effects in the same way.

我们可以用同样的方法添加更多效果。

Next, we create our desert terrain. Let’s assign a color to the terrain, and give it a name and its navigation location:

接下来，我们创建沙漠地形。为地形指定颜色，并为其命名和导航位置：

byte color = MyGame.terrains.Color.YELLOW;
int terrainName = builder.createString("Desert");
int navigationName = builder.createString("south");

We add more terrain metadata and the effects using the auto-generated static methods of the Terrain class:

我们使用自动生成的 Terrain 类静态方法添加更多地形元数据和效果：

int effectOffset = MyGame.terrains.Terrain.createEffectsVector(builder, effects);

startTerrain(builder);
addName(builder, terrainName);
addColor(builder, color);
addNavigation(builder, navigationName);
addEffects(builder, effectOffset);
int desert = endTerrain(builder);
builder.finish(desert);

Let’s now serialize our terrain and its effects in our flatbuffer. We can store the buffer or transmit it over the network to clients:

现在，让我们在 flatbuffer 中序列化我们的地形及其效果。我们可以存储缓冲区或通过网络将其传输给客户端：

ByteBuffer buf = builder.dataBuffer();

4.4. Deserialisation Using FlatBuffers

4.4.使用 FlatBuffers 反序列化

Let’s deserialize the flatbuffer object and access the terrain. We’ll start with a serialized array of bytes created from the buffer, and we’ll convert it into a ByteBuffer buffer:

让我们反序列化 flatbuffer 对象并访问地形。我们将从从缓冲区创建的字节序列化数组开始，并将其转换为 ByteBuffer 缓冲区：

ByteBuffer buf = java.nio.ByteBuffer.wrap(buffer);

This allows us to get an accessor to the root Terrain object from the buffer and access all its attributes:

这样，我们就可以从缓冲区中获取根 Terrain 对象的访问器，并访问其所有属性：

Terrain myTerrain = Terrain.getRootAsTerrain(buf)
Assert.assertEquals(terrain.name(), "Desert");
Assert.assertEquals(terrain.navigation(), "south");

The compiler-generated code shows that each of the entity’s attributes comes with an associated accessor. We can access the associated effects as well:

编译器生成的代码显示，实体的每个属性都有一个相关的访问器。我们还可以访问相关的效果：

Effect effect1 = terrain.effectsVector().get(0);
Effect effect2 = terrain.effectsVector().get(2);
Assert.assertEquals(effect1.name(), "Sandstorm");
Assert.assertEquals(effect2.name(), "Drought");

4.5. Mutating FlatBuffers

4.5.变异 FlatBuffers

FlatBuffers are mostly read-only, owing to their static template structure. However, we might face a scenario where we need to change something in a flatbuffer before sending it to another piece of code. Let’s say we want to update the damage score of a sandstorm effect from the existing value of 3 to 10.

由于 FlatBuffer 采用静态模板结构，因此大部分情况下都是只读的。然而，我们可能会遇到这样一种情况，即我们需要先更改 flatbuffer 中的内容，然后再将其发送给另一段代码。比方说，我们希望将沙暴效果的伤害值从现有的 3 更新为 10。

In such cases, in-place mutation of flatbuffers comes in handy.

在这种情况下，扁平缓冲区的就地突变就会派上用场。

Mutation of a flatbuffer is only possible if we build the schema with a –gen-mutable argument:

只有当我们使用 -gen-mutable 参数构建模式时，flatbuffer 的突变才有可能发生：

$ ./../flatc --gen-mutable --java terrain.fbs

This provides us with a mutate() method on all the accessors, which we can use to modify the value of a flatbuffer in place:

这为我们提供了所有访问器上的 mutate() 方法，我们可以使用该方法就地修改 flatbuffer 的值：

Assert.assertEquals(effect1.damage(), 3);
effect1.mutateDamage((short) 10);
Assert.assertEquals(effect1.damage(), 10);

5. JSON Conversion Using FlatBuffers

5.使用 FlatBuffers 进行 JSON 转换

The flatc compiler provides techniques to convert binary files to JSON and vice-versa. Let’s say we have a JSON file for our terrain. We can use the compiler to create a binary file out of the JSON file using the following code:

flatc 编译器提供了将二进制文件转换为 JSON 文件的技术，反之亦然。比方说，我们的地形有一个 JSON 文件。我们可以使用编译器从 JSON 文件创建二进制文件，代码如下：

flatc --binary <template file> <json file>

$ flatc --binary terrain.fbs sample_terrain.json

Conversely, we can convert a binary file to a full-fledged JSON file as well:

反之，我们也可以将二进制文件转换为完整的 JSON 文件：

flatc --json --raw-binary <template file> -- <binary file>

$ flatc --json --raw-binary terrain.fbs -- sample_terrain.bin

6. Benefits of Using FlatBuffers

6.使用 FlatBuffers 的好处

The usage of this cross-platform serialization library comes with a plethora of benefits:

使用这种跨平台序列化库有很多好处：

  • FlatBuffers organizes hierarchical data in a flat binary buffer, which we can directly access without the overhead of parsing or unpacking
  • Incremental changes to our data structure are automatically and cleanly incorporated, making it easy to maintain backward compatibility with our evolving models
  • They are also efficient in terms of memory utilization, as we only need the memory space occupied by the buffer to access your data
  • They leave a tiny code footprint. The generated code is minimal, and we only need a single small header as a dependency, making integration a breeze
  • FlatBuffers are strongly typed; hence, we can catch errors in compile time

7. Conclusion

7.结论

In this article, we explored the FlatBuffers library and its capabilities to serialize and deserialize complex data. We took a hands-on approach to implementing code using the library and looked at the benefits and use cases of flatbuffers.

在本文中，我们探讨了 FlatBuffers 库及其序列化和反序列化复杂数据的功能。我们采用动手实践的方法使用该库实现代码，并探讨了 flatbuffers 的优势和使用案例。

As usual, the code is available over on GitHub.

与往常一样，代码可在 GitHub 上获取。