Sardine named a Leader in The Forrester Wave™: Financial Crime Management Solutions, Q3 2026

Learn More

Open sourcing protobuf-to-bigquery converter

Kazuki Nishiura
Kazuki Nishiura
bg-image
bg-image
Open sourcing protobuf-to-bigquery converter
Subscribe to newsletter
Share

Key takeaways

  • Protobuf can serve as a shared schema. A single protobuf definition can be used to generate application code and describe structured data moving through a data pipeline.
  • Dynamic field mapping reduces boilerplate. Reading protobuf field descriptors programmatically avoids maintaining a separate hard-coded mapping for every field.
  • Schema changes still require pipeline management. Automating field mapping doesn't remove the need to update and redeploy a Dataflow job when the underlying protobuf definition changes.
  • The right approach depends on the pipeline. Google's current Dataflow templates provide a managed option for common Pub/Sub-to-BigQuery workflows, while custom code can still make sense when a pipeline requires additional control or application-specific behavior.
  • Data infrastructure benefits from shared definitions. Keeping application and analytics schemas aligned can reduce the amount of translation code required between services and data systems.

For another example of Sardine's approach to data engineering infrastructure, see our work on the data-engineering-led fraud and compliance platform.

At Sardine we use protocol buffers (protobuf) extensively to define data schemas and move structured data through our backend and analytics pipeline. It is a lightweight and language-neutral serialization format with typing.

We used the same protobuf definition to define the schema for BigQuery tables and also generate backend server (go) code. Our backend services populated the Golang struct (generated from protobuf), sent it to pubsub, and then used dataflow jobs to write that data to BigQuery.

Image

This simple pattern worked pretty well with one caveat. Converting a pubsub object into a BigQuery row initially required a lot of repetitive code.

The challenge of mapping protobuf data to BigQuery

When we started, there wasn’t a straightforward way to convert a pubsub object into a BigQuery row insert. We had to manually copy fields from one object to another.

TableRow clientMetadataRow = new TableRow() .set("session_key", clientMetadata.getSessionKey()) .set("client_id", clientMetadata.getClientId()) .set("revision", clientMetadata.getRevision()) .set("user_id", clientMetadata.getUserId())...

This was very tedious and error prone. When we added a new field in BigQuery table we needed to update the above boilerplate code too.

For a data pipeline that depends on a shared schema, maintaining this kind of boilerplate creates unnecessary work. We wanted the protobuf schema to do more of the work for us.

Dynamically mapping protobuf fields to BigQuery

It turns out that you can dynamically iterate protobuf object so instead of hard coding field names, we could do:

void copyFields(GeneratedMessageV3 fromProto, TableRow toRow) { allFields = fromProto.getDescriptorForType().getFields(); for (Descriptors.FieldDescriptor field: allFields) { Object value = fields.get(field); String columnName = field.getName(); switch (field.getJavaType()) { case STRING: if (value != null) { toRow.set(columnName, value); } break;...

That’s it! It reduced our boilerplate code and now we don’t need to worry about updating Java code (note you still need to redeploy dataflow job for any schema change so job runner knows about the latest proto definition)

For more background on how Sardine has used Dataflow and BigQuery in its data infrastructure, see our engineering deep dive on building a feature store with Golang and Dataflow.

We made the code public in below repo so the community don’t have to reinvent the wheel. The proto-to-bq-java repository contains the Java implementation for converting protobuf objects into BigQuery tablerow objects. The repository remains public today.

If you’re working with a custom protobuf-based pipeline, the code provides a starting point for dynamically mapping protobuf fields to BigQuery columns.

The Google Cloud ecosystem has evolved since we originally built this solution. Google now provides official pubsub Proto to BigQuery Dataflow templates for streaming protobuf data from a pubsub subscription into a BigQuery table. The template requires a protobuf schema and matching BigQuery destination schema, and can support transformation through a user-defined function.

That means the Sardine converter is best viewed as an open-source example for teams that need custom Java mapping logic, rather than as the only way to move protobuf data into BigQuery.

If you have any feedback, please reach out to me https://twitter.com/kazukinishiura

We’re hiring across all engineering roles, https://www.sardine.ai/careers

Frequently asked questions

What is a Protobuf to BigQuery converter?

A Protobuf to BigQuery converter maps fields from a protocol buffer object to columns in a BigQuery row. Instead of manually writing code for every field, a converter can use the protobuf schema to determine the fields and their corresponding values.

Why use protobuf with BigQuery?

Protobuf provides a typed, language-neutral way to define structured data. In a data pipeline, the same protobuf definition can be used across backend services and downstream systems, helping keep the structure of the data consistent as it moves through the pipeline.

How does a Protobuf to BigQuery converter reduce boilerplate?

A manual approach requires developers to write and maintain code that copies each protobuf field into a BigQuery row. A dynamic converter can iterate through the protobuf fields and map them programmatically, reducing the amount of field-specific code that needs to be maintained.

How do protobuf, Pub/Sub, Dataflow, and BigQuery work together?

A backend service can serialize structured data as a protobuf message and publish it to Pub/Sub. A Dataflow pipeline can consume those messages, process them, and write the resulting records to BigQuery. Google also provides a current Pub/Sub Proto to BigQuery Dataflow template for this type of streaming workflow.

What happens when the protobuf schema changes?

A converter can reduce the amount of manual field-mapping code that needs to change, but schema changes still need to be managed across the pipeline. In the Sardine implementation described here, the Dataflow job needed to be redeployed so it could use the updated protobuf definition.

Try the open-source converter

The Sardine protobuf-to-BigQuery converter is available as an open-source Java project. If you're working with protobuf, Pub/Sub, BigQuery, and Dataflow, you can use it as a reference for dynamically mapping protobuf fields into BigQuery rows.