ProtoBuff Data Types Schema Parser

A schema parser for Protocol Buffers data types. It plugs into @asyncapi/parser and lets you embed .proto message definitions directly in the payload (or schema) of an AsyncAPI document. During parsing, the Protobuf definition is converted into an AsyncAPI (JSON Schema based) schema so the rest of the AsyncAPI tooling can work with it like any other message payload.

Overview

AsyncAPI messages can carry payloads described in different schema languages. @asyncapi/parser delegates any payload whose schemaFormat it does not understand natively to a registered schema parser. This package is the schema parser for Protocol Buffers: you register it once, and every message payload tagged with a Protobuf schemaFormat is parsed and expanded in place into an equivalent AsyncAPI schema.

• Works with Protobuf 2 and Protobuf 3 schemas.
• Registers for the schema formats application/vnd.google.protobuf;version=2 and application/vnd.google.protobuf;version=3.
• Maps Protobuf messages, scalars, enums, repeated fields, oneof and nested types to JSON Schema.
• Understands leading comments as descriptions and a set of @-annotations for validation, defaults and examples.
• Optionally maps protoc-gen-validate and protovalidate rules to JSON Schema validators.
• Browser-compatible and ships both CommonJS and ES module builds.

There is no strict distinction between Protobuf 2 and 3. Declaring schemaFormat as application/vnd.google.protobuf;version=2 while providing a proto3 schema (or vice versa) does not by itself cause an error.

Version >= 2.0.0 of this package requires @asyncapi/parser >= 2.0.0; the 3.x line targets @asyncapi/parser >= 3.6.0.

• Installation
• Usage
• Supported input
• How it works
• Scalar type formats
• Comments and annotations
  • Per field annotation
  • Per message annotation
  • Head annotation
  • Head annotation "Option"
• Supported validation frameworks
• Development
• Contributing
• License

Installation

npm install @asyncapi/protobuf-schema-parser
# or
yarn add @asyncapi/protobuf-schema-parser

@asyncapi/parser is required to use this package. Requires Node.js >= 18.

Usage

Create a Parser, register the Protobuf schema parser on it, then parse an AsyncAPI document that uses a Protobuf schemaFormat. The Protobuf source goes into the message payload as a string.

import { Parser } from '@asyncapi/parser';
import { ProtoBuffSchemaParser } from '@asyncapi/protobuf-schema-parser';

const parser = new Parser();
parser.registerSchemaParser(ProtoBuffSchemaParser());

const asyncapiWithProto = `
asyncapi: 2.0.0
info:
  title: Example with ProtoBuff
  version: 0.1.0
channels:
  example:
    publish:
      message:
        schemaFormat: 'application/vnd.google.protobuf;version=3'
        payload: |
          message Point {
            required int32 x = 1;
            required int32 y = 2;
            optional string label = 3;
          }

          message Line {
            required Point start = 1;
            required Point end = 2;
            optional string label = 3;
          }
`;

const { document, diagnostics } = await parser.parse(asyncapiWithProto);

The same works with CommonJS:

const { Parser } = require('@asyncapi/parser');
const { ProtoBuffSchemaParser } = require('@asyncapi/protobuf-schema-parser');

const parser = new Parser();
parser.registerSchemaParser(ProtoBuffSchemaParser());

Notes:

• ProtoBuffSchemaParser is exported both as a named export and as the default export. Call it to obtain the parser instance, then pass that instance to registerSchemaParser.
• Place your Protobuf schema as a string in the message payload (AsyncAPI 2.x) or in the schema object's schema keyword (AsyncAPI 3.x) to get it parsed. See the fixtures under test/documents for both variants.
• After parsing, the message payload in document is the converted AsyncAPI schema.

Supported input

• Protobuf 2 and Protobuf 3 message definitions.
• The two registered schema formats:
  • application/vnd.google.protobuf;version=2
  • application/vnd.google.protobuf;version=3

References are not supported:

• No support for $ref inside the Protobuf payload.
• No support for import, with the exception of the bundled well-known definitions:
  • google/protobuf/* (provided by protobufjs)
  • google/type/* (bundled with this package)
  • the validation definitions validate/validate.proto and buf/validate/validate.proto

Any other import throws an error during parsing.

How it works

Each AsyncAPI document may contain several message payloads. For every payload whose schemaFormat matches one of the registered MIME types, the parser hands the raw Protobuf string to the converter, which uses protobufjs to parse it and then walks the message tree, emitting a JSON Schema based AsyncAPI schema.

Because a JSON Schema has a single root, the converter first determines the root message: the message that is not referenced as a field type by any other message. If several candidates remain, annotate the intended root with @RootNode; otherwise parsing fails with Found more than one root proto messages. If no root can be found, parsing fails with Not found a root proto messages.

The main mapping rules:

A field is in the required list if it is annotated with @Required, is a proto2 required field, or carries the required rule of a validation framework. A proto3 field is never required by itself, optional or not: proto3 has no required label, and a field without explicit presence is omitted from the wire and from ProtoJSON when it holds its default value. A required label in a proto3 file is not valid proto3; the parser ignores it and logs a warning naming the field.

Recursive messages are supported: when the same message type is encountered twice on the current branch, the converter stops descending to avoid an infinite loop.

Non-standard (x-) keywords are used to preserve Protobuf information that has no direct JSON Schema equivalent: x-primitive (original scalar type), x-type (referenced message/enum name), x-type-description (comment of the referenced message when a field comment replaced it), x-enum-mapping (enum name to numeric value), and x-oneof-item (the oneof field a variant came from).

Scalar type formats

Each Protobuf scalar type is mapped to a JSON Schema type plus a format that preserves the original Protobuf wire type. For example int64 becomes {"type": "integer", "format": "int64"} and bytes becomes {"type": "string", "format": "bytes"}. The Protobuf type is additionally kept in the non-standard x-primitive keyword. By default numeric types also carry minimum/maximum limits (see the primitiveTypesWithLimits option).

Comments and annotations

Each field of a message may have a comment which is reflected as the JSON Schema description. Furthermore, the comment can contain the following annotations:

message Point {
    /*
     * The coordinate on the x axis.
     * @Default 99
     * @Min 0
     * @Max 100
     */
    required int32 x = 1;

    /*
     * The coordinate on the y axis.
     * @Default 12
     * @Min 0
     * @Max 100
     */
    required int32 y = 2;
    optional string label = 3;
}

Per field annotation

Per message annotation

Head annotation

Head annotation "Option"

The @Option has to be followed by a space-separated option key and a space-separated value.

// @Option primitiveTypesWithLimits false

message Point {

}

Possible options are:

Supported validation frameworks

If you would like to add additional validation to your proto files, you can use one of the following validation frameworks. Their rules are translated into JSON Schema validation keywords during parsing.

• protoc-gen-validate — validation of lists is not 100% supported.
• protovalidate

The required rule of either framework adds the field to the JSON Schema required list: (buf.validate.field).required = true, or (validate.rules).<type>.required = true for the types message, duration, timestamp and any.

Development

This project is written in TypeScript and builds both an ES module (esm/) and a CommonJS (cjs/) output. Tests use Jest against snapshot-style fixtures in test/documents.

npm install          # install dependencies

npm run build        # build both esm/ and cjs/ (build:esm, build:cjs)
npm test             # run the Jest test suite with coverage
npm run lint         # run ESLint (use lint:fix to auto-fix)

Source layout:

Contributing

Read CONTRIBUTING to learn about the contribution process and the AsyncAPI Code of Conduct. Issues and pull requests are welcome.

License

Apache-2.0