AMQP Message metadata

Ongoing discussions within the SWIM Community of Interest

AMQP Message metadata

The following material describes different initiatives, at different maturity levels, that may help determine if (and, in case, which kind of) guidelines and best practices for AMQP 1.0 message properties definition may be needed in the ATM SWIM environment. They refer to different “domains” such as network management, meteorological data, and data link services but they all introduce AMQP metadata (i.e. properties) for multiple purposes (e.g. facilitate filtering, enable routing, tracing and troubleshooting etc..).

As they have been developed by different groups, they present some differences both regarding naming “style” and, in some cases, approaches on how to use the same properties. Therefore, this may suggest the need for guidelines providing some level of harmonization in the area of:

  • Consistent definition and usage of AMQP message properties (standard and application-specific)

  • Efficient filtering, routing, and processing of messages

  • Traceability with regulatory and operational requirement

 

NM B2B API Design Conventions 3.0

Status: Mature Draft

Context: Developed as part of the evolution of the Network Manager (NM) B2B services, aligning with the SWIM Technical Infrastructure (TI) Yellow Profile and the CP1 regulation. It aims to harmonize and future-proof NM B2B APIs, ensuring compliance, usability, and interoperability for both synchronous (HTTP/REST) and asynchronous (AMQP 1.0) exchanges.

Source: https://eurocontrol.sharepoint.com/:w:/r/sites/coll-SFG/Shared%20Documents/General/AMQP%20Properties/NM%20B2B%20API%20Design%20Conventions%203.0.docx?d=w73aa64c4b9f94300adc98961ff0b8029&csf=1&web=1&e=tvtJ2F

The approach taken in the NM B2B API Design Conventions is based on the need for harmonization, future-proofing, and regulatory compliance for Network Manager (NM) B2B services. The document addresses both synchronous (HTTP/REST) and asynchronous (AMQP) patterns and introduces a well-defined approach for the development of future B2B services in this context.

The conventions defined in the document establish a clear separation between different types of message properties— header, message, and application properties— each serving a distinct purpose. They expect the use of machine-processable definitions (OpenAPI for HTTP, AsyncAPI for AMQP) ensuring that both human and machine consumers can reliably interpret service contracts.

A key decision is to use structured and semantically meaningful values for properties like subject and application-specific fields, enabling efficient filtering, routing, and traceability. The conventions also provide for extensibility, allowing new business needs and technologies to be integrated without disrupting existing clients.

For AMQP 1.0, the conventions specify a layered property model:

  • Header properties (e.g., durable, ttl, delivery-count) are set by the messaging infrastructure and control delivery semantics.

  • Message properties (e.g., subject, content-type, absolute-expiry-time, correlation-id) are explicitly defined for each message type. The subject property, in particular, is used to encode structured routing and business context, such as flight-management/FLIGHT_DATA/FSA, enabling both topic-based routing and fine-grained filtering.

  • Application properties are reserved for business-level metadata and operational context. These include standardized fields such as NM_BUSINESS_ID, NM_TYPE, and NM_SERVICE_VERSION, as well as topic-specific fields (e.g., NM_CONCERNED_UNITS_ANU). These properties are important for supporting operational filtering, traceability, and integration with external systems.

 

EUROCONTROL Specification for Data Link Ground Distribution SWIM Services

Status: Published Specification

Context: This is a formal EUROCONTROL specification for SWIM services supporting the ground distribution of ADS-C and Logon Information, for initial trajectory information sharing in Europe. It defines requirements for the ground distribution of data link information, specifically ADS-C and DLIC (Datalink Initiation Capability) data, within a System Wide Information Management (SWIM) context. It enables the sharing of air-ground data through SWIM services, optimizing the use of air-ground and airborne resources. It details requirements for the establishment and operational deployment of these services by ATM stakeholders.

Source: https://www.eurocontrol.int/sites/default/files/2023-12/eurocontrol-specification-data-link-ground-distribution-swim-services.pdf

The EUROCONTROL Specification for Data Link Ground Distribution SWIM Services provides (among other aspects) a prescriptive, requirements-driven framework for the use of AMQP 1.0 message properties in the context of ground distribution of ADS-C and Logon Information. The specification mandates AMQP 1.0 for all asynchronous messaging and defines, for each interface, the exact set of required and optional message properties.

Header and message properties such as durable, priority, message-id, subject, content-type, and creation-time are specified with operational semantics. For example, the priority property is used to differentiate between routine and urgent messages, while durable is set to ensure message persistence as required by the operational context.

Application properties are central to the specification’s approach to filtering and routing. Properties such as aircraftAddress, flightId, departureAirport, destinationAirport, eobd, eobt, and adsVersion are defined as mandatory for certain message types, enabling consumers to use AMQP selectors to subscribe to messages matching specific operational criteria. The specification also defines the use of ISO 8601 formats for temporal properties.

The subject property is used to distinguish between business and technical messages, with values such as flightStatus, AdsDemandCnf, or serviceStatus. This enables both operational and administrative workflows to be handled in a unified, machine-interpretable way.

Payloads are strictly schema-driven, with all business messages validated against ASN.1-derived XML schemas. Error handling is standardized, with explicit conventions for status codes, reason phrases, and error details, ensuring interoperability and ease of integration with client systems.

 

MET SWIM AMQP guidance material

Status: Early Draft

Context: It is a draft guidance document from the EUROCONTROL MET3SG Task Team, focused on standardizing AMQP 1.0 message properties for meteorological (MET) data exchange in the context of MET-SWIM Services (CP1).

Source: https://github.com/iblsoft/swimdemo/blob/main/MET-SWIM-AMQP-Guidance.md

The MET-SWIM-AMQP-Guidance is a detailed, community-driven (early) draft that aims to standardize the use of AMQP 1.0 message properties for meteorological data exchange in the European MET-SWIM context. The guidance defines a comprehensive message structure, including addressing, header, standard, and application properties, with a focus on supporting both operational efficiency and interoperability.

Message addressing uses a simplified three-level hierarchy (e.g., weather.aviation.metar), inspired by the WIS 2 Topic Hierarchy but tailored for aviation MET data. This structure supports both precise and wildcard-based subscriptions, with an on-going discussion regarding the homogenous behavior of wildcards among different broker implementations (e.g., ActiveMQ Artemis, RabbitMQ, Qpid Proton).

Header and standard properties are mandated with operationally meaningful values. For example, priority is proposed to be set according to the operational importance of the message type (e.g., SIGMET and SPECI at priority 7, METAR at 4), and absolute-expiry-time is set based on the expected operational lifetime of the data (e.g., METAR/SPECI: +3h, TAF: +12h, SIGMET: +24h).

Application properties are categorized as mandatory, conditional, or optional:

  • Mandatory: topic, properties.icao_location_identifier, properties.icao_location_type, properties.pubtime.

  • Conditional: Temporal properties (properties.datetime, properties.start_datetime, properties.end_datetime), external links (links.count, links[0].href, links[0].rel, links[0].type), integrity checks (properties.integrity.method, properties.integrity.value), and geometry for spatial filtering (geometry.type, geometry.coordinates.latitude, etc.).

  • Optional: Additional links, payload integrity, geometry properties for advanced spatial queries.

The guidance provides detailed mapping and extraction rules for converting IWXXM XML data into AMQP properties, including XPath expressions for each property and handling of XML namespaces. It also discusses the use of cryptographic hashes for payload integrity, with recommendations for supported algorithms (e.g., SHA-512, SHA-256).

Filtering is specific concern: application properties are designed for server-side filtering using AMQP filter expressions, supporting operational use cases such as filtering by location, time, or report type. The guidance aligns with international standards (WMO WIS2, ICAO Annex 3). Nevertheless, discussions are on-going on property naming, optionality, and alignment with other SWIM documentation.

 

Summary Table: Key Features Across Initiatives

Initiative / Document

Scope

AMQP Property Focus

Filtering

Compliance

Initiative / Document

Scope

AMQP Property Focus

Filtering

Compliance

NM B2B API Design Conventions

Future NM B2B Services

Full property set (header, message, application); structured subject; topic-specific application properties

Yes (but expected to be requested through the “subscription” service)

SWIM-TI Yellow Profile, CP1

EUROCONTROL Data Link Ground Distribution

ADS-C, Logon SWIM Services

Strict property requirements per interface; business/technical message distinction

Yes (selectors, application properties)

SWIM-TI Yellow Profile, CP1

MET-SWIM-AMQP-Guidance

MET-SWIM (CP1)

Detailed mapping of standard and application properties; spatial/temporal properties; integrity

Yes (selectors, spatial/temporal)

SWIM-TI Yellow Profile, CP1, ICAO Annex 3

Use of Standard AMQP 1.0 Properties across the documents

Commonalities

  • All three initiatives require the use of core AMQP 1.0 standard properties such as subject, content-type, content-encoding, absolute-expiry-time, and priority for operational messages.

  • All use application properties for business/operational metadata and filtering.

Differences in Approach and Semantics

  • subject property:

    • NM B2B: Used as a structured string encoding business context (e.g., flight-management/FLIGHT_DATA/FSA). The structure is flexible and can be adapted per API or topic

    • Data Link Spec: Used to distinguish message types (e.g., flightStatus, AdsDemandCnf). The value is more type-oriented and less hierarchical

    • MET-SWIM: Mandates a specific, pattern-based structure for subject (e.g., DATA_TAF_LOWS_NORMAL_20250401051500), encoding report type, location, status, and issue time. This is more rigid and operationally focused. Note: This approach is currently under revision (it will likely rather host the “topic” name as foreseen in NM B2B but under the form weather.aviation.metar).

  • content-type property:

    • The approach and semantic is very well aligned. As it is normal, the expected values may differ depending on the expected content (e.g. application/xml for XML payloads, application/json for JSON payloads or other types for certain messages, depending on the payload and service).

  • priority property:

    • MET-SWIM: Explicitly assigns operational priorities (e.g., SIGMET/SPECI = 7, METAR = 4) and provides rationale for each

    • Data Link Spec: Uses default priorities, but less operational differentiation.

    • NM B2B: Priority is available but not as operationally prescribed as in MET-SWIM.

  • absolute-expiry-time property:

    • MET-SWIM: Specifies expiry times based on message type (e.g., METAR/SPECI: +3h, TAF: +12h, SIGMET: +24h)

    • Others: May set expiry, but not as operationally differentiated.

    • In general, they all link the value of this property to the value of ttl

  • Application properties:

    • All use application properties for filtering, but the naming, structure, and required fields differ (see below).

 

Naming Conventions for Properties

Commonalities

  • All use lowerCamelCase or snake_case for application property names, but with domain-specific variations

Differences

  • NM B2B: Uses uppercase with underscores for NM-specific properties (e.g., NM_BUSINESS_ID, NM_TYPE).

  • Data Link Spec: Uses lowerCamelCase for most application properties (e.g., aircraftAddress, flightId, departureAirport), but also includes some properties starting with uppercase (e.g. Delivery, Timeout)

  • MET-SWIM: Uses a mix of dot notation and snake_case for hierarchical properties (e.g., properties.icao_location_identifier, properties.pubtime, geometry.type), and sometimes prefixes with properties. or links. for grouping.

 

Topic/Addressing Structure and Symbols

Commonalities

  • All initiatives use some form of topic or address structure to organize message routing and filtering

Differences

  • NM B2B: Encodes topic and event type in the subject property using slashes (e.g., flight-management/FLIGHT_DATA/FSA). Even if not explicitly mentioned, data publication over AMQP is linked to a previous subscription obtained through a dedicated Subscription interface (SOAP or REST based). It dynamically returns an AMQP endpoint address to the specific consumer where it can connect to consume the messages.

  • Data Link Spec: Uses subscription management interfaces REST endpoint-based addressing (e.g., /flightStatusSubscriptions/{subscriptionId}) dynamically returning an AMQP endpoint address (e.g.: /<address> ) to the specific consumer. There is no reference to “topic” concept in the Specification as it is implicit in the endpoint/subscription context. subject property can be used to discern the specific message (e.g. AdsDemandCnf , AdsEventCnf, AdsPeriodicCnf…) that is received through the AMQP Endpoint. Filtering may be done via application properties, not topic hierarchy.

  • MET-SWIM: Uses a dot-separated, hierarchical topic structure (e.g., weather.aviation.metar). Wildcards (*, #) are supported for subscriptions, but their interpretation may depend on the broker (e.g., ActiveMQ Artemis, RabbitMQ, Qpid Proton - Note: this is under verification). The topic structure is explicitly mapped to operational domains and report types.

Note: AMQP 1.0 Standard does not include references to the concept of “topic”. <So, it might be useful to clarify if/how it is possible to use AMQP - in interoperable ways - in relation to “topic” based subscriptions>

Semantics and Usage Patterns

Commonalities

  • All initiatives use properties for both routing and filtering, and to encode operational context.

Differences

  • NM B2B: The semantics of properties are flexible and can be adapted per API or topic. The focus is on extensibility and model-driven design.

  • Data Link Spec: Semantics are tightly coupled to operational requirements, with strict validation and error handling. The meaning of each property is fixed and documented per interface.

  • MET-SWIM: Semantics are operationally driven and standardized across the MET domain, with detailed mapping from IWXXM XML to AMQP properties and explicit rules for each property’s meaning and usage.

 

Potential subjects for guidance material and/or best practices

  • Naming style of Application Properties (e.g. camelCase, sneak_case …)

  • Approach for Properties Grouping

  • Guidance/Best Practices for usage of standard AMQP properties

  • Best practices for realizing a subscription mechanism supporting topic hierarchies which may be “portable” across different Broker implementations (final aim being that perceived behavior - consumers side - is not dependent by the Broker implementation being used by the particular provider)

  • Best practices/approaches to support different Authorization scenarios while avoiding dependencies on Broker/product specific features. E.g.

    • Consumers are basically authorized to consume any kind of message once authenticated

    • Authenticated Consumers are authorized to consume only a given kind (topic?) of messages flowing over a queue

    • Authenticated Consumers are authorized to consume messages depending on values of some metadata attached to them (e.g. value of “subject” property, value of metadata related to geographical coverage etc..).

 

The following is a example for a possible generic guidance on AMQP Properties (Standard and Application)

 

Guidelines for AMQP v1.0 Message Properties Definition

1. Introduction

In the context of fostering interoperability among different stakeholders, a consistent and clear definition of message properties is very important. In the following, we are providing guidelines for effective use of both standard AMQP properties and custom application-specific properties. These are meant to enhance message clarity, facilitate efficient routing and processing, and ultimately contribute to a more resilient and interoperable messaging ecosystem.

2. AMQP 1.0 Message Structure Overview

An AMQP 1.0 message is composed of several sections, each serving a distinct purpose. Below is a graphical representation as found in the AMQP 1.0 Standard:

AMQP-metadata.png
AMQP Message structure

While we will focus specifically on message properties, it's useful to briefly understand their context within the overall AMQP 1.0 message structure:

  • Properties Section: Contains a set of standard, well-defined message properties (e.g., message-id, content-type, correlation-id).

  • Application Properties Section: A map of application-defined properties. These are custom key-value pairs that carry application-specific metadata.

  • Data Section(s): Contains the actual message payload, which can be binary, string, or a sequence of AMQP data types.

  • Delivery Annotations Section: A map of delivery-specific annotations.

  • Message Annotations Section: A map of message-specific annotations.

  • Header Section: Contains durable, priority, and time-to-live information.

  • Footer Section: Contains information about the message's outcome.

Our primary focus will be on the Properties and Application Properties sections.

3. Standard AMQP Message Properties

The AMQP v1.0 specification defines a set of standard properties within the properties section of a message. These properties provide common metadata that can be understood and processed by generic AMQP intermediaries and applications.

Here's an explanation of key standard properties, their aim, and best practices for their usage:

3.1 message-id

  • Aim: A unique identifier for the message. It allows for duplicate detection, message tracking, and idempotent processing.

  • Best Use:

    • Uniqueness: Ensure global uniqueness, typically by using UUIDs (Universally Unique Identifiers).

    • Tracking: Used by monitoring systems to trace the message's journey through the system.

    • Idempotency: Consumers can use the message-id to ensure that processing a message multiple times (e.g., due to retries) has the same effect as processing it once.

  • Example:

    message-id: "urn:uuid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6"
    • Scenario: An order placement system sends an OrderCreated event. The message-id ensures that if the event is redelivered, the order is not processed twice.

correlation-id

  • Aim: An identifier that links a response message to a request message, or relates messages within a conversation or workflow.

  • Best Use:

    • Request-Reply Patterns: The sender of a request message includes a correlation-id. The receiver of the request copies this correlation-id into the response message, allowing the original sender to match the response to its request.

    • Workflow Tracing: Link related messages in a complex business process (e.g., an OrderCreated message and a subsequent PaymentProcessed message that are part of the same order fulfillment workflow).

  • Example:

    • Request Message:

      message-id: "req-12345" reply-to: "queue://my.reply.queue"
    • Response Message:

      message-id: "res-67890" correlation-id: "req-12345" to: "queue://my.reply.queue"
    • Scenario: A client sends a request to a microservice. The microservice processes the request and sends a response. The correlation-id allows the client to identify which request the response belongs to.

3.3 content-type

  • Aim: Describes the type of the message body (payload), typically using a MIME type.

  • Best Use:

    • Payload Interpretation: Crucial for consumers to correctly deserialize and interpret the message content.

    • Standardization: Use well-known MIME types (e.g., application/json, application/xml, text/plain, application/octet-stream).

    • Character Encoding: Often combined with content-encoding for text-based payloads.

  • Example:

    content-type: "application/json" content-encoding: "utf-8"
    • Scenario: A service sends a message containing customer data in JSON format. The content-type tells the receiving service how to parse the message body.

3.4 content-encoding

  • Aim: Describes the encoding of the message body, typically used for compression or character encoding.

  • Best Use:

    • Compression: Indicate if the payload is compressed (e.g., gzip, deflate).

    • Character Sets: For text-based content, specify the character encoding (e.g., utf-8, iso-8859-1).

  • Example:

    content-type: "application/json" content-encoding: "gzip"
    • Scenario: A large JSON payload is compressed using Gzip before being sent. The content-encoding property informs the receiver to decompress it.

3.5 to

  • Aim: The address of the intended destination for the message.

  • Best Use:

    • Direct Routing: Used by intermediaries to route the message to a specific queue or topic.

    • Recipient Identification: Helps the recipient identify if the message is intended for them, especially in scenarios where a single consumer might listen to multiple addresses.

  • Example:

    to: "queue://orders.processing"
    • Scenario: An order service sends a message directly to the orders.processing queue.

3.6 reply-to

  • Aim: The address to which a reply message should be sent.

  • Best Use:

    • Request-Reply Patterns: Specifies the return address for responses in an asynchronous request-reply interaction.

    • Dynamic Reply Queues: Often used with temporary or dynamically created queues for replies.

  • Example:

    reply-to: "queue://temp.reply.queue.user123"
    • Scenario: A client sends a request and specifies a temporary queue where it expects the response.

3.7 subject

  • Aim: A descriptive subject for the message.

  • Best Use:

    • Human Readability: Useful for logging, debugging, and monitoring systems to quickly understand the message's purpose without inspecting the payload.

    • Filtering (Limited): Can sometimes be used for basic filtering, though application-properties are generally preferred for programmatic filtering.

  • Example:

    subject: "New Customer Registration"
    • Scenario: An event indicating a new customer has registered. The subject provides a high-level summary.

3.8 creation-time

  • Aim: The time at which the message was created.

  • Best Use:

    • Auditing: Essential for auditing and compliance, providing a timestamp of when the event occurred.

    • Ordering (Informal): Can be used as a hint for message ordering, though guaranteed ordering requires specific messaging patterns.

    • Latency Measurement: Helps in calculating message latency.

  • Example:

    creation-time: 1678886400000 // Unix timestamp in milliseconds (e.g., March 15, 2023 12:00:00 PM UTC)
    • Scenario: A system generates a sensor reading. The creation-time records when the reading was taken.

3.9 absolute-expiry-time

  • Aim: The time at which the message is considered expired and should be discarded.

  • Best Use:

    • Time-Sensitive Messages: For messages that are only relevant for a limited duration (e.g., real-time stock quotes, temporary alerts).

    • Resource Management: Prevents stale messages from accumulating in queues.

    • Alternative to TTL: Provides an absolute timestamp for expiry, rather than a relative time-to-live.

  • Example:

    absolute-expiry-time: 1678886460000 // 60 seconds after creation-time example
    • Scenario: A message containing a one-time password (OTP) that is valid for 60 seconds.

3.10 group-id

  • Aim: An identifier for a group of messages that are related and should be processed together.

  • Best Use:

    Status: Working Material