Skip to main content

External Signing Hashing Algorithm

Introduction

This document specifies the encoding algorithm used to produce a deterministic hash of a com.daml.ledger.api.v2.interactive.PreparedTransaction. The resulting hash is signed by the holder of the external party’s private key. The signature authorizes the ledger changes described by the transaction on behalf of the external party. The specification can be implemented in any language, but certain encoding patterns are biased due to Canton being implemented in a JVM-based language and using the Java protobuf library. Those biases are made explicit in the specification. Protobuf serialization is unsuitable for signing cryptographic hashes because it is not canonical. We must define a more precise encoding specification that can be re-implemented deterministically across languages and provide the required cryptographic guarantees. See https://protobuf.dev/programming-guides/serialization-not-canonical/ for more information on the topic.

Versioning

Hashing Scheme Version

The hashing algorithm as a whole is versioned. This enables updates to accommodate changes in the underlying Daml format, or, for instance, to the way the protocol verifies signatures. The implementation must respect the specification of the version it implements.
The hashing algorithm is tied to the protocol version of the synchronizer used to synchronize the transaction. Specifically, each hashing scheme version is supported on one or several protocol versions. Implementations must use a hashing scheme version supported on the synchronizer on which the transaction is submitted.
Protocol VersionSupported Hashing Schemes
v34V2
v35V2, V3
devV2, V3, V4

Transaction Nodes

Transaction nodes are additionally individually versioned with a Daml version (also called LF version). The encoding version is decoupled from the LF version and implementations should only focus on the hashing version. However, new LF versions may introduce new fields in nodes or new node types. For that reason, the protobuf representation of a node is versioned to accommodate those future changes. In practice, every new Daml language version results in a new hashing version.

Specification

General approach

The hash of the PreparedTransaction is computed by encoding the fields specified by this section to byte arrays, and feeding those encoded values into a SHA-256 hash builder. The rest of this section details how to deterministically encode every proto message into a byte array. Sometimes during the process, partially encoded results are hashed with SHA-256, and the resulting hash value serves as the encoding in messages further up. This is explicit when necessary. Big Endian notation is used for numeric values. Furthermore, protobuf numeric values are encoded according to their Java type representation. Refer to the official protobuf documentation for more information about protobuf to Java type mappings: https://protobuf.dev/programming-guides/proto3/#scalar In particular:
In Java, unsigned 32-bit and 64-bit integers are represented using their signed counterparts, with the top bit simply being stored in the sign bit
Additionally, this is the java library used under the hood in Canton to serialize and deserialize protobuf: https://github.com/protocolbuffers/protobuf/tree/v3.25.5/java

Summary of differences between V3 and V4

V4 is currently unstable and subject to changes. For this reason it targets Protocol Version Dev only.
  • New fields: Exercise nodes with version dev include a new external_call_results field. This binds the recorded external-call result payloads shown in prepared transaction review to the external party’s signature.
  • Final hash: The hashing scheme version byte changes from 0x03 (V3) to 0x04 (V4).

Summary of differences between V2 and V3

The following is a summary of the differences between V2 and V3. The full specification for each version is provided in the sections below, where version-specific details are shown using tabs.
  • Removed prefixes: V3 removes the 0x01 “Node encoding version” byte prefix from all node encodings, and the 0x01 “Metadata encoding version” prefix from the metadata encoding.
  • New fields: key_opt (optional, composite) is added to Create, Exercise, and Fetch nodes. by_key (boolean) is added to Exercise and Fetch nodes.
  • New node type: QueryByKey is introduced in V3.
  • Metadata: max_record_time is added to the metadata encoding.
  • Final hash: The hashing scheme version byte changes from 0x02 (V2) to 0x03 (V3).
V3 introduces support for contract keys. Usage of contract keys in externally signed transactions requires usage of V3. Contract keys will not work on V2. Also note that V3 is supported from protocol version 35 onwards.

Notation and Utility Functions

  • encode: Function that takes a protobuf message or primitive type T and transforms it into an array of bytes: encode: T => byte[]
e.g:
  • to_utf_8: Function converting a Java String to its UTF-8 encoded version: to_utf_8: string => byte[]
e.g:
  • len: Function returning the size of a collection (array, list etc…) as a signed 4 bytes integer: len: Col => Int
e.g:
  • split: Function converting a Java String to a list of String, by splitting the input using the provided delimiter: split: (string, char) => byte[]
e.g:
  • ||: Symbol representing concatenation of byte arrays
e.g:
  • []: Empty byte array. Denotes that the value should not be encoded.
  • from_hex_string: Function that takes a string in the hexadecimal format as input and decodes it as a byte array: from_hex_string: string => byte[]
e.g:
  • int_to_string: Function that takes an int and converts it to a string : int_to_string: int => string
e.g:
  • some: Value wrapped in a defined optional. Should be encoded as a defined optional value: some: T => optional T
e.g:
See encoding of optional values below for details.
  • encode_key_with_maintainers (from V3 only): Function encoding a GlobalKeyWithMaintainers composite value. This is used for the key field in Create, Exercise, Fetch, and QueryByKey nodes.
  • encode_external_call_result: Function encoding an ExternalCallResult composite value. This is used for the external_call_results field in Exercise nodes.

Primitive Types

Unless otherwise specified, this is how primitive protobuf types should be encoded.
Not all protobuf types are described here, only the ones necessary to encode a PreparedTransaction message.
Even default values must be included in the encoding. For instance if an int32 field is not set in the serialized protobuf, its default value (0) should be encoded. Similarly, an empty repeated field still results in a 0x00 byte encoding (see the repeated section below for more details)

google.protobuf.Empty

bool

int64 - uint64 - sint64 - sfixed64

e.g:

int32 - uint32 - sint32 - sfixed32

e.g:

bytes / byte[]

e.g

string

e.g

Collections / Wrappers

repeated

repeated protobuf fields represent an ordered collection of values of a specific message of type T. It is critical that the order of values in the list is not modified, both for the encoding process and in the protobuf itself when submitting the transaction for execution. Below is the pseudocode algorithm encoding a protobuf value repeated T list;
This encoding function also applies to lists generated from utility functions (e.g: split).

optional

is_set returns true if the value was set in the protobuf, false otherwise.

map

The ordering of map entries in protobuf serialization is not guaranteed, making it problematic for deterministic encoding. To address this, repeated values are used instead of map throughout the protobuf definitions.

gRPC Ledger API Value

Encoding for the Value message defined in com.daml.ledger.api.v2.value.proto For clarity, all value types are exhaustively listed here. Each value is prefixed by a tag unique to its type, which is explicitly specified for each value below.

Unit

Protobuf Definition

Bool

Protobuf Definition

Int64

Protobuf Definition

Numeric

Protobuf Definition

Timestamp

Protobuf Definition

Date

Protobuf Definition

Party

Protobuf Definition

Text

Protobuf Definition

Contract_id

Protobuf Definition

Optional

Protobuf Definition Note this is conceptually the same as for the primitive optional protobuf modifier, with the addition of the type tag prefix.

List

Protobuf Definition

TextMap

Protobuf Definition TextMap.Entry
Protobuf Definition

Record

Protobuf Definition RecordField
Protobuf Definition

Variant

Protobuf Definition

Enum

Protobuf Definition

GenMap

Protobuf Definition GenMap.Entry
Protobuf Definition

Identifier

Protobuf Definition

Transaction

A transaction is a forest (list of trees). It is represented with a following protobuf message found here. The encoding function for a transaction is
encode_node_ids(node_ids) encodes lists in the same way as described before, except the encoding of a node_id is NOT done by encoding it as a string, but instead uses the following encode(node_id) function:
encode(node_id) effectively finds the corresponding node in the list of nodes and encodes the node. The node_id is an opaque value only used to reference nodes and is itself never encoded. Additionally, each node’s encoding is hashed using the sha_256 hashing algorithm. This is relevant when encoding root nodes here as well as when recursively encoding sub-nodes of Exercise and Rollback nodes as seen below.

Node

Each node’s encoding is prefixed with additional meta-information about the node, this is made explicit in the encoding of each node.
Exercise and Rollback nodes both have a children field that references other nodes by their NodeId. The following find_seed: NodeId => optional bytes function is used in the encoding:
some represents a set optional field, none an empty optional field.

Create

Exercise

For Exercise nodes, the node seed MUST be defined. Therefore it is encoded as a non optional field, as noted via the .get in find_seed(node.node_id).get. If the seed of an exercise node cannot be found in the list of node_seeds, encoding must be stopped and it should be reported as a bug.

Fetch

Rollback

Rollback nodes do not have an lf version.

QueryByKey

This node type is only present in hashing scheme V3 and higher.

Transaction Hash

Once the transaction is encoded, the hash is obtained by running sha_256 over the encoded byte array, with a hash purpose prefix:

Metadata

The final part of PreparedTransaction is metadata. Note that all fields of the metadata need to be signed. Only some fields contribute to the ledger change triggered by the transaction. The rest of the fields are required by the Canton protocol but either have no impact on the ledger change, or have already been signed indirectly by signing the transaction itself.

ProcessedDisclosedContract

Metadata Hash

Once the metadata is encoded, the hash is obtained by running sha_256 over the encoded byte array, with a hash purpose prefix:

Final Hash

Finally, compute the hash that needs to be signed to commit to the ledger changes.
This resulting hash must be signed with the protocol signing private key(s) used to onboard the external party. Both the signature along with the PreparedTransaction must be sent to the API to submit the transaction to the ledger.

Example

Example V3 implementation in Python
Example V2 implementation in Python
Both versions make use of the following common code: