Discovery

The discovery system provides a pluggable mechanism for nodes to discover peer nodes and verify their TLS identities. This decouples the cluster from specific infrastructure implementations and allows operators to provide discovery information that matches their environment.

Overview

Nodes use their own node_id as their network identity when establishing connections to peers (e.g., node1:8323), and this identifier should be treated as the canonical service identity for routing, service mesh registration, and discovery integration (Istio, Consul, etcd, etc.). If your infrastructure does not provide a service mesh or internal DNS resolution, the host field serves as the fallback network address; Cue nodes prioritize host when explicitly configured, otherwise default to resolving the peer via its node_id.

Each node in the cluster needs to know: 1. The network address (host:port) of every other peer node 2. The expected identity (subject alternative name) in each peer's TLS certificate

The discovery system abstracts this information retrieval, supporting both static configurations and dynamic HTTP-based discovery.

Discovery Modes

Static Discovery

Static discovery reads peer information from a local YAML file. This is suitable for fixed, known cluster topologies where node addresses and identities don't change.

Configuration:

cluster:
  discovery_kind: "static"
  discovery_yml_path: "./discovery.yml"

Discovery File Format (discovery.yml):

nodes:
  - node_id: node1
    host: 10.0.1.11
    identity:
      kind: dns
      value: node1.localhost

  - node_id: node2
    host: 10.0.1.12
    identity:
      kind: dns
      value: node2.localhost

  - node_id: node3
    host: 10.0.1.13
    identity:
      kind: dns
      value: node3.localhost

File Fields: - node_id: Unique identifier matching the node's configured node_id - host: IP address or hostname of the node - identity: TLS certificate verification information - kind: Type of identity check (dns, ip, or spiffe) - value: The expected value to match against the certificate's SAN

HTTP Discovery

HTTP discovery queries an external HTTP endpoint that you implement. This is ideal for dynamic environments where: - Nodes come and go (auto-scaling) - IP addresses change frequently (cloud environments) - TLS certificates are dynamically issued - You want to centralize cluster membership information

Configuration:

cluster:
  discovery_kind: "http"
  discovery_http_host: "http://discovery-service:8080"

HTTP Request: - Method: GET - Path: /peers (the endpoint is fixed) - Full URL: {discovery_http_host}/peers - Headers: None required

HTTP Response Format: The endpoint must return a JSON response with the following structure:

{
  "nodes": [
    {
      "node_id": "node1",
      "host": "10.0.1.11",
      "identity": {
        "kind": "dns",
        "value": "node1.localhost"
      }
    },
    {
      "node_id": "node2",
      "host": "10.0.1.12",
      "identity": {
        "kind": "dns",
        "value": "node2.localhost"
      }
    },
    {
      "node_id": "node3",
      "host": "10.0.1.13",
      "identity": {
        "kind": "dns",
        "value": "node3.localhost"
      }
    }
  ]
}

Response Fields: - nodes (array): List of known nodes in the cluster - node_id (string): Unique node identifier - should match node ids used in cluster - host (string): IP address or hostname where the node listens for Raft traffic - identity (object): TLS certificate verification information - kind (string): Identity verification type: dns, ip, or spiffe - value (string): Expected value to match against the certificate

Best Practices for HTTP Discovery Implementation:

1. Endpoint Requirements:
2. Must be accessible from all cluster nodes
3. Should return the complete list of all nodes in the cluster

4. Should include the node's own entry

5. Update Frequency:

6. The system polls the endpoint periodically like every 2 seconds

7. Changes are detected and applied automatically

8. Error Handling:

9. If the endpoint is unreachable, the node continues with the last known peer list
10. HTTP errors (4xx, 5xx) are logged and the previous peer list is retained

Identity Verification Types

The identity field specifies how to verify a peer's TLS certificate:

Kind	Description	Value Example
dns	Verify against DNS Subject Alternative Name (SAN)	node1.cluster.local
ip	Verify against SAN by IP
spiffe	Verify against SPIFFE ID	spiffe://example.org/node1