This is an incremental tutorial on how to share JetStream assets between accounts.
The options are not going to be fully explained unless they are vital. The tools used
are nats cli and nsc, and both have excellent --help.
Note that this tutorial is expected to be followed in sequence.
- Understand imports and exports and JWT
- Be aware of JetStream Protocol subjects
First lets create an operator and system account
~/cross-account $ nsc add operator O
[ OK ] generated and stored operator key "OAO65EOMFETMKBGLI2RIPKYBRXV43OP6RLH7RXH3OWWE6F4VLXZ7HPFZ"
[ OK ] added operator "O"
[ OK ] When running your own nats-server, make sure they run at least version 2.2.0
~/cross-account $ nsc add account SYS
[ OK ] generated and stored account key "AAB5RTVCXPEMQLLDRGGQ44Z45Z3RMFPX6UXMMILEMUKAVEZGTVZDDFDB"
[ OK ] added account "SYS"
~/cross-account $ nsc edit operator --system-account SYS
[ OK ] set system account "AAB5RTVCXPEMQLLDRGGQ44Z45Z3RMFPX6UXMMILEMUKAVEZGTVZDDFDB"
[ OK ] edited operator "O"
Now we are going to create the account hosting the JetStream that we are going to access from different scenarios
~/cross-account $nsc add account A
[ OK ] generated and stored account key "AAGVYTYA44QM2ECWGPALJPFJE4VMZF5ZZEIOYUZOW562REI43QKEILKY"
[ OK ] added account "A"
# here we enable jetstream on account A
~/cross-account $ nsc edit account A --js-disk-storage -1 --js-mem-storage -1
[ OK ] changed global max mem storage to -1
[ OK ] changed global max disk storage to -1
[ OK ] edited account "A"
In order for another account to access JetStream assets in account A, account A must define some exports that the foreign account must import. To keep the example concise, the entire JetStream from account A will be exported. You can refer to other tutorials that allow you to clamp on permissions.
The principal export, will export the complete JS API - this will allow the importer to create and reference APIs that manipulate account A's JetStream.
~/cross-account $ nsc add export --account A --subject \$JS.API.\> --service
[ OK ] added public service export "$JS.API.>"
Note that the above export is all that is required to access JetStream from account A, so long as you are using pull consumers. For now let's ignore push consumers.
Now we create an account that is going to have access to the JetStream from account A
~/cross-account $ nsc add account B
[ OK ] generated and stored account key "AAKRP54IJWX2TJKVHXJ6PJTTPWLNJPFIG5V3IQ5BTDDRFL7TZY564MY7"
[ OK ] added account "B"
Add the import for the JetStream APIs on account A
~/cross-account $ nsc add import --account B --src-account A --remote-subject \$JS.API.\> --local-subject \$JS.A.API.\> --service --name \$JS.A.API.\>
[ OK ] added service import "$JS.API.>"
Now that we have the operator, system account, and the accounts A and B, we can generate the server configuration
# generate the config
~/cross-account $ nsc generate config --mem-resolver --config-file accounts.conf --force
[ OK ] wrote server configuration to `~/cross-account/accounts.conf`
To use the configuration, we are going to create a hub configuration that will load the accounts.conf this will allow
other more complex setups to share the account configurations. The base configuration for the server hub.conf should have
this content:
port: 4222
server_name: hub-server
jetstream {
store_dir: "./hub"
}
leafnodes {
port: 7422
}
include ./accounts.conf
You can now start the server:
~/cross-account $ nats-server -c hub.conf
[68331] 2024/02/23 10:08:38.964858 [INF] Starting nats-server
[68331] 2024/02/23 10:08:38.964938 [INF] Version: 2.10.11
[68331] 2024/02/23 10:08:38.964940 [INF] Git: [not set]
[68331] 2024/02/23 10:08:38.964942 [INF] Name: hub-server
[68331] 2024/02/23 10:08:38.964944 [INF] Node: N45hlKXl
[68331] 2024/02/23 10:08:38.964945 [INF] ID: NAVU3I7TRKMLJFQXTTEV75GSGKF5JTTEU3MAPEH54S3WK425J4JBB5M3
[68331] 2024/02/23 10:08:38.964955 [INF] Using configuration file: hub.conf
[68331] 2024/02/23 10:08:38.964957 [INF] Trusted Operators
[68331] 2024/02/23 10:08:38.964958 [INF] System : ""
[68331] 2024/02/23 10:08:38.964961 [INF] Operator: "O"
[68331] 2024/02/23 10:08:38.964962 [INF] Issued : 2024-02-23 09:19:10 -0400 AST
[68331] 2024/02/23 10:08:38.964978 [INF] Expires : Never
[68331] 2024/02/23 10:08:38.965464 [INF] Starting JetStream
[68331] 2024/02/23 10:08:38.965743 [INF] _ ___ _____ ___ _____ ___ ___ _ __ __
[68331] 2024/02/23 10:08:38.965748 [INF] _ | | __|_ _/ __|_ _| _ \ __| /_\ | \/ |
[68331] 2024/02/23 10:08:38.965749 [INF] | || | _| | | \__ \ | | | / _| / _ \| |\/| |
[68331] 2024/02/23 10:08:38.965751 [INF] \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_| |_|
[68331] 2024/02/23 10:08:38.965752 [INF]
[68331] 2024/02/23 10:08:38.965753 [INF] https://docs.nats.io/jetstream
[68331] 2024/02/23 10:08:38.965754 [INF]
[68331] 2024/02/23 10:08:38.965756 [INF] ---------------- JETSTREAM ----------------
[68331] 2024/02/23 10:08:38.965759 [INF] Max Memory: 48.00 GB
[68331] 2024/02/23 10:08:38.965761 [INF] Max Storage: 371.22 GB
[68331] 2024/02/23 10:08:38.965762 [INF] Store Directory: "hub/jetstream"
[68331] 2024/02/23 10:08:38.965764 [INF] Domain: hub
[68331] 2024/02/23 10:08:38.965765 [INF] -------------------------------------------
[68331] 2024/02/23 10:08:38.966233 [INF] Listening for leafnode connections on 0.0.0.0:7422
[68331] 2024/02/23 10:08:38.966516 [INF] Listening for client connections on 0.0.0.0:4222
[68331] 2024/02/23 10:08:38.966676 [INF] Server is ready
Now let's create a user in account A, that will create a stream:
~/cross-account $ nsc add user a -a A
[ OK ] generated and stored user key "UCS7EF3IBDPDB7XAR27T32YFOZU5MA6JQH72UXK3LDAA3I64UW2IB4BP"
[ OK ] generated user creds file `~/.local/share/nats/nsc/keys/creds/O/A/a.creds`
[ OK ] added user "a" to account "A"
~/cross-account $ cp ~/.local/share/nats/nsc/keys/creds/O/A/a.creds a.creds
Now let's use nats cli tool to create the stream S capturing subjects S.>:
~/cross-account $ nats --creds ./a.creds stream add
? Stream Name S
? Subjects S.>
? Storage file
? Replication 1
? Retention Policy Limits
? Discard Policy Old
? Stream Messages Limit -1
? Per Subject Messages Limit -1
? Total Stream Size -1
? Message TTL -1
? Max Message Size -1
? Duplicate tracking time window 2m0s
? Allow message Roll-ups No
? Allow message deletion Yes
? Allow purging subjects or the entire stream Yes
Stream S was created
Information for Stream S created 2024-02-23 10:09:25
Subjects: S.>
Replicas: 1
Storage: File
Options:
Retention: Limits
Acknowledgments: true
Discard Policy: Old
Duplicate Window: 2m0s
Direct Get: true
Allows Msg Delete: true
Allows Purge: true
Allows Rollups: false
Limits:
Maximum Messages: unlimited
Maximum Per Subject: unlimited
Maximum Bytes: unlimited
Maximum Age: unlimited
Maximum Message Size: unlimited
Maximum Consumers: unlimited
State:
Messages: 0
Bytes: 0 B
First Sequence: 0
Last Sequence: 0
Active Consumers: 0
Now that we have a stream, lets add a message:
~/cross-account $ nats --creds ./a.creds req S.1 "hello"
10:10:04 Sending request on "S.1"
10:10:04 Received with rtt 271.235µs
{"stream":"S", "domain":"hub", "seq":1}
Now that we have the source asset in account A, let's create a consumer from account B. Note that we have not enabled JetStream in account B, we are simply going to use the exported APIs from account A to create a consumer that lives in account A, but is used by account B.
The first step is to create a user for B:
~/cross-account $ nsc add user b -a B
[ OK ] generated and stored user key "UAXKKVZXODR4GDDBVJRW7AP5VXWLZO2NI4YDYXCDQ7ZUHI36GVNLZYDO"
[ OK ] generated user creds file `~/.local/share/nats/nsc/keys/creds/O/B/b.creds`
[ OK ] added user "b" to account "B"
~/cross-account $ cp ~/.local/share/nats/nsc/keys/creds/O/B/b.creds b.creds
Now let's connect to the server as the user in B, and create the consumer.
Note that we are providing the --js-domain with a value of A. Note that
our server is not configured to use domains in this example, but the convention
on how we defined the import on account B (--remote-subject \$JS.API.\> --local-subject \$JS.A.API.\> ,
makes this work exactly the same, so tooling that is domain aware will work because it will use the
value specified as the domain as $JS.{domain}.API.> when building JetStream API requests:
nats --creds ./b.creds --js-domain A consumer add
? Consumer name b
? Delivery target (empty for Pull Consumers)
? Start policy (all, new, last, subject, 1h, msg sequence) all
? Acknowledgment policy explicit
? Replay policy instant
? Filter Stream by subjects (blank for all)
? Maximum Allowed Deliveries -1
? Maximum Acknowledgments Pending 0
? Deliver headers only without bodies No
? Add a Retry Backoff Policy No
? Select a Stream S
Information for Consumer S > b created 2024-02-23T10:14:54-04:00
Configuration:
Name: b
Pull Mode: true
Deliver Policy: All
Ack Policy: Explicit
Ack Wait: 30.00s
Replay Policy: Instant
Max Ack Pending: 1,000
Max Waiting Pulls: 512
State:
Last Delivered Message: Consumer sequence: 0 Stream sequence: 0
Acknowledgment Floor: Consumer sequence: 0 Stream sequence: 0
Outstanding Acks: 0 out of maximum 1,000
Redelivered Messages: 0
Unprocessed Messages: 1
Waiting Pulls: 0 of maximum 512
If you read carefully, there is a message already waiting for the consumer. Now let's use a pull consumer in account B.
Let's access have the consumer read the message:
~/cross-account $ nats --creds ./b.creds --js-domain A consumer next S b
[10:16:20] subj: S.1 / tries: 1 / cons seq: 1 / str seq: 1 / pending: 0
hello
Acknowledged message
To simplify the initial example, we punted push consumers. Push consumers can be thought as subscribers. The listen on a subject and receive messages. This is in contrast to a pull consumer where the consumer makes a request providing a subject that it will listen on for messages. The push consumer is auto-detected by the server when it subscribes to its matching subject. The instant the server detects interest, it will start pushing messages.
To enable account B to get messages from account A sent to a push consumer we need another export in account A.
Push consumer are discouraged when you write a service, but are required for some of the underlying functionality. If you want to use watchers on KV, etc you will run into them.
~/cross-account $ nsc add export --account A --subject "toaccount.*.>" --account-token-position 2
[ OK ] added public stream export "toaccount.*.>"
To enable flow control messages between the server and consumers we also need to add another export:
~/cross-account $ nsc add export --account A --subject \$JS.FC.\> --service
[ OK ] added public service export "$JS.FC.>"
Instead of defining very many exports (one for each consumer in each account), we are going to do just
one. The subject for the export has a prefix toaccount followed by two wildcards. The first wildcard
must be the account ID of the importing account, this will provide safety. The second wildcard is a differentiator that will be used to identify the consumer,
and enable the ability to use prefixes later on.
~/cross-account $ nsc add import --account B --src-account A --remote-subject "toaccount.AAKRP54IJWX2TJKVHXJ6PJTTPWLNJPFIG5V3IQ5BTDDRFL7TZY564MY7.>" --name from_A
[ OK ] added stream import "toaccount.AAKRP54IJWX2TJKVHXJ6PJTTPWLNJPFIG5V3IQ5BTDDRFL7TZY564MY7.>"
~/cross-account $ nsc add import --account B --src-account A --remote-subject \$JS.FC.\> --service
[ OK ] added service import "$JS.FC.>"
Note that the first added import keeps the last wildcard, we will set those that name on the consumer. The first one, as required by the export, is the account ID of the importing account (B)
Regenerate the account configuration
~/cross-account $ nsc generate config --mem-resolver --config-file accounts.conf --force
[ OK ] wrote server configuration to `~/cross-account/accounts.conf`
Success!! - generated `~/cross-account/accounts.conf`
And restart the server:
~/cross-account $ killall nats-server; nats-server -c hub.conf
[68994] 2024/02/23 11:28:14.798026 [INF] Starting nats-server
[68994] 2024/02/23 11:28:14.798111 [INF] Version: 2.10.11
[68994] 2024/02/23 11:28:14.798114 [INF] Git: [not set]
[68994] 2024/02/23 11:28:14.798115 [INF] Name: hub-server
[68994] 2024/02/23 11:28:14.798117 [INF] Node: N45hlKXl
[68994] 2024/02/23 11:28:14.798119 [INF] ID: NCAEIOV7RTAF7L3BFEYPHXEDD2QVF7MPEH2HH7ZIOBKGCTXB3DDIFX6P
[68994] 2024/02/23 11:28:14.798129 [INF] Using configuration file: hub.conf
[68994] 2024/02/23 11:28:14.798131 [INF] Trusted Operators
[68994] 2024/02/23 11:28:14.798132 [INF] System : ""
[68994] 2024/02/23 11:28:14.798135 [INF] Operator: "O"
[68994] 2024/02/23 11:28:14.798137 [INF] Issued : 2024-02-23 09:19:10 -0400 AST
[68994] 2024/02/23 11:28:14.798151 [INF] Expires : Never
[68994] 2024/02/23 11:28:14.798678 [INF] Starting JetStream
[68994] 2024/02/23 11:28:14.799007 [INF] _ ___ _____ ___ _____ ___ ___ _ __ __
[68994] 2024/02/23 11:28:14.799013 [INF] _ | | __|_ _/ __|_ _| _ \ __| /_\ | \/ |
[68994] 2024/02/23 11:28:14.799015 [INF] | || | _| | | \__ \ | | | / _| / _ \| |\/| |
[68994] 2024/02/23 11:28:14.799016 [INF] \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_| |_|
[68994] 2024/02/23 11:28:14.799017 [INF]
[68994] 2024/02/23 11:28:14.799019 [INF] https://docs.nats.io/jetstream
[68994] 2024/02/23 11:28:14.799020 [INF]
[68994] 2024/02/23 11:28:14.799022 [INF] ---------------- JETSTREAM ----------------
[68994] 2024/02/23 11:28:14.799025 [INF] Max Memory: 48.00 GB
[68994] 2024/02/23 11:28:14.799027 [INF] Max Storage: 375.03 GB
[68994] 2024/02/23 11:28:14.799028 [INF] Store Directory: "hub/jetstream"
[68994] 2024/02/23 11:28:14.799030 [INF] Domain: hub
[68994] 2024/02/23 11:28:14.799031 [INF] -------------------------------------------
[68994] 2024/02/23 11:28:14.799961 [INF] Starting restore for stream 'AAGVYTYA44QM2ECWGPALJPFJE4VMZF5ZZEIOYUZOW562REI43QKEILKY > S'
[68994] 2024/02/23 11:28:14.800334 [INF] Restored 1 messages for stream 'AAGVYTYA44QM2ECWGPALJPFJE4VMZF5ZZEIOYUZOW562REI43QKEILKY > S' in 0s
[68994] 2024/02/23 11:28:14.800397 [INF] Recovering 1 consumers for stream - 'AAGVYTYA44QM2ECWGPALJPFJE4VMZF5ZZEIOYUZOW562REI43QKEILKY > S'
[68994] 2024/02/23 11:28:14.800879 [INF] Listening for leafnode connections on 0.0.0.0:7422
[68994] 2024/02/23 11:28:14.801150 [INF] Listening for client connections on 0.0.0.0:4222
[68994] 2024/02/23 11:28:14.801276 [INF] Server is ready
Let's create the push consumer:
~/cross-account [1]$ nats --creds ./b.creds --js-domain A consumer add
? Delivery target (empty for Pull Consumers) toaccount.AAKRP54IJWX2TJKVHXJ6PJTTPWLNJPFIG5V3IQ5BTDDRFL7TZY564MY7.bb
? Delivery Queue Group
? Start policy (all, new, last, subject, 1h, msg sequence) all
? Acknowledgment policy explicit
? Replay policy instant
? Filter Stream by subjects (blank for all)
? Maximum Allowed Deliveries -1
? Maximum Acknowledgments Pending 0
? Idle Heartbeat 30s
? Enable Flow Control, ie --flow-control Yes
? Deliver headers only without bodies No
? Add a Retry Backoff Policy No
? Select a Stream S
Information for Consumer S > bb created 2024-02-23T11:33:03-04:00
Configuration:
Name: bb
Delivery Subject: toaccount.AAKRP54IJWX2TJKVHXJ6PJTTPWLNJPFIG5V3IQ5BTDDRFL7TZY564MY7.bb
Deliver Policy: All
Ack Policy: Explicit
Ack Wait: 30.00s
Replay Policy: Instant
Max Ack Pending: 1,000
Idle Heartbeat: 30.00s
Flow Control: true
State:
Last Delivered Message: Consumer sequence: 0 Stream sequence: 0
Acknowledgment Floor: Consumer sequence: 0 Stream sequence: 0
Outstanding Acks: 0 out of maximum 1,000
Redelivered Messages: 0
Unprocessed Messages: 1
Active Interest: No interest
And try it out:
~/cross-account $ nats --creds ./b.creds --js-domain A consumer sub S bb
Subscribing to topic toaccount.AAKRP54IJWX2TJKVHXJ6PJTTPWLNJPFIG5V3IQ5BTDDRFL7TZY564MY7.bb auto acknowlegement: true
Consumer Info:
Ack Policy: Explicit
Ack Wait: 30s
[11:35:48] subj: S.1 / tries: 1 / cons seq: 1 / str seq: 1 / pending: 0
hello
Note the above consumer subscribed to the target subject which is allowed because of the export on account A.
The previous example enable account B to access JetStream hosted by account A. We are going to add a new example where we add a third account that mirrors the stream in A on its own account - that is the mirror is populated with data from A, but the stream and consumers are owned by the new account.
# add the account
~/cross-account $ nsc add account C
[ OK ] generated and stored account key "ADXJSK7I2VHJORTGIWCJC6L2JJEQ644ESFWTCKC6DKEHY5O5GBLBZWAJ"
[ OK ] added account "C"
# enable jetstream
~/cross-account $ nsc edit account --js-disk-storage -1 --js-mem-storage -1
[ OK ] changed global max mem storage to -1
[ OK ] changed global max disk storage to -1
[ OK ] edited account "C"
# import jetstream from account A
~/cross-account $ nsc add import --account C --src-account A --remote-subject \$JS.API.\> --local-subject \$JS.A.API.\> --service --name \$JS.A.API.\>
[ OK ] added service import "$JS.API.>"
# enable stream mirroring
~/cross-account $ nsc add import --account C --src-account A --remote-subject toaccount.ADXJSK7I2VHJORTGIWCJC6L2JJEQ644ESFWTCKC6DKEHY5O5GBLBZWAJ.\> --name from_A
[ OK ] added stream import "toaccount.ADXJSK7I2VHJORTGIWCJC6L2JJEQ644ESFWTCKC6DKEHY5O5GBLBZWAJ.>"
~/cross-account $ nsc add import --account C --src-account A --remote-subject \$JS.FC.\> --service
[ OK ] added service import "$JS.FC.>"
# add an user to account C
~/cross-account $ nsc add user c --account C
[ OK ] generated and stored user key "UCOSMKMJDLOYKAU4PNVWMC3NWYLLXD36SIFGHP5333UZMDKYFJJWYGRZ"
[ OK ] generated user creds file `~/.local/share/nats/nsc/keys/creds/O/C/c.creds`
[ OK ] added user "c" to account "C"
# copy the creds
~/cross-account $ cp ~/.local/share/nats/nsc/keys/creds/O/C/c.creds c.creds
Need to regenerate the account configuration and restart the server
~/cross-account [1]$ nsc generate config --mem-resolver --config-file accounts.conf --force
[ OK ] wrote server configuration to `~/cross-account/accounts.conf`
Success!! - generated `~/cross-account/accounts.conf`
~/cross-account $ killall nats-server; nats-server -c hub.conf
[69230] 2024/02/23 11:50:09.798905 [INF] Starting nats-server
[69230] 2024/02/23 11:50:09.798984 [INF] Version: 2.10.11
[69230] 2024/02/23 11:50:09.798986 [INF] Git: [not set]
[69230] 2024/02/23 11:50:09.798987 [INF] Name: hub-server
[69230] 2024/02/23 11:50:09.798990 [INF] Node: N45hlKXl
[69230] 2024/02/23 11:50:09.798991 [INF] ID: NBPMZODZU42Z2QM2C2LR2XUFZXZWZFQUOPFAKJZ4DTLCCBYVM266H4ZB
[69230] 2024/02/23 11:50:09.799001 [INF] Using configuration file: hub.conf
[69230] 2024/02/23 11:50:09.799003 [INF] Trusted Operators
[69230] 2024/02/23 11:50:09.799005 [INF] System : ""
[69230] 2024/02/23 11:50:09.799007 [INF] Operator: "O"
[69230] 2024/02/23 11:50:09.799009 [INF] Issued : 2024-02-23 09:19:10 -0400 AST
[69230] 2024/02/23 11:50:09.799027 [INF] Expires : Never
[69230] 2024/02/23 11:50:09.799790 [INF] Starting JetStream
[69230] 2024/02/23 11:50:09.800035 [INF] _ ___ _____ ___ _____ ___ ___ _ __ __
[69230] 2024/02/23 11:50:09.800040 [INF] _ | | __|_ _/ __|_ _| _ \ __| /_\ | \/ |
[69230] 2024/02/23 11:50:09.800042 [INF] | || | _| | | \__ \ | | | / _| / _ \| |\/| |
[69230] 2024/02/23 11:50:09.800043 [INF] \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_| |_|
[69230] 2024/02/23 11:50:09.800044 [INF]
[69230] 2024/02/23 11:50:09.800051 [INF] https://docs.nats.io/jetstream
[69230] 2024/02/23 11:50:09.800053 [INF]
[69230] 2024/02/23 11:50:09.800054 [INF] ---------------- JETSTREAM ----------------
[69230] 2024/02/23 11:50:09.800057 [INF] Max Memory: 48.00 GB
[69230] 2024/02/23 11:50:09.800059 [INF] Max Storage: 375.02 GB
[69230] 2024/02/23 11:50:09.800060 [INF] Store Directory: "hub/jetstream"
[69230] 2024/02/23 11:50:09.800062 [INF] -------------------------------------------
[69230] 2024/02/23 11:50:09.800803 [INF] Starting restore for stream 'AAGVYTYA44QM2ECWGPALJPFJE4VMZF5ZZEIOYUZOW562REI43QKEILKY > S'
[69230] 2024/02/23 11:50:09.801312 [INF] Restored 1 messages for stream 'AAGVYTYA44QM2ECWGPALJPFJE4VMZF5ZZEIOYUZOW562REI43QKEILKY > S' in 0s
[69230] 2024/02/23 11:50:09.801374 [INF] Recovering 2 consumers for stream - 'AAGVYTYA44QM2ECWGPALJPFJE4VMZF5ZZEIOYUZOW562REI43QKEILKY > S'
[69230] 2024/02/23 11:50:09.802038 [INF] Listening for leafnode connections on 0.0.0.0:7422
[69230] 2024/02/23 11:50:09.802290 [INF] Listening for client connections on 0.0.0.0:4222
[69230] 2024/02/23 11:50:09.802425 [INF] Server is ready
Let's create the mirror:
~/cross-account [1]$ nats --creds ./c.creds stream add --mirror S
? Stream Name SS
? Storage file
? Replication 1
? Retention Policy Limits
? Discard Policy Old
? Stream Messages Limit -1
? Total Stream Size -1
? Message TTL -1
? Max Message Size -1
? Allow message Roll-ups No
? Allow message deletion Yes
? Allow purging subjects or the entire stream Yes
? Adjust mirror start No
? Adjust mirror filter and transform No
? Import mirror from a different JetStream domain Yes
? Foreign JetStream domain name A
? Delivery prefix toaccount.ADXJSK7I2VHJORTGIWCJC6L2JJEQ644ESFWTCKC6DKEHY5O5GBLBZWAJ
Stream SS was created
Information for Stream SS created 2024-02-23 12:11:55
Replicas: 1
Storage: File
Options:
Retention: Limits
Acknowledgments: true
Discard Policy: Old
Duplicate Window: 0s
Direct Get: true
Allows Msg Delete: true
Allows Purge: true
Allows Rollups: false
Limits:
Maximum Messages: unlimited
Maximum Per Subject: unlimited
Maximum Bytes: unlimited
Maximum Age: unlimited
Maximum Message Size: unlimited
Maximum Consumers: unlimited
Replication:
Mirror: S, API Prefix: $JS.A.API, Delivery Prefix: toaccount.ADXJSK7I2VHJORTGIWCJC6L2JJEQ644ESFWTCKC6DKEHY5O5GBLBZWAJ
Mirror Information:
Stream Name: S
Lag: 0
Last Seen: never
Ext. API Prefix: $JS.A.API
Ext. Delivery Prefix: toaccount.ADXJSK7I2VHJORTGIWCJC6L2JJEQ644ESFWTCKC6DKEHY5O5GBLBZWAJ
State:
Messages: 0
Bytes: 0 B
First Sequence: 0
Last Sequence: 0
Active Consumers: 0
Now we create a consumer in account C from stream SS
~/cross-account $ nats --creds ./c.creds consumer add
? Consumer name cc
? Delivery target (empty for Pull Consumers)
? Start policy (all, new, last, subject, 1h, msg sequence) all
? Acknowledgment policy explicit
? Replay policy instant
? Filter Stream by subjects (blank for all)
? Maximum Allowed Deliveries -1
? Maximum Acknowledgments Pending 0
? Deliver headers only without bodies No
? Add a Retry Backoff Policy No
? Select a Stream SS
Information for Consumer SS > cc created 2024-02-23T12:17:41-04:00
Configuration:
Name: cc
Pull Mode: true
Deliver Policy: All
Ack Policy: Explicit
Ack Wait: 30.00s
Replay Policy: Instant
Max Ack Pending: 1,000
Max Waiting Pulls: 512
State:
Last Delivered Message: Consumer sequence: 0 Stream sequence: 0
Acknowledgment Floor: Consumer sequence: 0 Stream sequence: 0
Outstanding Acks: 0 out of maximum 1,000
Redelivered Messages: 0
Unprocessed Messages: 1
Waiting Pulls: 0 of maximum 512
And pull messages from it:
~/cross-account [1]$ nats --creds ./c.creds consumer next SS cc
[12:18:29] subj: S.1 / tries: 1 / cons seq: 1 / str seq: 1 / pending: 0
hello
Acknowledged message
A leafnode is a server that is part of a cluster but provides its own authentication domain.
This means that locally authenticated clients are collocated in the remote account; traffic
in the remote account can flow to the leafnode and vice versa. With JetStream
the leafnode can specify a domain enabling two different JetStream clusters to coexist.
In order to target the different JetStream clusters, we need to modify the server configuration in hub.conf, by
specifying the domain option in the jetstream section.
port: 4222
server_name: hub-server
jetstream {
store_dir: "./hub"
domain: hub
}
leafnodes {
port: 7422
}
include ./accounts.conf
Go ahead and restart your server
~/cross-account $ nats-server -c hub.conf
[70256] 2024/02/23 13:52:47.462127 [INF] Starting nats-server
[70256] 2024/02/23 13:52:47.462221 [INF] Version: 2.10.11
[70256] 2024/02/23 13:52:47.462223 [INF] Git: [not set]
[70256] 2024/02/23 13:52:47.462225 [INF] Name: hub-server
[70256] 2024/02/23 13:52:47.462227 [INF] Node: N45hlKXl
[70256] 2024/02/23 13:52:47.462228 [INF] ID: NBGXQLOTF442NWKHHD7MKQFJRKYY2CC5GXW76D76O3N52LLEYU47H7JQ
[70256] 2024/02/23 13:52:47.462239 [INF] Using configuration file: hub.conf
[70256] 2024/02/23 13:52:47.462241 [INF] Trusted Operators
[70256] 2024/02/23 13:52:47.462242 [INF] System : ""
[70256] 2024/02/23 13:52:47.462245 [INF] Operator: "O"
[70256] 2024/02/23 13:52:47.462247 [INF] Issued : 2024-02-23 09:19:10 -0400 AST
[70256] 2024/02/23 13:52:47.462281 [INF] Expires : Never
[70256] 2024/02/23 13:52:47.462921 [INF] Starting JetStream
[70256] 2024/02/23 13:52:47.463201 [INF] _ ___ _____ ___ _____ ___ ___ _ __ __
[70256] 2024/02/23 13:52:47.463206 [INF] _ | | __|_ _/ __|_ _| _ \ __| /_\ | \/ |
[70256] 2024/02/23 13:52:47.463207 [INF] | || | _| | | \__ \ | | | / _| / _ \| |\/| |
[70256] 2024/02/23 13:52:47.463209 [INF] \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_| |_|
[70256] 2024/02/23 13:52:47.463210 [INF]
[70256] 2024/02/23 13:52:47.463211 [INF] https://docs.nats.io/jetstream
[70256] 2024/02/23 13:52:47.463213 [INF]
[70256] 2024/02/23 13:52:47.463214 [INF] ---------------- JETSTREAM ----------------
[70256] 2024/02/23 13:52:47.463217 [INF] Max Memory: 48.00 GB
[70256] 2024/02/23 13:52:47.463219 [INF] Max Storage: 375.01 GB
[70256] 2024/02/23 13:52:47.463220 [INF] Store Directory: "hub/jetstream"
[70256] 2024/02/23 13:52:47.463222 [INF] Domain: A
[70256] 2024/02/23 13:52:47.463223 [INF] -------------------------------------------
[70256] 2024/02/23 13:52:47.464045 [INF] Starting restore for stream 'AAGVYTYA44QM2ECWGPALJPFJE4VMZF5ZZEIOYUZOW562REI43QKEILKY > S'
[70256] 2024/02/23 13:52:47.464384 [INF] Restored 1 messages for stream 'AAGVYTYA44QM2ECWGPALJPFJE4VMZF5ZZEIOYUZOW562REI43QKEILKY > S' in 0s
[70256] 2024/02/23 13:52:47.464432 [INF] Recovering 1 consumers for stream - 'AAGVYTYA44QM2ECWGPALJPFJE4VMZF5ZZEIOYUZOW562REI43QKEILKY > S'
[70256] 2024/02/23 13:52:47.465018 [INF] Starting restore for stream 'ADXJSK7I2VHJORTGIWCJC6L2JJEQ644ESFWTCKC6DKEHY5O5GBLBZWAJ > SS'
[70256] 2024/02/23 13:52:47.465243 [INF] Restored 1 messages for stream 'ADXJSK7I2VHJORTGIWCJC6L2JJEQ644ESFWTCKC6DKEHY5O5GBLBZWAJ > SS' in 0s
[70256] 2024/02/23 13:52:47.465340 [INF] Recovering 1 consumers for stream - 'ADXJSK7I2VHJORTGIWCJC6L2JJEQ644ESFWTCKC6DKEHY5O5GBLBZWAJ > SS'
[70256] 2024/02/23 13:52:47.465686 [INF] Listening for leafnode connections on 0.0.0.0:7422
[70256] 2024/02/23 13:52:47.465948 [INF] Listening for client connections on 0.0.0.0:4222
[70256] 2024/02/23 13:52:47.466101 [INF] Server is ready
Let's define the leafnode's authentication domain. We could use JWT as we have done on the hub, but for sake of simplicity we are simply going to have a leaf server that has no authentication. If you want to use JWT authentication as we did for the hub, you can do so by creating a new Operator/System account and client accounts. You cannot reuse the set of identities created earlier.
For the leafnode server configuration let's do:
port: 4111
server_name: leaf-server
jetstream {
store_dir: "./leaf"
domain: leaf
}
leafnodes {
remotes = [
{
urls: ["nats://0.0.0.0:7422"]
credentials: "./a.creds"
}
]
}
Note above the remotes entry. The entry doesn't specify an account because we have not configured any accounts
on the leafnode. If you were to define accounts, you would need to add the account field to the remotes entry with a value
matching the name of the account (if config) or the ID of the account to place the remote on the local account.
We can start the leafnode:
~/cross-account $ nats-server -c leaf.conf
[74049] 2024/02/24 09:22:08.673381 [INF] Starting nats-server
[74049] 2024/02/24 09:22:08.673561 [INF] Version: 2.10.11
[74049] 2024/02/24 09:22:08.673563 [INF] Git: [not set]
[74049] 2024/02/24 09:22:08.673565 [INF] Cluster: leaf-server
[74049] 2024/02/24 09:22:08.673567 [INF] Name: leaf-server
[74049] 2024/02/24 09:22:08.673569 [INF] Node: 1nWZLJcM
[74049] 2024/02/24 09:22:08.673570 [INF] ID: NBINLHRJF7MR2GWDWFH4NDCLNGTPUEELNB45SPMECIB7BCTYK3PD54BI
[74049] 2024/02/24 09:22:08.673580 [INF] Using configuration file: leaf.conf
[74049] 2024/02/24 09:22:08.673963 [INF] Starting JetStream
[74049] 2024/02/24 09:22:08.674161 [INF] _ ___ _____ ___ _____ ___ ___ _ __ __
[74049] 2024/02/24 09:22:08.674166 [INF] _ | | __|_ _/ __|_ _| _ \ __| /_\ | \/ |
[74049] 2024/02/24 09:22:08.674168 [INF] | || | _| | | \__ \ | | | / _| / _ \| |\/| |
[74049] 2024/02/24 09:22:08.674169 [INF] \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_| |_|
[74049] 2024/02/24 09:22:08.674170 [INF]
[74049] 2024/02/24 09:22:08.674172 [INF] https://docs.nats.io/jetstream
[74049] 2024/02/24 09:22:08.674173 [INF]
[74049] 2024/02/24 09:22:08.674174 [INF] ---------------- JETSTREAM ----------------
[74049] 2024/02/24 09:22:08.674177 [INF] Max Memory: 48.00 GB
[74049] 2024/02/24 09:22:08.674179 [INF] Max Storage: 374.99 GB
[74049] 2024/02/24 09:22:08.674181 [INF] Store Directory: "leaf/jetstream"
[74049] 2024/02/24 09:22:08.674182 [INF] Domain: leaf
[74049] 2024/02/24 09:22:08.674183 [INF] -------------------------------------------
[74049] 2024/02/24 09:22:08.675421 [INF] Starting restore for stream '$G > MS'
[74049] 2024/02/24 09:22:08.676374 [INF] Restored 1 messages for stream '$G > MS' in 1ms
[74049] 2024/02/24 09:22:08.677043 [INF] Listening for client connections on 0.0.0.0:4111
[74049] 2024/02/24 09:22:08.677216 [INF] Server is ready
[74049] 2024/02/24 09:22:08.677426 [INF] 127.0.0.1:7422 - lid:8 - Leafnode connection created for account: $G
[74049] 2024/02/24 09:22:08.679528 [INF] 127.0.0.1:7422 - lid:8 - JetStream using domains: local "leaf", remote "hub"
If we connect to the leafnode, we can see what JetStream assets are available:
~/cross-account $ nats -s localhost:4111 stream ls
No Streams defined
Although we are in the leaf node, and we share traffic with account A, no assets are listed. The reason why is that we are querying the local domain (leaf) for assets. And we have none!.
We can access the remote assets by specifying the domain for the hub:
~/cross-account $ nats -s localhost:4111 stream ls --js-domain hub
╭───────────────────────────────────────────────────────────────────────────╮
│ Streams │
├──────┬─────────────┬─────────────────────┬──────────┬──────┬──────────────┤
│ Name │ Description │ Created │ Messages │ Size │ Last Message │
├──────┼─────────────┼─────────────────────┼──────────┼──────┼──────────────┤
│ S │ │ 2024-02-23 10:09:25 │ 1 │ 38 B │ 23h20m42s │
│ AA │ │ 2024-02-23 14:36:21 │ 1 │ 39 B │ 18h54m8s │
╰──────┴─────────────┴─────────────────────┴──────────┴──────┴──────────────╯
Now let's create a mirror of S, we will store the data for this stream in the leaf domain:
~/cross-account $ nats -s localhost:4111 stream add --mirror S
? Stream Name MS
? Storage file
? Replication 1
? Retention Policy Limits
? Discard Policy Old
? Stream Messages Limit -1
? Total Stream Size -1
? Message TTL -1
? Max Message Size -1
? Allow message Roll-ups No
? Allow message deletion Yes
? Allow purging subjects or the entire stream Yes
? Adjust mirror start No
? Adjust mirror filter and transform No
? Import mirror from a different JetStream domain Yes
? Foreign JetStream domain name hub
? Delivery prefix FromS
Stream MS was created
Information for Stream MS created 2024-02-24 09:23:47
Replicas: 1
Storage: File
Options:
Retention: Limits
Acknowledgments: true
Discard Policy: Old
Duplicate Window: 0s
Direct Get: true
Allows Msg Delete: true
Allows Purge: true
Allows Rollups: false
Limits:
Maximum Messages: unlimited
Maximum Per Subject: unlimited
Maximum Bytes: unlimited
Maximum Age: unlimited
Maximum Message Size: unlimited
Maximum Consumers: unlimited
Replication:
Mirror: S, API Prefix: $JS.hub.API, Delivery Prefix: FromS
Mirror Information:
Stream Name: S
Lag: 0
Last Seen: never
Ext. API Prefix: $JS.hub.API
Ext. Delivery Prefix: FromS
State:
Messages: 0
Bytes: 0 B
First Sequence: 0
Last Sequence: 0
Active Consumers: 0
Getting information on the stream, reveals that we have a mirror, and our message from the original stream is already duplicated in the mirror in the leaf:
~/cross-account $ nats -s localhost:4111 stream info
? Select a Stream MS
Information for Stream MS created 2024-02-24 09:23:47
Replicas: 1
Storage: File
Options:
Retention: Limits
Acknowledgments: true
Discard Policy: Old
Duplicate Window: 0s
Direct Get: true
Allows Msg Delete: true
Allows Purge: true
Allows Rollups: false
Limits:
Maximum Messages: unlimited
Maximum Per Subject: unlimited
Maximum Bytes: unlimited
Maximum Age: unlimited
Maximum Message Size: unlimited
Maximum Consumers: unlimited
Replication:
Mirror: S, API Prefix: $JS.hub.API, Delivery Prefix: FromS
Cluster Information:
Name: leaf-server
Leader: leaf-server
Mirror Information:
Stream Name: S
Lag: 0
Last Seen: 510ms
Ext. API Prefix: $JS.hub.API
Ext. Delivery Prefix: FromS
State:
Messages: 1
Bytes: 38 B
First Sequence: 1 @ 2024-02-23 10:10:04 UTC
Last Sequence: 1 @ 2024-02-23 10:10:04 UTC
Active Consumers: 0
Number of Subjects: 1