Random issue with RabbitMQ Streams: badmatch,{error,einval} and offset_out_of_range

osiris_offset_out_of_range.log.txt

2022-09-09 19:35:30.047624+00:00 [debug] <0.11717.17> rabbit_stream_coordinator: running action: 'start_writer' for acqokxym_s1_1662752130037252903 on node 'rabbit@test-tall-wheat-goat-01' in epoch 1
2022-09-09 19:35:30.048177+00:00 [info] <0.18479.445> Stream: acqokxym_s1_1662752130037252903 will use /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903 for osiris log data directory
2022-09-09 19:35:30.048344+00:00 [info] <0.18478.445> rabbit_stream_coordinator: started writer acqokxym_s1_1662752130037252903 on 'rabbit@test-tall-wheat-goat-01' in 1
2022-09-09 19:35:30.048482+00:00 [debug] <0.18479.445> osiris_log:init/1 stream acqokxym_s1_1662752130037252903 max_segment_size_bytes: 1234,
2022-09-09 19:35:30.048482+00:00 [debug] <0.18479.445>            max_segment_size_chunks 256000, retention [{max_bytes,1234}]
2022-09-09 19:35:30.049105+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000000.segment
2022-09-09 19:35:30.049718+00:00 [info] <0.18479.445> osiris_writer:init/1: name: acqokxym_s1_1662752130037252903 last offset: -1 committed chunk id: -1 epoch: 1
2022-09-09 19:35:30.051112+00:00 [debug] <0.11717.17> rabbit_stream_coordinator: running action: 'start_replica' for acqokxym_s1_1662752130037252903 on node 'rabbit@test-tall-wheat-goat-03' in epoch 1
2022-09-09 19:35:30.051430+00:00 [debug] <0.11717.17> rabbit_stream_coordinator: running action: 'start_replica' for acqokxym_s1_1662752130037252903 on node 'rabbit@test-tall-wheat-goat-02' in epoch 1
2022-09-09 19:35:30.051637+00:00 [debug] <0.11717.17> rabbit_stream_coordinator: running action: 'update_mnesia' for acqokxym_s1_1662752130037252903
2022-09-09 19:35:30.051795+00:00 [debug] <0.18482.445> rabbit_stream_coordinator: running mnesia update for acqokxym_s1_1662752130037252903: #{epoch => 1,event_formatter => {rabbit_stream_queue,format_osiris_event,[{resource,<<97,99,...>>,queue,<<...>>}]},leader_node => 'rabbit@test-tall-wheat-goat-01',leader_pid => <0.18479.445>,max_segment_size_bytes => 1234,name => [97,99,113,111,107,120,121,109|...],nodes => ['rabbit@test-tall-wheat-goat-01','rabbit@test-tall-wheat-goat-03','rabbit@test-tall-wheat-goat-02'],reference => {resource,<<97,99,113,111,107,120,...>>,queue,<<115,49>>},replica_nodes => ['rabbit@test-tall-wheat-goat-03','rabbit@test-tall-wheat-goat-02'],...}
2022-09-09 19:35:30.063257+00:00 [debug] <0.18490.445> trying to connect to replica at [{127,0,1,1},"test-tall-wheat-goat-02"]
2022-09-09 19:35:30.063525+00:00 [debug] <0.18490.445> trying to connect to {127,0,1,1}
2022-09-09 19:35:30.064090+00:00 [debug] <0.18490.445> osiris replica connection refused, host:{127,0,1,1} - port: 6398
2022-09-09 19:35:30.064257+00:00 [debug] <0.18490.445> trying to connect to "test-tall-wheat-goat-02"
2022-09-09 19:35:30.070231+00:00 [debug] <0.18490.445> successfully connected to host "test-tall-wheat-goat-02"
2022-09-09 19:35:30.070975+00:00 [debug] <0.18490.445> osiris_segment:init_data_reader/2 acqokxym_s1_1662752130037252903 at 0 prev empty local range: empty
2022-09-09 19:35:30.071590+00:00 [info] <0.18490.445> starting osiris replica reader acqokxym_s1_1662752130037252903 at offset 0
2022-09-09 19:35:30.071770+00:00 [debug] <0.18490.445> sent committed offset information to the leader at <0.18479.445>
2022-09-09 19:35:30.071927+00:00 [debug] <0.18488.445> trying to connect to replica at [{127,0,1,1},"test-tall-wheat-goat-03"]
2022-09-09 19:35:30.072074+00:00 [debug] <0.18488.445> trying to connect to {127,0,1,1}
2022-09-09 19:35:30.072441+00:00 [debug] <0.18488.445> osiris replica connection refused, host:{127,0,1,1} - port: 6018
2022-09-09 19:35:30.072591+00:00 [debug] <0.18488.445> trying to connect to "test-tall-wheat-goat-03"
2022-09-09 19:35:30.072958+00:00 [info] <0.18481.445> rabbit_stream_coordinator: acqokxym_s1_1662752130037252903: replica started on rabbit@test-tall-wheat-goat-02 in 1 pid <46471.9973.713>
2022-09-09 19:35:30.076388+00:00 [debug] <0.18488.445> successfully connected to host "test-tall-wheat-goat-03"
2022-09-09 19:35:30.077184+00:00 [debug] <0.18488.445> osiris_segment:init_data_reader/2 acqokxym_s1_1662752130037252903 at 0 prev empty local range: empty
2022-09-09 19:35:30.077923+00:00 [info] <0.18488.445> starting osiris replica reader acqokxym_s1_1662752130037252903 at offset 0
2022-09-09 19:35:30.078099+00:00 [debug] <0.18488.445> sent committed offset information to the leader at <0.18479.445>
2022-09-09 19:35:30.078937+00:00 [info] <0.18477.445> rabbit_stream_coordinator: acqokxym_s1_1662752130037252903: replica started on rabbit@test-tall-wheat-goat-03 in 1 pid <46479.7769.35>

2022-09-09 19:36:13.133946+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000004.segment
2022-09-09 19:36:13.135367+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000000.segment
2022-09-09 19:36:13.135343+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000006.segment
2022-09-09 19:36:13.136630+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000009.segment
2022-09-09 19:36:13.137237+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002143s
2022-09-09 19:36:13.137772+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000006.segment
2022-09-09 19:36:13.138039+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000013.segment
2022-09-09 19:36:13.138332+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000004.segment
2022-09-09 19:36:13.139476+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000017.segment
2022-09-09 19:36:13.140407+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002888s
2022-09-09 19:36:13.140745+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000019.segment
2022-09-09 19:36:13.141057+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000013.segment
2022-09-09 19:36:13.141768+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000009.segment
2022-09-09 19:36:13.143348+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002571s
2022-09-09 19:36:13.143826+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000017.segment
2022-09-09 19:36:13.144663+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000021.segment
2022-09-09 19:36:13.144922+00:00 [debug] <0.9765.443> osiris_log: segment 00000000000000000019.segment has trailing data 115 1326
2022-09-09 19:36:13.145410+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.001816s
2022-09-09 19:36:13.145849+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000024.segment
2022-09-09 19:36:13.146101+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000019.segment
2022-09-09 19:36:13.146814+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000028.segment
2022-09-09 19:36:13.148140+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000031.segment
2022-09-09 19:36:13.148294+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002451s
2022-09-09 19:36:13.148991+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000028.segment
2022-09-09 19:36:13.149452+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000024.segment
2022-09-09 19:36:13.149968+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000034.segment
2022-09-09 19:36:13.150030+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000021.segment
2022-09-09 19:36:13.151059+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000036.segment
2022-09-09 19:36:13.152449+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000039.segment
2022-09-09 19:36:13.152731+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.004102s
2022-09-09 19:36:13.153478+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000036.segment
2022-09-09 19:36:13.154057+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000034.segment
2022-09-09 19:36:13.154536+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000031.segment
2022-09-09 19:36:13.154628+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000043.segment
2022-09-09 19:36:13.155859+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000045.segment
2022-09-09 19:36:13.155824+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002880s
2022-09-09 19:36:13.156592+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000043.segment
2022-09-09 19:36:13.157078+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000039.segment
2022-09-09 19:36:13.157655+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000048.segment
2022-09-09 19:36:13.158628+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000050.segment
2022-09-09 19:36:13.158501+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002253s
2022-09-09 19:36:13.159148+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000048.segment
2022-09-09 19:36:13.159755+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000045.segment
2022-09-09 19:36:13.160024+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000052.segment
2022-09-09 19:36:13.161015+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002174s
2022-09-09 19:36:13.161227+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000054.segment
2022-09-09 19:36:13.161502+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000050.segment
2022-09-09 19:36:13.162415+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000056.segment

2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445> ** Generic server <0.18490.445> terminating
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445> ** Last message in was {'$gen_cast',{more_data,54}}
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445> ** When Server state == {state,
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                          {osiris_log,
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                           {cfg,
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                            "/var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903",
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                            "acqokxym_s1_1662752130037252903",500000000,256000,
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                            #{},[],
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                            {write_concurrency,
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                             #Ref<0.1524210109.3311796226.198342>},
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                            undefined,#Fun<osiris_writer.4.42972666>,
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                            #Fun<osiris_log.3.43777157>},
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                           {read,data,#Ref<0.1524210109.3311796226.198279>,46,
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                            48,tcp,all},
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                           "00000000000000000045.segment",undefined,
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                           {file_descriptor,prim_file,
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                            #{handle => #Ref<0.1524210109.3079536648.171820>,
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                              owner => <0.18490.445>,r_ahead_size => 0,
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                              r_buffer =>
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                               #Ref<0.1524210109.3311796226.203606>}}},
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                          "acqokxym_s1_1662752130037252903",tcp,
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                          #Port<0.9309342>,<46471.9973.713>,<0.18479.445>,
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                          #Ref<0.1524210109.3311665154.198343>,
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                          {write_concurrency,
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                           #Ref<0.1524210109.3311796226.198342>},
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                          {osiris_replica_reader,
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                           {resource,<<"acqokxym">>,queue,<<"s1">>},
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                           "test-tall-wheat-goat-02",6398},
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                          46,47}

2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445> ** Reason for termination ==
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445> ** {{badmatch,{error,einval}},
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>     [{osiris_log,setopts,3,[{file,"src/osiris_log.erl"},{line,2164}]},
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>      {osiris_log,send_file,3,[{file,"src/osiris_log.erl"},{line,1492}]},
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>      {osiris_replica_reader,do_sendfile,1,
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                             [{file,"src/osiris_replica_reader.erl"},
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                              {line,341}]},
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>      {osiris_replica_reader,handle_cast,2,
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                             [{file,"src/osiris_replica_reader.erl"},
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>                              {line,228}]},
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>      {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,1120}]},
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>      {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1197}]},
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>      {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}
2022-09-09 19:36:13.161886+00:00 [error] <0.18490.445>
2022-09-09 19:36:13.163380+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002138s
2022-09-09 19:36:13.163844+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000059.segment
2022-09-09 19:36:13.164070+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000054.segment
2022-09-09 19:36:13.164577+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000052.segment
2022-09-09 19:36:13.164962+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000062.segment
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>   crasher:
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>     initial call: osiris_replica_reader:init/1
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>     pid: <0.18490.445>
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>     registered_name: []
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>     exception error: no match of right hand side value {error,einval}
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>       in function  osiris_log:setopts/3 (src/osiris_log.erl, line 2164)
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>       in call from osiris_log:send_file/3 (src/osiris_log.erl, line 1492)
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>       in call from osiris_replica_reader:do_sendfile/1 (src/osiris_replica_reader.erl, line 341)
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>       in call from osiris_replica_reader:handle_cast/2 (src/osiris_replica_reader.erl, line 228)
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>       in call from gen_server:try_dispatch/4 (gen_server.erl, line 1120)
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>       in call from gen_server:handle_msg/6 (gen_server.erl, line 1197)
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>     ancestors: [osiris_replica_reader_sup,osiris_sup,<0.208.0>]
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>     message_queue_len: 1
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>     messages: [{tcp_closed,#Port<0.9309342>}]
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>     links: [<46471.9973.713>,<0.9773.443>]
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>     dictionary: [{guid,{{1063673388,3375881614,690310450,1082373317},8}},
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>                   {{xtype_to_module,topic},rabbit_exchange_type_topic}]
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>     trap_exit: true
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>     status: running
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>     heap_size: 17731
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>     stack_size: 28
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>     reductions: 128745
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>   neighbours:
2022-09-09 19:36:13.164590+00:00 [error] <0.18490.445>
2022-09-09 19:36:13.166106+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000064.segment
2022-09-09 19:36:13.166508+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002724s
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>     supervisor: {local,osiris_replica_reader_sup}
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>     errorContext: child_terminated
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>     reason: {{badmatch,{error,einval}},
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>              [{osiris_log,setopts,3,[{file,"src/osiris_log.erl"},{line,2164}]},
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>               {osiris_log,send_file,3,
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>                           [{file,"src/osiris_log.erl"},{line,1492}]},
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>               {osiris_replica_reader,do_sendfile,1,
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>                                      [{file,"src/osiris_replica_reader.erl"},
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>                                       {line,341}]},
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>               {osiris_replica_reader,handle_cast,2,
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>                                      [{file,"src/osiris_replica_reader.erl"},
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>                                       {line,228}]},
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>               {gen_server,try_dispatch,4,
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>                           [{file,"gen_server.erl"},{line,1120}]},
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>               {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1197}]},
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>               {proc_lib,init_p_do_apply,3,
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>                         [{file,"proc_lib.erl"},{line,240}]}]}
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>     offender: [{pid,<0.18490.445>},
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>                {id,#Ref<46471.307943040.835977217.202990>},
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>                {mfargs,{osiris_replica_reader,start_link,undefined}},
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>                {restart_type,temporary},
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>                {significant,false},
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>                {shutdown,5000},
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>                {child_type,worker}]
2022-09-09 19:36:13.166008+00:00 [error] <0.9773.443>
2022-09-09 19:36:13.166984+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000062.segment
2022-09-09 19:36:13.167380+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000067.segment
2022-09-09 19:36:13.167372+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000059.segment
2022-09-09 19:36:13.167880+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000056.segment
2022-09-09 19:36:13.168355+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000069.segment
2022-09-09 19:36:13.169203+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000072.segment
2022-09-09 19:36:13.169327+00:00 [debug] <0.18488.445> osiris_replica_reader: 'acqokxym_s1_1662752130037252903' Socket closed. Exiting...
2022-09-09 19:36:13.169366+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002655s
2022-09-09 19:36:13.170179+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000069.segment
2022-09-09 19:36:13.170739+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000067.segment
2022-09-09 19:36:13.171075+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000064.segment
2022-09-09 19:36:13.171933+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000074.segment
2022-09-09 19:36:13.172060+00:00 [debug] <0.11717.17> rabbit_stream_coordinator: running action: 'start_replica' for acqokxym_s1_1662752130037252903 on node 'rabbit@test-tall-wheat-goat-02' in epoch 1
2022-09-09 19:36:13.172758+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002909s
2022-09-09 19:36:13.173155+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000072.segment
2022-09-09 19:36:13.173594+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000076.segment
2022-09-09 19:36:13.174538+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.001586s
2022-09-09 19:36:13.174655+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000079.segment
2022-09-09 19:36:13.175105+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000074.segment
2022-09-09 19:36:13.176054+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000081.segment
2022-09-09 19:36:13.176632+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.001846s
2022-09-09 19:36:13.176968+00:00 [warning] <0.18596.445> rabbit_stream_coordinator: Error while starting replica for acqokxym_s1_1662752130037252903 on node rabbit@test-tall-wheat-goat-02 in 1 : {{{case_c
lause,missing_file},[{osiris_replica,init,1,[{...}|...]},{gen_server,init_it,2,[...]},{gen_server,init_it,6,...},{proc_lib,init_p_do_apply,...}]},{child,undefined,[97,99,113,111|...],{osiris_replica,start
_link,[...]},temporary,false,5000,...}}
2022-09-09 19:36:13.177137+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000084.segment
2022-09-09 19:36:13.177168+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000079.segment
2022-09-09 19:36:13.177756+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000076.segment
2022-09-09 19:36:13.177777+00:00 [debug] <0.11717.17> rabbit_stream_coordinator: running action: 'start_replica' for acqokxym_s1_1662752130037252903 on node 'rabbit@test-tall-wheat-goat-03' in epoch 1
2022-09-09 19:36:13.178438+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000087.segment
2022-09-09 19:36:13.179555+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000090.segment
2022-09-09 19:36:13.179410+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002598s
2022-09-09 19:36:13.180052+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000087.segment
2022-09-09 19:36:13.180299+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000084.segment
2022-09-09 19:36:13.180454+00:00 [debug] <0.11717.17> rabbit_stream_coordinator: running action: 'start_replica' for acqokxym_s1_1662752130037252903 on node 'rabbit@test-tall-wheat-goat-02' in epoch 1
2022-09-09 19:36:13.180918+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000081.segment
2022-09-09 19:36:13.181289+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000096.segment
2022-09-09 19:36:13.182411+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002625s
2022-09-09 19:36:13.182537+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000098.segment
2022-09-09 19:36:13.183133+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000090.segment
2022-09-09 19:36:13.183480+00:00 [warning] <0.18602.445> rabbit_stream_coordinator: Error while starting replica for acqokxym_s1_1662752130037252903 on node rabbit@test-tall-wheat-goat-03 in 1 : {{{case_c
lause,missing_file},[{osiris_replica,init,1,[{...}|...]},{gen_server,init_it,2,[...]},{gen_server,init_it,6,...},{proc_lib,init_p_do_apply,...}]},{child,undefined,[97,99,113,111|...],{osiris_replica,start_link,[...]},temporary,false,5000,...}}
2022-09-09 19:36:13.184464+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.001776s
2022-09-09 19:36:13.184510+00:00 [warning] <0.18606.445> rabbit_stream_coordinator: Error while starting replica for acqokxym_s1_1662752130037252903 on node rabbit@test-tall-wheat-goat-02 in 1 : {{{case_c
lause,missing_file},[{osiris_replica,init,1,[{...}|...]},{gen_server,init_it,2,[...]},{gen_server,init_it,6,...},{proc_lib,init_p_do_apply,...}]},{child,undefined,[97,99,113,111|...],{osiris_replica,start
_link,[...]},temporary,false,5000,...}}
2022-09-09 19:36:13.184925+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000096.segment
2022-09-09 19:36:13.186103+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000101.segment
2022-09-09 19:36:13.186081+00:00 [debug] <0.11717.17> rabbit_stream_coordinator: running action: 'start_replica' for acqokxym_s1_1662752130037252903 on node 'rabbit@test-tall-wheat-goat-03' in epoch 1
2022-09-09 19:36:13.186677+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002023s
2022-09-09 19:36:13.187199+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000098.segment
2022-09-09 19:36:13.187517+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000103.segment
2022-09-09 19:36:13.187515+00:00 [debug] <0.11717.17> rabbit_stream_coordinator: running action: 'start_replica' for acqokxym_s1_1662752130037252903 on node 'rabbit@test-tall-wheat-goat-02' in epoch 1
2022-09-09 19:36:13.188724+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000107.segment
2022-09-09 19:36:13.188890+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.001980s
2022-09-09 19:36:13.189650+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000103.segment
2022-09-09 19:36:13.189804+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000112.segment
2022-09-09 19:36:13.190194+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000101.segment
2022-09-09 19:36:13.190964+00:00 [warning] <0.18603.445> rabbit_stream_coordinator: Error while starting replica for acqokxym_s1_1662752130037252903 on node rabbit@test-tall-wheat-goat-03 in 1 : {{{case_clause,missing_file},[{osiris_replica,init,1,[{...}|...]},{gen_server,init_it,2,[...]},{gen_server,init_it,6,...},{proc_lib,init_p_do_apply,...}]},{child,undefined,[97,99,113,111|...],{osiris_replica,start_link,[...]},temporary,false,5000,...}}
2022-09-09 19:36:13.191277+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000114.segment
2022-09-09 19:36:13.191734+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002341s
2022-09-09 19:36:13.192252+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000112.segment
2022-09-09 19:36:13.192583+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000107.segment
2022-09-09 19:36:13.192763+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000118.segment
2022-09-09 19:36:13.199321+00:00 [debug] <0.11717.17> rabbit_stream_coordinator: running action: 'start_replica' for acqokxym_s1_1662752130037252903 on node 'rabbit@test-tall-wheat-goat-03' in epoch 1
2022-09-09 19:36:13.199771+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.007742s
2022-09-09 19:36:13.199768+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000120.segment
2022-09-09 19:36:13.200007+00:00 [warning] <0.18605.445> rabbit_stream_coordinator: Error while starting replica for acqokxym_s1_1662752130037252903 on node rabbit@test-tall-wheat-goat-02 in 1 : {{{case_clause,missing_file},[{osiris_replica,init,1,[{...}|...]},{gen_server,init_it,2,[...]},{gen_server,init_it,6,...},{proc_lib,init_p_do_apply,...}]},{child,undefined,[97,99,113,111|...],{osiris_replica,start_link,[...]},temporary,false,5000,...}}
2022-09-09 19:36:13.200355+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000118.segment
2022-09-09 19:36:13.200768+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000114.segment
2022-09-09 19:36:13.201779+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000124.segment
2022-09-09 19:36:13.202168+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002111s
2022-09-09 19:36:13.202586+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000120.segment
2022-09-09 19:36:13.202940+00:00 [debug] <0.11717.17> rabbit_stream_coordinator: running action: 'start_replica' for acqokxym_s1_1662752130037252903 on node 'rabbit@test-tall-wheat-goat-02' in epoch 1
2022-09-09 19:36:13.203167+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000142.segment
2022-09-09 19:36:13.204756+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000146.segment
2022-09-09 19:36:13.204774+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002373s
2022-09-09 19:36:13.205403+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000142.segment
2022-09-09 19:36:13.205818+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000148.segment
2022-09-09 19:36:13.205887+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000124.segment
2022-09-09 19:36:13.207116+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000153.segment
2022-09-09 19:36:13.207777+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002734s
2022-09-09 19:36:13.208229+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000148.segment
2022-09-09 19:36:13.208444+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000156.segment
2022-09-09 19:36:13.208666+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000146.segment
2022-09-09 19:36:13.209711+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000159.segment
2022-09-09 19:36:13.209789+00:00 [warning] <0.18609.445> rabbit_stream_coordinator: Error while starting replica for acqokxym_s1_1662752130037252903 on node rabbit@test-tall-wheat-goat-02 in 1 : {{{case_clause,missing_file},[{osiris_replica,init,1,[{...}|...]},{gen_server,init_it,2,[...]},{gen_server,init_it,6,...},{proc_lib,init_p_do_apply,...}]},{child,undefined,[97,99,113,111|...],{osiris_replica,start:
2022-09-09 19:36:13.210122+00:00 [debug] <0.18611.445> trying to connect to replica at [{127,0,1,1},"test-tall-wheat-goat-03"]
2022-09-09 19:36:13.210483+00:00 [debug] <0.18611.445> trying to connect to {127,0,1,1}
2022-09-09 19:36:13.210509+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002514s
2022-09-09 19:36:13.211122+00:00 [debug] <0.18611.445> osiris replica connection refused, host:{127,0,1,1} - port: 6296
2022-09-09 19:36:13.211325+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000162.segment
2022-09-09 19:36:13.211322+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000156.segment
2022-09-09 19:36:13.211453+00:00 [debug] <0.18611.445> trying to connect to "test-tall-wheat-goat-03"
2022-09-09 19:36:13.211853+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000153.segment
2022-09-09 19:36:13.212575+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000166.segment
2022-09-09 19:36:13.212821+00:00 [debug] <0.11717.17> rabbit_stream_coordinator: running action: 'start_replica' for acqokxym_s1_1662752130037252903 on node 'rabbit@test-tall-wheat-goat-02' in epoch 1
2022-09-09 19:36:13.213343+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002563s
2022-09-09 19:36:13.214008+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000162.segment
2022-09-09 19:36:13.214355+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000159.segment
2022-09-09 19:36:13.214559+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000169.segment
2022-09-09 19:36:13.214830+00:00 [debug] <0.18611.445> successfully connected to host "test-tall-wheat-goat-03"
2022-09-09 19:36:13.215615+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002097s
2022-09-09 19:36:13.215672+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000172.segment
2022-09-09 19:36:13.216077+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000166.segment
2022-09-09 19:36:13.216781+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000176.segment
2022-09-09 19:36:13.217866+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000179.segment
2022-09-09 19:36:13.217927+00:00 [debug] <0.9765.443> osiris_log:evaluate_retention/2 ([{max_bytes,1234}]) completed in 0.002125s
2022-09-09 19:36:13.218895+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000176.segment
2022-09-09 19:36:13.219312+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000172.segment
2022-09-09 19:36:13.219609+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000182.segment
2022-09-09 19:36:13.219834+00:00 [debug] <0.18611.445> osiris_segment:init_data_reader/2 acqokxym_s1_1662752130037252903 at 124 prev empty local range: {169,175}
2022-09-09 19:36:13.220155+00:00 [debug] <0.9765.443> osiris_log: deleting segment /var/lib/rabbitmq/rabbit@test-tall-wheat-goat-01/stream/acqokxym_s1_1662752130037252903/00000000000000000169.segment
2022-09-09 19:36:13.220770+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000184.segment
2022-09-09 19:36:13.220770+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000184.segment
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>   crasher:
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>     initial call: osiris_replica_reader:init/1
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>     pid: <0.18611.445>
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>     registered_name: []
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>     exception error: no match of right hand side value
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>                      {error,{offset_out_of_range,{169,175}}}
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>       in function  osiris_replica_reader:init/1 (src/osiris_replica_reader.erl, line 166)
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>       in call from gen_server:init_it/2 (gen_server.erl, line 848)
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>       in call from gen_server:init_it/6 (gen_server.erl, line 811)
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>     ancestors: [osiris_replica_reader_sup,osiris_sup,<0.208.0>]
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>     message_queue_len: 0
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>     messages: []
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>     links: [<0.9773.443>,#Port<0.9309356>]
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>     dictionary: [{guid,{{517520751,530762943,1205467364,1424706632},5}},
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>                   {{xtype_to_module,topic},rabbit_exchange_type_topic}]
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>     trap_exit: true
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>     status: running
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>     heap_size: 6772
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>     stack_size: 28
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>     reductions: 24847
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>   neighbours:
2022-09-09 19:36:13.220371+00:00 [error] <0.18611.445>
2022-09-09 19:36:13.221234+00:00 [warning] <0.18613.445> rabbit_stream_coordinator: Error while starting replica for acqokxym_s1_1662752130037252903 on node rabbit@test-tall-wheat-goat-03 in 1 : {{{case_clause,{error,{{{badmatch,...},[...]},{child,...}}}},[{osiris_replica_reader,start,2,[{...}|...]},{osiris_replica,init,1,[...]},{gen_server,init_it,2,...},{gen_server,init_it,...},{proc_lib,...}]},{child,undefined,[97,99,113,111|...],{osiris_replica,start_link,[...]},temporary,false,5000,...}}
2022-09-09 19:36:13.221349+00:00 [warning] <0.18615.445> rabbit_stream_coordinator: Error while starting replica for acqokxym_s1_1662752130037252903 on node rabbit@test-tall-wheat-goat-02 in 1 : {{{case_c
lause,missing_file},[{osiris_replica,init,1,[{...}|...]},{gen_server,init_it,2,[...]},{gen_server,init_it,6,...},{proc_lib,init_p_do_apply,...}]},{child,undefined,[97,99,113,111|...],{osiris_replica,start_link,[...]},temporary,false,5000,...}}
2022-09-09 19:36:13.222021+00:00 [debug] <0.18479.445> osiris_log: open_new_segment acqokxym_s1_1662752130037252903: 00000000000000000190.segment