antonyc

could not start the compute node: failed to get basebackup@0/169F070 from pageserver host=neon-stress-ps-2.local port=6400 Caused by: 0: failed to iterate over archive 1: db error: ERROR: Timed out while waiting for WAL record at LSN 0/169F070 to arrive, last_record_lsn 0/16960E8 disk consistent LSN=0/16960E8 2: ERROR: Timed out while waiting for WAL record at LSN 0/169F070 to arrive, last_record_lsn 0/16960E8 disk consistent LSN=0/16960E8

antonyc

f6fde993-80e7-434c-a12a-873d95a57e8b
3bfbbc48-46da-4f14-aec9-1fb039fc1ed9
4a74a8b4-c8ca-45e3-93a7-247ed1d03381
ca9094db-efc0-49b9-a6f6-0999b841d0f2
1071c69f-af6c-48ce-8ea4-83802b0ea3d0
32069fd5-d3bf-4ed7-b9f7-2138d91cec1e
e5946b76-2c66-45a1-9b9d-c401eed6398d
2ec25980-f8a9-4a3a-aee4-cf9eba9d1765
4d87428c-a258-4886-8cbf-8b9514fa54b6
a8366b1a-752d-430a-9e96-8b18570499aa

antonyc

7bf565f3-0cd2-4cdf-9d19-cd3fdfc0e029

antonyc

 => CANCELED [go-build  6/11] RUN go mod download                                                                               458.2s
 => ERROR [node-build 10/11] RUN yarn exec webpack                                                                              192.7s
------
 > [node-build 10/11] RUN yarn exec webpack:
#34 1.465 yarn exec v1.22.5
#34 5.870 Browserslist: caniuse-lite is outdated. Please run:
#34 5.870   npx browserslist@latest --update-db
#34 5.870   Why you should do it regularly: https://github.com/browserslist/browserslist#browsers-data-updating
#34 191.1 assets by path *.js 7.95 MiB
#34 191.1   assets by status 3.23 MiB [big] 5 assets

antonyc

interface conversion: interface {} is int64, not string","request":"GET /cable HTTP/1.1
Host: console.stage.neon.tech
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3
Cache-Control: no-cache
Connection: upgrade
Cookie: _ga_18NPNJD444=GS1.1.1654084627.42.1.1654084657.0; _ga=GA1.1.1353749678.1650437855; _ga_1N4XQ6V7JN=GS1.1.1654084627.42.1.1654084657.0; _ga_DEJRSHR980=GS1.1.1654243661.29.0.1654243662.0; _ga_C9WQGCS3KL=GS1.1.1654243661.29.0.1654243662.0; _gorilla_csrf=MTY1NDE3NzMwNXxJa3BrV0dWT1JrUlhhVUp6T1N0alRUbEthM1p6ZWxoTE5VNVNVVVZ0U0drd1RrdFhjMDFoV1hScmFrMDlJZ289fNShpDqaxJsFG4nLCYqbsuP7dOiPavatoP2B7MSb0oVB; zenith=MTY1MzA0MzgyNHxEdi1CQkFFQ180SUFBUkFCRUFBQWZfLUNBQU1HYzNSeWFXNW5EQWNBQlhOMFlYUmxCbk4wY21sdVp3d2FBQmhDY20xdmRHNTBabGhFU0hGRmRESnRaak51UzJOUlBUMEdjM1J5YVc1bkRBZ0FCbWx1ZG1sMFpRWnpkSEpwYm1jTUVBQU9lbVZ1YVhSb0xYUmxZVzB0TkRJR2MzUnlhVzVuREFrQUIzVnpaWEpmYVdRRmFXNTBOalFFQkFELUFRbz18DRTTm4OoZGV3xR7pImkYdK2lgSkvCftGYzj9jv2CMR8=
Dnt: 1
Origin: https://console.st

antonyc

m ~/D/z/c/h/zenith-console  git:(feature/1244_statefulset_1_replica) ✗ helm diff upgrade zenith-console ./ --values values-stage.yaml # --atomic --wait  --timeout 10m0s
default, zenith-console, Deployment (apps) has been removed:
- # Source: zenith-console/templates/deployment.yaml
- apiVersion: apps/v1
- kind: Deployment
- metadata:
-   name: zenith-console
-   labels:
-     helm.sh/chart: zenith-console-0.1.1
-     app.kubernetes.io/name: zenith-console

antonyc

2022-05-20T19:36:01.698+0300	DEBUG	cloudbench	Will execute PGPASSWORD=mn4KaXIbZcj2 pgbench --no-vacuum --initialize --scale 1 --host rough-resonance-341607.cloud.stage.neon.tech --port 5432 --username antonyc main	{"unit": 0}
2022-05-20T19:36:10.990+0300	INFO	cloudbench	pgbench finished successfully	{"unit": 0}
2022-05-20T19:36:10.990+0300	DEBUG	cloudbench	pgbench output: "dropping old tables...\nNOTICE:  table \"pgbench_accounts\" does not exist, skipping\nNOTICE:  table \"pgbench_branches\" does not exist, skipping\nNOTICE:  table \"pgbench_history\" does not exist, skipping\nNOTICE:  table \"pgbench_tellers\" does not exist, skipping\ncreating tables...\ngenerating data (client-side)...\n100000 of 100000 tuples (100%) done (elapsed 1.49 s, remaining 0.00 s)\ncreating primary keys...\ndone in 7.94 s (drop tables 0.16 s, create tables 0.76 s, client-side generate 6.25 s, primary keys 0.77 s).\n"	{"unit": 0}
2022-05-20T19:36:10.990+0300	INFO	cloudbench	finished unit	{"unit": 0}
2022-05-20T19:36:44.467+0300	DE

antonyc

admin@zenith-us-stage-ps-2:~$ journalctl -u pageserver.service --since "2022-05-13 19:53:29" | grep -m1 "f36b7783b92cc5b06e9f76963526cea7" -A 40
May 13 19:53:29 zenith-us-stage-ps-2.local pageserver[12668]: 2022-05-13T19:53:29.714258Z  INFO load tenantconf from /storage/pageserver/data/tenants/f36b7783b92cc5b06e9f76963526cea7/config
May 13 19:53:29 zenith-us-stage-ps-2.local pageserver[12668]: 2022-05-13T19:53:29.714354Z  INFO saving tenantconf: save tenantconf to /storage/pageserver/data/tenants/f36b7783b92cc5b06e9f76963526cea7/config
May 13 19:53:29 zenith-us-stage-ps-2.local pageserver[12668]: 2022-05-13T19:53:29.715372Z  WARN loading local timeline: unrecognized filename in timeline dir: 000000067F000040020000800A00000089EF-XXX__000000008007E8C1-00000000B4940549.temp
May 13 19:53:29 zenith-us-stage-ps-2.local pageserver[12668]: 2022-05-13T19:53:29.715530Z  WARN loading local timeline: unrecognized filename in timeline dir: 000000067F000032AC000040040000000000-XXX__000000008007E8C1-00000000B4940549.temp
Ma

antonyc

m ~/D/z/c/h/zenith-console  git:(release) ✗ helm diff upgrade zenith-console ./ --values values-release.yaml
default, zenith-console, Deployment (apps) has changed:
  # Source: zenith-console/templates/deployment.yaml
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: zenith-console
    labels:
      helm.sh/chart: zenith-console-0.1.1
      app.kubernetes.io/name: zenith-console

antonyc

m ~/D/z/c/h/zenith-console  git:(main) ✗ helm diff --debug upgrade zenith-console ./ --values values-stage.yaml
Executing helm version
Executing helm get manifest zenith-console --namespace default
Executing helm template zenith-console ./ --namespace default --values values-stage.yaml --validate --is-upgrade
Executing helm get hooks zenith-console --namespace default
default, zenith-console, Deployment (apps) has changed:
  # Source: zenith-console/templates/deployment.yaml
  apiVersion: apps/v1
  kind: Deployment
  metadata: