Skip to content

Instantly share code, notes, and snippets.

@wwwy3y3

wwwy3y3/ingress.yaml

Last active October 26, 2023 13:08
Show Gist options
  • Save wwwy3y3/e3d56f6c6da7c9b4ec9d9205fd5d7b39 to your computer and use it in GitHub Desktop.
Save wwwy3y3/e3d56f6c6da7c9b4ec9d9205fd5d7b39 to your computer and use it in GitHub Desktop.
Download ZIP
test
Raw
ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx-public
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
nginx.ingress.kubernetes.io/rewrite-target: /$1
labels:
app.kubernetes.io/managed-by: cannerFlowDeployer
app.kubernetes.io/part-of: default
name: vulcan-sql
namespace: default
spec:
rules:
- http:
paths:
- backend:
serviceName: vulcan-sql
servicePort: 80
path: /api/(doc.*)
pathType: ImplementationSpecific
- http:
paths:
- backend:
serviceName: vulcan-sql
servicePort: 80
path: /(api/?.*)
pathType: ImplementationSpecific
Raw
test.yaml
apiVersion: v1
kind: Service
metadata:
namespace: default
name: vulcan-sql
labels:
app.kubernetes.io/name: vulcan-sql
app.kubernetes.io/instance: vulcan-sql-outkh1
app.kubernetes.io/version: 0.1.1
app.kubernetes.io/managed-by: cannerFlowDeployer
annotations:
alb.ingress.kubernetes.io/healthcheck-path: /doc/spec
spec:
type: NodePort
ports:
- name: http
protocol: TCP
port: 80
targetPort: http
selector:
app.kubernetes.io/name: vulcan-sql
app.kubernetes.io/instance: vulcan-sql-outkh1
app.kubernetes.io/version: 0.1.1
app.kubernetes.io/managed-by: cannerFlowDeployer
@cvmdoc
Copy link

cvmdoc commented Oct 26, 2023 

SQL State  : 23502
Error Code : 0
Message    : ERROR: null value in column "creator" violates not-null constraint
  Detail: Failing row contains (c347f4f8-41b6-8f92-1d7f-23ddd065ca00, DATA_SOURCE_TABLE, {"schema": "testasgsdfg23071626", "tableType": null, "dataSource..., {"tagIds": [], "dataResourceName": "1"}, 2023-10-26 09:46:42.971+00, 2023-10-26 09:46:42.971+00, null, null).
Location   : /flyway/migration/V1_70__update_data_resource_metadata.sql (/flyway/migration/V1_70__update_data_resource_metadata.sql)
Line       : 25
Statement  : -- The purpose of this code is to fill in the missing data for previously overlooked entries that have the same table name.
-- The code inserts new rows into the `data_resource_metadata` table using data retrieved from the `data_source_tables_snapshot` table.
-- It constructs the values for the `reference_keys` and `metadata` fields based on the corresponding columns in the `data_source_tables_snapshot` table.

@cvmdoc
Copy link

cvmdoc commented Oct 26, 2023 


Caused by: org.postgresql.util.PSQLException: ERROR: null value in column "creator" violates not-null constraint
  Detail: Failing row contains (21e39868-f168-de05-808f-43c6f4529103, DATA_SOURCE_TABLE, {"schema": "companydata23031329", "tableType": "TABLE", "dataSou..., {"tagIds": [], "dataResourceName": "0420companyID"}, 2023-10-26 10:38:14.843+00, 2023-10-26 10:38:14.843+00, null, null).
grant TABLES privileges on database web to web-user

@cvmdoc
Copy link

cvmdoc commented Oct 26, 2023

Caused by: org.postgresql.util.PSQLException: ERROR: null value in column "creator" violates not-null constraint
Detail: Failing row contains (059eb88c-5654-865e-039d-140e6cc3a4d7, DATA_SOURCE_TABLE, {"schema": "companydata23031329", "tableType": "TABLE", "dataSou..., {"tagIds": [], "dataResourceName": "0420companyID"}, 2023-10-26 11:09:28.957+00, 2023-10-26 11:09:28.957+00, null, null).

@wwwy3y3
Copy link
Author

wwwy3y3 commented Oct 26, 2023 

SELECT
	md5(random()::text || clock_timestamp()::text)::uuid AS id,
	'DATA_SOURCE_TABLE' AS TYPE,
	-- reference_keys: {"schema": "public", "tableType": "TABLE", "dataSourceId": "2e597fad-8b9f-4f31-ad3a-68fa02922b89", "tableSqlName": "table_sql_name"}
	jsonb_build_object ('schema',
		data_source_tables_snapshot.schema -> 'properties' ->> 'schema',
		'tableType',
		data_source_tables_snapshot.schema ->> 'tableType',
		'dataSourceId',
		data_source_tables_snapshot.data_source_id,
		'tableSqlName',
		data_source_tables_snapshot.sql_name),
	-- metadata: {"tagIds": [], "dataResourceName": "public.table_sql_name"}
	jsonb_build_object ('tagIds',
		JSONB '[]',
		'dataResourceName',
		data_source_tables_snapshot.schema ->> 'displayName'),
	CURRENT_TIMESTAMP AS created_at,
	CURRENT_TIMESTAMP AS updated_at,
	-- This assumes that entries with the same table name are `created` by the same user and allows you to populate the `creator` column without introducing non-existent user IDs.
	-- This approach helps maintain consistency within the `data_resource_metadata` table and ensures that the `creator` column is populated with valid values.
	(
		SELECT
			creator
		FROM
			data_resource_metadata
		WHERE (reference_keys ->> 'dataSourceId')::uuid = data_source_tables_snapshot.data_source_id
		AND reference_keys ->> 'tableType' = data_source_tables_snapshot.schema ->> 'tableType'
		AND reference_keys ->> 'tableSqlName' = data_source_tables_snapshot.sql_name
		AND TYPE = 'DATA_SOURCE_TABLE'
	LIMIT 1)
FROM
	data_source_tables_snapshot
WHERE
	NOT EXISTS (
		SELECT
			1
		FROM
			data_resource_metadata
		WHERE (reference_keys ->> 'dataSourceId')::uuid = data_source_tables_snapshot.data_source_id
		AND reference_keys ->> 'tableType' = data_source_tables_snapshot.schema ->> 'tableType'
		AND reference_keys ->> 'tableSqlName' = data_source_tables_snapshot.sql_name
		AND TYPE = 'DATA_SOURCE_TABLE'
		AND metadata ->> 'dataResourceName' = data_source_tables_snapshot.schema ->> 'displayName');

@wwwy3y3
Copy link
Author

wwwy3y3 commented Oct 26, 2023 

SELECT    md5(random()::text || clock_timestamp()::text)::uuid AS id,    'DATA_SOURCE_TABLE' AS TYPE,    jsonb_build_object ('schema',        data_source_tables_snapshot.schema -> 'properties' ->> 'schema',        'tableType',        data_source_tables_snapshot.schema ->> 'tableType',        'dataSourceId',        data_source_tables_snapshot.data_source_id,        'tableSqlName',        data_source_tables_snapshot.sql_name),    jsonb_build_object ('tagIds',        JSONB '[]',        'dataResourceName',        data_source_tables_snapshot.schema ->> 'displayName'),    CURRENT_TIMESTAMP AS created_at,    CURRENT_TIMESTAMP AS updated_at,    (        SELECT            creator        FROM            data_resource_metadata        WHERE (reference_keys ->> 'dataSourceId')::uuid = data_source_tables_snapshot.data_source_id        AND reference_keys ->> 'tableType' = data_source_tables_snapshot.schema ->> 'tableType'        AND reference_keys ->> 'tableSqlName' = data_source_tables_snapshot.sql_name        AND TYPE = 'DATA_SOURCE_TABLE'    LIMIT 1)FROM    data_source_tables_snapshotWHERE    NOT EXISTS (        SELECT            1        FROM            data_resource_metadata        WHERE (reference_keys ->> 'dataSourceId')::uuid = data_source_tables_snapshot.data_source_id        AND reference_keys ->> 'tableType' = data_source_tables_snapshot.schema ->> 'tableType'        AND reference_keys ->> 'tableSqlName' = data_source_tables_snapshot.sql_name        AND TYPE = 'DATA_SOURCE_TABLE'        AND metadata ->> 'dataResourceName' = data_source_tables_snapshot.schema ->> 'displayName');

@wwwy3y3
Copy link
Author

wwwy3y3 commented Oct 26, 2023 

SELECT
    md5(random()::text || clock_timestamp()::text)::uuid AS id,
    'DATA_SOURCE_TABLE' AS TYPE,
    jsonb_build_object ('schema',
        data_source_tables_snapshot.schema -> 'properties' ->> 'schema',
        'tableType',
        data_source_tables_snapshot.schema ->> 'tableType',
        'dataSourceId',
        data_source_tables_snapshot.data_source_id,
        'tableSqlName',
        data_source_tables_snapshot.sql_name),
    jsonb_build_object ('tagIds',
        JSONB '[]',
        'dataResourceName',
        data_source_tables_snapshot.schema ->> 'displayName'),
    CURRENT_TIMESTAMP AS created_at,
    CURRENT_TIMESTAMP AS updated_at,
    (
        SELECT
            creator
        FROM
            data_resource_metadata
        WHERE (reference_keys ->> 'dataSourceId')::uuid = data_source_tables_snapshot.data_source_id
        AND reference_keys ->> 'tableType' = data_source_tables_snapshot.schema ->> 'tableType'
        AND reference_keys ->> 'tableSqlName' = data_source_tables_snapshot.sql_name
        AND TYPE = 'DATA_SOURCE_TABLE'
    LIMIT 1)
FROM
    data_source_tables_snapshot
WHERE
    NOT EXISTS (
        SELECT
            1
        FROM
            data_resource_metadata
        WHERE (reference_keys ->> 'dataSourceId')::uuid = data_source_tables_snapshot.data_source_id
        AND reference_keys ->> 'tableType' = data_source_tables_snapshot.schema ->> 'tableType'
        AND reference_keys ->> 'tableSqlName' = data_source_tables_snapshot.sql_name
        AND TYPE = 'DATA_SOURCE_TABLE'
        AND metadata ->> 'dataResourceName' = data_source_tables_snapshot.schema ->> 'displayName');

@wwwy3y3
Copy link
Author

wwwy3y3 commented Oct 26, 2023 •
edited
Loading 

- name: AUTH_CRYPTO_PASSWORD
  valueFrom:
    secretKeyRef:
      name: sql-engine-encrypt-credential
      key: key

@cvmdoc
Copy link

cvmdoc commented Oct 26, 2023

2023-10-26 12:26:54.075 INFO [ACCESS_LOG] --> ::ffff:10.42.0.174 -- "GET /api/data_engineering_63116/etest" -- size: none
2023-10-26 12:26:54.077 INFO [ACCESS_LOG] -> header: {"host":"tiumcanner01","x-request-id":"f8159c0fc15fe008a6a570900979ce29","x-real-ip":"10.42.0.1","x-forwarded-for":"10.42.0.1","x-forwarded-host":"tiumcanner01","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-scheme":"https","x-scheme":"https","authorization":"canner-pat NmQ5OTc2OWUtMjBiYy00OGViLTlkNzctMzBjN2E0MWM0NjNiX3Rlc3Q6WmFkR3Z3dmExcnBHQ2phdjNpcnF5N3Z0b01wUkxoNVk=","user-agent":"PostmanRuntime/7.34.0","accept":"/","postman-token":"b4465288-af99-498d-a618-2bb145038a5e","accept-encoding":"gzip, deflate, br"}
2023-10-26 12:26:54.078 INFO [ACCESS_LOG] -> query: {"skillid":"1"}
2023-10-26 12:26:54.079 INFO [ACCESS_LOG] -> params: undefined
2023-10-26 12:26:54.166 INFO [ACCESS_LOG] <-- status: 404 -- size: none
2023-10-26 12:26:54.167 INFO [ACCESS_LOG] <- header: {"vary":"Origin","x-ratelimit-limit":"10000","x-ratelimit-remaining":"9999","x-ratelimit-reset":"1698323275"}

@wwwy3y3
Copy link
Author

wwwy3y3 commented Oct 26, 2023 

'{"name":"vulcan-sql","description":"VulcanSQL project integrated with
    Canner Enterprise","version":"0.5.1","template":{"provider":"LocalFile","folderPath":"sqls","codeLoader":"InMemory"},"artifact":{"provider":"Canner","filePath":"fe54258b-da25-408b-a019-efb3b36f1b47"},"schema-parser":{"reader":"LocalFile","folderPath":"sqls"},"document-generator":{"specs":["oas3"]},"types":["RESTFUL"],"extensions":{"duckdb":"@vulcan-sql/extension-driver-duckdb","canner-driver":"@vulcan-sql/extension-driver-canner","canner-store":"@vulcan-sql/extension-store-canner","canner-auth":"@vulcan-sql/extension-authenticator-canner"},"activity-log":{"enabled":true,"options":{"http-logger":{"connection":{"ssl":false,"host":"web","port":9777,"path":"/api/internal/activity-log/data-service"}}}},"profiles":[{"type":"Canner","options":{"path":"fe54258b-da25-408b-a019-efb3b36f1b47"}}],"rate-limit":{"options":{"interval":{"min":1},"max":10000}},"enforce-https":{"enabled":false},"auth":{"enabled":true},"response-format":{"enabled":true,"options":{"default":"json","formats":["json","csv"]}},"canner-auth":{"options":{"canner-pat":{"host":"web","ssl":false}}},"cache":{"type":"parquet","folderPath":"tmp","loader":"duckdb"}}'

@cvmdoc
Copy link

cvmdoc commented Oct 26, 2023 


 InternalError  Failed to get workspace request "/v2/async-queries" data, remote server does not response. request [object Object]}
details:
{
  code: 'vulcan.internalError',
  nestedError: undefined,
  httpCode: 500,
  exitCode: 3,
  description: undefined,
  requestId: undefined
}
error stack:
• cannerAdapter.ts:92 <anonymous>
    packages/extension-driver-canner/src/lib/cannerAdapter.ts:92:13

• tslib.js:167 rejected
    node_modules/tslib/tslib.js:167:69

• task_queues:96 processTicksAndRejections
    node:internal/process/task_queues:96:5

@cvmdoc
Copy link

cvmdoc commented Oct 26, 2023

cp duckdb/extensions/httpfs.duckdb_extension duckdb/extensions/.duckdb/extensions/v0.7.1/linux_amd64/

@cvmdoc
Copy link

cvmdoc commented Oct 26, 2023 •
edited
Loading 

{"restfulApiBaseUrl":"https://tiumcanner01/web/api/v1","restfulApiBaseEndpoint":"https://tiumcanner01/web/api","restfulApiSupportedVersion":["v1","v2"],"cannerflowVersion":"3.0.0","cannerflowAccount":"e2f025bb-88dd-4f26-9b81-6e13f0f5b629","env":"production","oidcPublicWellKnownEndpoint":"https://tiumcanner01/sso/auth/realms/canner-skcb-testing/.well-known/openid-configuration","oidcInternalWellKnownEndpoint":"http://keycloak.default.svc.cluster.local/sso/auth/realms/canner-skcb-testing/.well-known/openid-configuration"}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment