Skip to content

Instantly share code, notes, and snippets.

Avatar
💭
in the

Naoya Moritani n0531m

💭
in the
View GitHub Profile
@n0531m
n0531m / README.md
Last active Jul 18, 2021
Extract metadata (Reports/Data Srouces) from Data Studio
View README.md

Extract metadata from Data Studio

this is some example code I used to extract metadata (REPORT/DATA_SOURCE) from Data Studio.

Warning : Information provided via Data Studio API very limited. So if you are looking for ways to understand more details such as what are the constructs of a REPORT or what backend a DATA_SOURCE is connecting to, unfortunately these will not help. That said, it can still be better than nothing so that you know what assets are existing within the organization.

Extracted information can be potentially cataloged in some other services such as Data Catalog. However, that is beyond what this sample covers.

products in use

Data Studio

@n0531m
n0531m / install_basics.sh
Last active Apr 11, 2021
setting up my chromebook
View install_basics.sh
##!/bin/bash
## setting up the least to a clean linux env on a chromebook
## mostly a copy&paste from instructions found on the web, which is why the approach is not consistent...
## i might clean it up in the future but as long as it works it does the job.
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install apt-transport-https ca-certificates gnupg software-properties-common apt-utils
@n0531m
n0531m / indonesia_admin_boundaries_to_bigquery.md
Last active Feb 22, 2021
Indonesia Admin boundaries to BigQuery #bigquery #opendata #indonesia
View indonesia_admin_boundaries_to_bigquery.md
@n0531m
n0531m / bearing.sql
Last active May 24, 2020
UDF to calculate bearing in BigQuery
View bearing.sql
#
# https://en.wikipedia.org/wiki/Radian
# https://towardsdatascience.com/calculating-the-bearing-between-two-geospatial-coordinates-66203f57e4b4
CREATE TEMP FUNCTION
PI (degree float64) AS (ACOS(-1));
CREATE TEMP FUNCTION
RADIANS (degree FLOAT64) AS (degree * ACOS(-1) / 180);
CREATE TEMP FUNCTION
@n0531m
n0531m / covic19_publicdataset_locationname_fluctuation_check.sql
Last active Apr 4, 2020
covic19_publicdataset_locationname_fluctuation
View covic19_publicdataset_locationname_fluctuation_check.sql
#DECLARE # PRECISION INT64 DEFAULT 6;
DECLARE PRECISION_GEOHASH INT64 DEFAULT 2;
WITH # first table : only aggregating the rows with exact match on country_region, province_state, longitude, latitude
# also rounding lat/lng to a set precision. (defined in declard param)
# also adding a geohash value based point point
t1 AS
( SELECT CASE
WHEN country_region IN ("South Korea",
"Korea, South",
@n0531m
n0531m / listDlvmImages.sh
Created Mar 11, 2020
GCP : list deep learning VMs
View listDlvmImages.sh
#!/bin/bash
gcloud --project deeplearning-platform-release \
compute images list --no-standard-images
View getBqTempTableByJobid.sh
function getBqTempTableByJobid {
local JOBID=$1
bq show --format json -j ${JOBID} \
| jq -r '.configuration.query.destinationTable | .projectId +":"+.datasetId+"."+.tableId'
}
@n0531m
n0531m / date_range.sql
Last active Jun 11, 2021
BigQuery : Generate range of dates
View date_range.sql
SELECT day
FROM UNNEST(
GENERATE_DATE_ARRAY(DATE('2020-01-01'), DATE('2020-01-31'), INTERVAL 1 DAY)
) as day
@n0531m
n0531m / gist:3a163ae52c343731be1df8703177aa0f
Created Dec 2, 2019
show the actual rotation and not auto-rotate the image when in preview
View gist:3a163ae52c343731be1df8703177aa0f
defaults write com.apple.Preview PVImagePrintingAutoRotate 0
@n0531m
n0531m / pom.xml
Created Aug 19, 2019
postgres-with-cloud-sql-socket-factory
View pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<!--
# full credit to one of the comments here https://stackoverflow.com/questions/55880595/cant-connect-cloud-data-fusion-with-google-cloud-sql-for-postgresql
# just updated the dependency and mvn plugin versions
$ mvn clean package
-->
<modelVersion>4.0.0</modelVersion>
<groupId>com.altostrat.moritani</groupId>