This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def unionAll(dfs: Seq[DataFrame]): DataFrame = { | |
val spark = SparkSession.builder().getOrCreate() | |
spark.sqlContext.createDataFrame( | |
spark.sparkContext.union(dfs.map(df => df.rdd)), | |
dfs.head.schema | |
) | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Define Java environment | |
RUN mkdir /usr/lib/jvm/ | |
RUN chmod -R +x /usr/lib/jvm/ | |
COPY /install/zulu11.31.11-ca-jdk11.0.3-linux_musl_x64.tar.gz /usr/lib/jvm/ | |
RUN tar -xzvf /usr/lib/jvm/zulu11.31.11-ca-jdk11.0.3-linux_musl_x64.tar.gz -C /usr/lib/jvm/ \ | |
&& ln -s /usr/lib/jvm/zulu11.31.11-ca-jdk11.0.3-linux_musl_x64 /usr/lib/jvm/default-jvm \ | |
&& ln -s /usr/lib/jvm/default-jvm/bin/java /usr/bin/java \ | |
&& rm /usr/lib/jvm/zulu11.31.11-ca-jdk11.0.3-linux_musl_x64.tar.gz | |
ENV JAVA_HOME /usr/lib/jvm/default-jvm/ | |
ENV LD_LIBRARY_PATH /usr/lib/jvm/default-jvm/lib:/usr/lib/jvm/default-jvm/lib/server |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import requests | |
import base64 | |
DOMAIN = '' | |
TOKEN = b'' | |
user_name = "" | |
parent_name = "data-engineers" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
print('Feature Importance:') | |
featureImportances = model.stages[7].featureImportances | |
for keyPair in spark.sparkContext \ | |
.parallelize(zip(features, | |
featureImportances.toArray())) \ | |
.sortBy(lambda pair: -pair[1]) \ | |
.collect(): | |
print('{0}: {1}'.format(keyPair[0], keyPair[1])) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import re | |
p_celular = re.compile("^(0){0,1}([1-9]{2}){0,1}([7-8]|9[1-9])[0-9]{3}[0-9]{4}$") | |
print("Match no celular: ", re.search(p_celular, '022999860983')) | |
print("Não é um celular: ", re.search(p_celular, '01938735599')) | |
p_fixo = re.compile("^(0){0,1}([1-9]{2}){0,1}([2-6])[0-9]{3}[0-9]{4}$") | |
print("Match no fixo: ", re.search(p_fixo, '01938735599')) | |
print("Não é um fixo: ", re.search(p_fixo, '022999860983')) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
require('jsonlite') | |
#Get genders for a vector of first names. | |
#Usage: genders <- getGenders(c('joão', 'fábio', 'lúcia', 'rúbens')) | |
getGenders <- function (raw_names) { | |
url_api <- 'http://api.genderize.io?' | |
for(i in 1:length(raw_names)) { | |
url_api <- paste(url_api, 'name[', i - 1, ']=', | |
iconv(raw_names[i], to='ASCII//TRANSLIT'), | |
'&', |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import urllib | |
import urllib2 | |
url = '<%=odiRef.getContext("SSAS_URL")%>' | |
data = '<?xml version="1.0"?><Envelope xmlns="http://schemas.xmlsoap.org/soap/envelope/">' + | |
'<Body>' + | |
'<Execute xmlns="urn:schemas-microsoft-com:xml-analysis"><Command>' + | |
'<Batch xmlns="http://schemas.microsoft.com/analysisservices/2003/engine">' + | |
'<Parallel>' + | |
'<Process xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ddl2="http://schemas.microsoft.com/analysisservices/2003/engine/2" xmlns:ddl2_2="http://schemas.microsoft.com/analysisservices/2003/engine/2/2" xmlns:ddl100_100="http://schemas.microsoft.com/analysisservices/2008/engine/100/100" xmlns:ddl200="http://schemas.microsoft.com/analysisservices/2010/engine/200" xmlns:ddl200_200="http://schemas.microsoft.com/analysisservices/2010/engine/200/200" xmlns:ddl300="http://schemas.microsoft.com/analysisservices/2011/engine/300" xmlns:ddl300_300="http://schemas.microsoft.com/analysisservic |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
declare @dbname varchar(255); | |
DECLARE dbname_cursor CURSOR FOR | |
SELECT name--, database_id, create_date, STATE_DESC | |
FROM sys.databases | |
WHERE STATE_DESC in ('SINGLE_USER', 'SUSPECT', 'RECOVERY_PENDING'); | |
OPEN dbname_cursor | |
FETCH NEXT FROM dbname_cursor | |
INTO @dbname |