Skip to content

Instantly share code, notes, and snippets.

View gcsfred's full-sized avatar

Gustavo Frederico gcsfred

View GitHub Profile
@gcsfred
gcsfred / pandas_udf_sample.py
Created November 15, 2018 12:42
pandas_udf fill missing and empty string
from pyspark.sql.functions import pandas_udf
#...
# Use pandas_udf to define a Pandas UDF
@pandas_udf('string')
# Input/output are both a pandas.Series of string
def pandas_not_null(s):
return s.fillna("_NO_₦Ӑ_").replace('', '_NO_ӖӍΡṬΫ_')
@gcsfred
gcsfred / sample.py
Created November 15, 2018 12:38
OneHotEncoder
from pyspark.ml.feature import StringIndexer
from pyspark.ml.feature import OneHotEncoder
# ...
def one_hot_encode(_df, input_column, output_column):
indexer = StringIndexer(inputCol=input_column, outputCol=input_column+"_indexed", handleInvalid='skip')
_model = indexer.fit(_df)
_td = _model.transform(_df)
encoder = OneHotEncoder(inputCol=input_column+"_indexed", outputCol=output_column, dropLast=True)
@gcsfred
gcsfred / web.config
Created October 4, 2013 05:21
web.config no projeto TesteSimpleMembershipEmpty
<?xml version="1.0" encoding="utf-8"?>
<!--
Para obter mais informações sobre como configurar o aplicativo ASP.NET, visite
http://go.microsoft.com/fwlink/?LinkId=169433
-->
<configuration>
<configSections>
<!-- For more information on Entity Framework configuration, visit http://go.microsoft.com/fwlink/?LinkID=237468 -->
<section name="entityFramework" type="System.Data.Entity.Internal.ConfigFile.EntityFrameworkSection, EntityFramework, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" requirePermission="false" />
<sectionGroup name="dotNetOpenAuth" type="DotNetOpenAuth.Configuration.DotNetOpenAuthSection, DotNetOpenAuth.Core">
@gcsfred
gcsfred / AuthConfig.cs
Created October 4, 2013 04:33
AuthConfig.cs no projeto TesteSimpleMembershipEmpty
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.Web.WebPages.OAuth;
using WebMatrix.WebData;
namespace TesteSimpleMembershipEmpty
{
public static class AuthConfig