Skip to content

Instantly share code, notes, and snippets.

@markwalkom
Created July 22, 2017 02:07
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save markwalkom/acc34ad4319965f06b5126a816086aa0 to your computer and use it in GitHub Desktop.
Logstash config to process PayPal transaction history (downloaded in CSV)
input {
stdin {}
}
filter {
csv {
columns => ["Date","Time","Time Zone","Name","Type","Status","Currency","Gross","Fee","Net","From Email Address","To Email Address","Transaction ID","Counterparty Status","Shipping address","Address Status","Item Title","Item ID","Shipping and Handling Amount","Compensation Amount","GST","Option 1 Name","Option 1 Value","Option 2 Name","Option 2 Value","Auction Site","Buyer ID","Item URL","Closing Date","Escrow ID","Invoice ID","Reference Txn ID","Invoice Number","Custom Number","Quantity","Receipt ID","Balance","Contact Phone Number"]
add_field => [ "timestamp", "%{Date} %{Time}" ]
remove_field => [ "Date", "Time", "Time Zone" ]
}
date {
match => [ "timestamp", "dd/MMM/yyyy HH:mm:ss" ]
timezone => "Australia/Sydney"
remove_field => [ "timestamp" ]
}
mutate {
convert => {
"Gross" => integer
"Net" => integer
"GST" => integer
"Shipping and Handling Amount" => integer
"Fee" => integer
"Balance" => integer
"Quantity" => integer
}
}
}
output {
elasticsearch {
hosts => ["http://localhost:9201"]
user => "elastic"
password => "changeme"
index => "paypal-fy-%{yyyy}"
}
}
@markwalkom
Copy link
Author

markwalkom commented Jul 22, 2017

Improvements;

  • Merge applicable Authorisation + PayPal Express Checkout Payment Sent into a single transaction
  • Figure out if the various $ amounts can be converted into the account's main currency
  • Use the APIs instead of a CSV download

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment