Skip to content

Instantly share code, notes, and snippets.

View lobster1234's full-sized avatar
🎯
Focusing

Manish Pandit lobster1234

🎯
Focusing
View GitHub Profile
@lobster1234
lobster1234 / lambda_logging.md
Last active October 12, 2017 06:10
Logstash logging with lambdas
  • Its a pain to log events with a lambda, given that there is no server to run the agents (splunk, filebeat, etc.) on. Here is a quick and easy way to set up ELK logging by writing directly to logstash via TCP.

  • Make sure the lambda is running in the right Subnet and has the right Security Group to be able to talk to Logstash server:port.

pom.xml

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>4.11</version>
@lobster1234
lobster1234 / Dockerfile
Created August 23, 2017 01:36
Dockerfile for Solr 6.5.1
FROM java:8-jdk
WORKDIR /tmp
RUN wget http://archive.apache.org/dist/lucene/solr/6.5.1/solr-6.5.1.tgz
RUN tar xvfz solr-6.5.1.tgz
EXPOSE 8983
CMD /tmp/solr-6.5.1/bin/solr start -force -f
# docker build .
# Note the image ID
# docker run -p8983:8983 <imageid>
# point the browser to localhost:8983 for solr console
@lobster1234
lobster1234 / schema.xml
Created August 16, 2017 08:19
Fixed up schema.xml to work with Nutch 2.3.1 and Solr 6.5.1
<?xml version="1.0" encoding="UTF-8"?>
<!-- Solr managed schema - automatically generated - DO NOT EDIT -->
<schema name="nutch" version="1.5">
<uniqueKey>id</uniqueKey>
<defaultSearchField>text</defaultSearchField>
<solrQueryParser defaultOperator="OR"/>
<fieldType name="binary" class="solr.BinaryField"/>
<fieldType name="date" class="solr.TrieDateField" omitNorms="true" positionIncrementGap="0" precisionStep="0"/>
<fieldType name="double" class="solr.TrieDoubleField" omitNorms="true" positionIncrementGap="0" precisionStep="0"/>
<fieldType name="float" class="solr.TrieFloatField" omitNorms="true" positionIncrementGap="0" precisionStep="0"/>
@lobster1234
lobster1234 / ivy.xml
Created August 16, 2017 03:55
The ivy/ivy.xml from Nutch 2.x to enable MongoDB
<?xml version="1.0" ?>
<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor
license agreements. See the NOTICE file distributed with this work for additional
information regarding copyright ownership. The ASF licenses this file to
You under the Apache License, Version 2.0 (the "License"); you may not use
this file except in compliance with the License. You may obtain a copy of
the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required
by applicable law or agreed to in writing, software distributed under the
License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
@lobster1234
lobster1234 / gora.properties
Created August 16, 2017 03:48
Nutch 2.x gora.properties file to support MongoDB
############################
# MongoDBStore properties #
############################
gora.datastore.default=org.apache.gora.mongodb.store.MongoStore
gora.mongodb.override_hadoop_configuration=false
gora.mongodb.mapping.file=/gora-mongodb-mapping.xml
gora.mongodb.servers=localhost:27017
gora.mongodb.db=nutchdb
@lobster1234
lobster1234 / nutch-site.xml
Created August 16, 2017 03:44
Nutch Configuration file for MongoDB Store
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
@lobster1234
lobster1234 / Main.java
Last active August 9, 2017 04:20
Quick Test for reflection vs. new vs. cached
package com.marqeta.mqpay;
import java.util.HashMap;
import java.util.Map;
public class Main {
public static void main(String[] args) {
System.out.println("via Reflection took " + instantiateViaReflection() + " milliseconds");
System.out.println("via New took " + instantiateAsNew() + " milliseconds");
@lobster1234
lobster1234 / Dockerfile
Created July 27, 2017 09:49
Dockerfile to run tomcat in an ubuntu container
FROM ubuntu:latest
RUN apt-get -y update && apt-get -y upgrade
RUN apt-get -y install openjdk-8-jdk wget
RUN mkdir /usr/local/tomcat
RUN wget http://www-us.apache.org/dist/tomcat/tomcat-8/v8.5.16/bin/apache-tomcat-8.5.16.tar.gz -O /tmp/tomcat.tar.gz
RUN cd /tmp && tar xvfz tomcat.tar.gz
RUN cp -Rv /tmp/apache-tomcat-8.5.16/* /usr/local/tomcat/
EXPOSE 8080
CMD /usr/local/tomcat/bin/catalina.sh run
@lobster1234
lobster1234 / config.xml
Created April 25, 2017 04:56
Hello World API Packer Jenkins Job
<?xml version='1.0' encoding='UTF-8'?>
<project>
<actions/>
<description></description>
<keepDependencies>false</keepDependencies>
<properties>
<hudson.model.ParametersDefinitionProperty>
<parameterDefinitions>
<hudson.model.StringParameterDefinition>
<name>Build</name>
@lobster1234
lobster1234 / config.xml
Created April 24, 2017 23:45
This is the config.xml for Jenkins Build of helloworld-api project
<?xml version='1.0' encoding='UTF-8'?>
<maven2-moduleset plugin="maven-plugin@2.15.1">
<actions/>
<description></description>
<keepDependencies>false</keepDependencies>
<properties/>
<scm class="hudson.plugins.git.GitSCM" plugin="git@3.2.0">
<configVersion>2</configVersion>
<userRemoteConfigs>
<hudson.plugins.git.UserRemoteConfig>