Skip to content

Instantly share code, notes, and snippets.

View lobster1234's full-sized avatar
🎯
Focusing

Manish Pandit lobster1234

🎯
Focusing
View GitHub Profile

EC2

  • Can start an AMI to create any number of on-demand instances by using the RunInstances call. Default max is 20 per account.
  • If the request is good, the RunInstances call returns success with a list of DNS names, and instances begin to launch within 10 minutes.
  • The launch request is associated with a reservation ID. One reservation ID can map to multiple instances which are all a part of one launch request.
  • The status can be checked via DescribeInstances call
  • Can be terminated using TerminateInstances call
  • If a running instances uses EBS-backed root volume (boot partition), it can be stopped by StopInstances call. This preseves the data.
  • The same can be done via CLI or the Console instead of REST APIs
  • EBS based root device store will preserve data, hence it is not tied to the life of the instance. Local instance based store presisence is tied to the life of the instance. There is no "stop" option.
@lobster1234
lobster1234 / nutch-site.xml
Created August 16, 2017 03:44
Nutch Configuration file for MongoDB Store
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
@lobster1234
lobster1234 / gora.properties
Created August 16, 2017 03:48
Nutch 2.x gora.properties file to support MongoDB
############################
# MongoDBStore properties #
############################
gora.datastore.default=org.apache.gora.mongodb.store.MongoStore
gora.mongodb.override_hadoop_configuration=false
gora.mongodb.mapping.file=/gora-mongodb-mapping.xml
gora.mongodb.servers=localhost:27017
gora.mongodb.db=nutchdb
@lobster1234
lobster1234 / ivy.xml
Created August 16, 2017 03:55
The ivy/ivy.xml from Nutch 2.x to enable MongoDB
<?xml version="1.0" ?>
<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor
license agreements. See the NOTICE file distributed with this work for additional
information regarding copyright ownership. The ASF licenses this file to
You under the Apache License, Version 2.0 (the "License"); you may not use
this file except in compliance with the License. You may obtain a copy of
the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required
by applicable law or agreed to in writing, software distributed under the
License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
@lobster1234
lobster1234 / Dockerfile
Created August 23, 2017 01:36
Dockerfile for Solr 6.5.1
FROM java:8-jdk
WORKDIR /tmp
RUN wget http://archive.apache.org/dist/lucene/solr/6.5.1/solr-6.5.1.tgz
RUN tar xvfz solr-6.5.1.tgz
EXPOSE 8983
CMD /tmp/solr-6.5.1/bin/solr start -force -f
# docker build .
# Note the image ID
# docker run -p8983:8983 <imageid>
# point the browser to localhost:8983 for solr console
@lobster1234
lobster1234 / lambda_logging.md
Last active October 12, 2017 06:10
Logstash logging with lambdas
  • Its a pain to log events with a lambda, given that there is no server to run the agents (splunk, filebeat, etc.) on. Here is a quick and easy way to set up ELK logging by writing directly to logstash via TCP.

  • Make sure the lambda is running in the right Subnet and has the right Security Group to be able to talk to Logstash server:port.

pom.xml

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>4.11</version>
@lobster1234
lobster1234 / schema.xml
Created August 16, 2017 08:19
Fixed up schema.xml to work with Nutch 2.3.1 and Solr 6.5.1
<?xml version="1.0" encoding="UTF-8"?>
<!-- Solr managed schema - automatically generated - DO NOT EDIT -->
<schema name="nutch" version="1.5">
<uniqueKey>id</uniqueKey>
<defaultSearchField>text</defaultSearchField>
<solrQueryParser defaultOperator="OR"/>
<fieldType name="binary" class="solr.BinaryField"/>
<fieldType name="date" class="solr.TrieDateField" omitNorms="true" positionIncrementGap="0" precisionStep="0"/>
<fieldType name="double" class="solr.TrieDoubleField" omitNorms="true" positionIncrementGap="0" precisionStep="0"/>
<fieldType name="float" class="solr.TrieFloatField" omitNorms="true" positionIncrementGap="0" precisionStep="0"/>
@lobster1234
lobster1234 / ReverseString.scala
Last active February 21, 2018 22:37
Reverse a String - Scala
def reverseString(s:String) = (for(i<-s.length-1 to 0 by -1) yield s(i)).mkString
//Java Version here:
public class Test {
public final String reverse(String s) {
StringBuffer b = new StringBuffer(s.length());
for(int i = s.length()-1; i >=0; i--){
b.append(s.charAt(i));
@lobster1234
lobster1234 / merge.py
Last active September 14, 2018 16:03
Quick and Dirty utility to merge swagger.json files for microservices.
import json
from collections import OrderedDict
from operator import itemgetter
import requests
import sys
if len(sys.argv) < 2:
print("Usage : python merge.py [http(s)://path_to_swagger_json(s)]")
sys.exit(-1)
# Loop through the URLs
files = list()
@lobster1234
lobster1234 / Miner.java
Last active March 27, 2019 19:23
A miner for demo during talks/presentations
package org.lobster1234;
import org.apache.commons.codec.digest.DigestUtils;
public class Miner {
public static void main(String[] args){
Block block = new Block("185f8db32271fe25f561a6fc938b2e264306ec304eda518007d1764826381969"