Skip to content

Instantly share code, notes, and snippets.

View mgpradeepa's full-sized avatar
:octocat:

mgpradeepa mgpradeepa

:octocat:
View GitHub Profile
@mgpradeepa
mgpradeepa / Parallelism in map-reduce.md
Last active August 7, 2018 04:42
Think on parallelism

In the distributed world, while writing map-reduce codes, there are many situations where the input data seems to be non partitiionable. In which case all the data though would be picked up by multiple mappers, it gets mapped to the same key. Once all the mappers are run and if we end up in having one key with huge list of values, then we would be burdening the reducers. When I say burdening reducers it invovles burdening all the steps after mapper till the data enters the reducer nodes.

Lets take an example to deal more on this situation and see how this can be resolved.

Challenge: Finding of average of natural numbers.

Bird-view:

@mgpradeepa
mgpradeepa / Spring_Integration_handbook.md
Created October 10, 2016 06:58
Spring Integration handbook from ref manual

Spring Integration framework.

Mainly focusses on the Enterprise Integration Application Framework. Design patterns are obtained and utilized as a integration solution from those.

One of the component is Message

Message is a generic wrapper for a java object combined with the metadata used by the framework. Payload & Headers are the key components of a message. Payload:- Can be of anything and any type data. Headers:-
Holds commonly required information like id, time_stamp, correlation_id and return address.

Create user
Login as root
$ sudo su -
$ useradd -m <userName> -p <password>
Add to the group
$ usermod -aG <someGroup> <userName>
For granting sudo or root access
visudo
@mgpradeepa
mgpradeepa / Kafka_setup_debug
Last active March 14, 2023 08:22
Kafka how to configure and debug when in need what to use to check.
OS: Centos
Requisite: JAVA 7+8
sudo vi /etc/profile
export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk
export JRE_HOME=/usr/lib/jvm/jre
Download Kafka from any of the given website: the one use here is kafka_2.11-0.10.1.1.tgz
Untar: tar -xvf kafka_2.11-0.10.1.1.tgz
sudo mv kafka_2.11-0.10.1.1.tgz /opt
@mgpradeepa
mgpradeepa / install_monaco_font.sh
Created April 14, 2019 08:15 — forked from rogerleite/install_monaco_font.sh
Install Monaco font in Linux
#!/bin/bash
#script extraido de: http://paulocassiano.wordpress.com/2008/08/29/deixando-o-gedit-com-a-cara-do-textmate/
#tip for better "resolution" here: http://blog.siverti.com.br/2008/05/22/fonte-monaco-no-ubuntugedit/
cd /usr/share/fonts/truetype/
#TODO: put validation if folder already exists
sudo mkdir ttf-monaco
Single responsibility to single method
No more than 10 lines of code in a method.
when ever you see extract it right away
have your code comments on the way
use shortcuts on your side
IntelliJ is the best ide to slide
move the responsibility to the classes of its own
The docker system in which docker is installed or running can run into dangling state.
To check whats the disk consumption due to Docker
docker system df
Primarily the issues comes up is with low disk space. Of course this is re-claimable.
Cause for Low disk space are
(a) docker images created during build could be in dangling state
or would have been stored or persisted for a longer time.
In WINDOWS machines use the following steps
Start putty gen tool in windows
Look at ACTIONS section and choose "Load" button
Browse through the folders to pull the ppk file fo which you need to create a pem file
Key passphrase is optional and cick YES to ignore
From the menu , choose Conversions --> Drop down --> Export OpenSSH Key.
Just say 'Yes' for warning of not entering passphrase
Save the file as <xyz>.pem
@mgpradeepa
mgpradeepa / kafka_handson.md
Last active July 7, 2023 05:59
understanding about Kafka

Apache kafka

Platform for real-time distributed streaming which just behaves like any other MESSAGING system

Design goals

  • Scalability
  • High Volume
  • Data Transformations
  • Low Latency
  • Fault Tolerant
@mgpradeepa
mgpradeepa / boot_directory_fullpop.md
Last active May 13, 2022 11:47
Its obvious to encounter boot space filled. Steps to clear it

** /boot folder gets filled up as and when upgrades are done. How to free it up **

Lets check by auto remove

sudo apt autoremove If the above does not remove old ones lets first check the current kernel thats being used

uname -r

Lists the current used kernel version. Lets remove the earlier versions of the kernel upgrades