Skip to content

Instantly share code, notes, and snippets.

View trisberg's full-sized avatar

Thomas Risberg trisberg

543 followers · 1 following
View GitHub Profile
@trisberg
trisberg / scdf-shell-roles
Last active April 18, 2017 13:53
if (securityInfo.isAuthenticated()) {
final List<String> roles = securityInfo.getRoles();
if (!securityInfo.isAuthorizationEnabled() && roles.isEmpty()) {
roles.add("No roles assigned");
}
modelBuilder.addRow().addValue("Roles").addValue(roles);
rowsWithThinSeparators.add(rowIndex++);
}
@trisberg
trisberg / spring-hadoop-test-notes
Created June 19, 2017 16:16
Spring for Apache Hadoop
test notes:
export GRADLE_OPTS="-Xmx1024m -Xms256m -XX:MaxPermSize=512m"
(export GRADLE_OPTS="-Xmx1024m -Xms256m -XX:MaxPermSize=512m -Djava.awt.headless=true")
./gradlew clean build -Pdistro=cdh5 -x test
./gradlew clean build -Ptest.forkEvery=1 -Phd.fs=hdfs://borneo:8020 -Phd.rm=borneo:8032 -Phd.jh=borneo:10020
@trisberg
trisberg / cloudera-mr-job.log
Created June 22, 2017 16:48
2017-06-22 09:26:27,933 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1498143912858_0016_000001
2017-06-22 09:26:28,706 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2017-06-22 09:26:28,706 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@2556af33)
2017-06-22 09:26:29,249 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2017-06-22 09:26:30,461 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
2017-06-22 09:26:30,546 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-06-22 09:26:30,549 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2017-06-22 09:26:30,600 INFO [main]
@trisberg
trisberg / hadoop-site-cdh5.xml
Created June 23, 2017 01:52
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>yarn.application.classpath</name>
<value>$HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*,$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$MR2_CLASSPATH</value>
@trisberg
trisberg / hdp26-testing.txt
Created June 23, 2017 12:41
Hortonworks
-----------
cd ~/Developer/HDP
./start_sandbox-hdp.sh
ssh root@localshot -p 2222
su - hdfs
@trisberg
trisberg / kafka.yaml
Last active August 9, 2017 21:27
apiVersion: v1
kind: Service
metadata:
name: kafka
labels:
app: kafka
component: kafka-broker
spec:
ports:
- port: 9092
@trisberg
trisberg / cf-papertrail.log
Created August 7, 2017 14:10
UnknownError_10001.log
Aug 06 18:03:52 scdf-ci.scdf-acceptance-test.scdf-server 83a38855-4ea3-4d4d-bbc8-6ef40308c1f7/[APP/PROC/WEB/0]: 2017-08-07 01:03:52.243 INFO 20 --- [Pool-2-worker-3] o.s.c.d.s.c.r.LRUCleaningResourceLoader : Free Disk Space = 77.69%, Target Free Space >25.00%
Aug 06 18:03:52 scdf-ci.scdf-acceptance-test.scdf-server 83a38855-4ea3-4d4d-bbc8-6ef40308c1f7/[APP/PROC/WEB/0]: 2017-08-07 01:03:52.435 INFO 20 --- [nio-8080-exec-6] o.s.c.d.s.c.r.LRUCleaningResourceLoader : Free Disk Space = 77.91%, Target Free Space >25.00%
Aug 06 18:03:52 scdf-ci.scdf-acceptance-test.scdf-server 83a38855-4ea3-4d4d-bbc8-6ef40308c1f7/[APP/PROC/WEB/0]: 2017-08-07 01:03:52.448 INFO 20 --- [Pool-2-worker-2] o.s.c.d.s.c.r.LRUCleaningResourceLoader : Free Disk Space = 77.92%, Target Free Space >25.00%
Aug 06 18:03:52 scdf-ci.scdf-acceptance-test.scdf-server 83a38855-4ea3-4d4d-bbc8-6ef40308c1f7/[APP/PROC/WEB/0]: 2017-08-07 01:03:52.533 INFO 20 --- [Pool-2-worker-1] o.s.c.d.s.c.AppRegistryController : Eagerly fetching maven://
@trisberg
trisberg / uppercase-notes.adoc
Last active September 13, 2019 22:24
Uppercase Processor Notes

Uppercase Processor Notes

Create the project using Spring Initializr, unzip it and open it up in your IDE (we are selecting the cloud-stream-binder-kafka dependency):

curl https://start.spring.io/starter.zip -d dependencies=cloud-stream-binder-kafka \
     -d groupId=com.springdeveloper.stream -d artifactId=uppercase-processor-kafka \
     -d packageName=com.springdeveloper.stream -d name=uppercase-processor-kafka \
     -d baseDir=uppercase-processor-kafka -o uppercase-processor-kafka.zip
unzip uppercase-processor-kafka.zip
@trisberg
trisberg / scdf-helm-chart.adoc
Created August 31, 2017 14:29
Simplified installation of Data Flow for Kubernetes

Simplified installation of Data Flow for Kubernetes

With the new Helm chart for Spring Cloud Data Flow for Kubernetes, we have a much simpler way of installing the software.

Helm is a package manager for Kubernetes, similar to apt, yum or homebrew that you might be familiar with. It is very easy to install and it greatly simplifies installation of an application and its dependencies into your Kubernetes cluster. The application package contents and configuration is defined in a chart and when you install it you can override any default configuration values. Helm will install any required services in addition to the ones defined in the chart. For Spring Cloud Data Flow we have three required services — MySQL and Redis are used as the stores for Spring Cloud Data Flow state and RabbitMQ is used for the pipelines' messaging layer.

Once we have the He

@trisberg
trisberg / topic-gateway.log
Created September 27, 2017 18:25
Sleuth Logs from Topic Gateway app
Aruba:sk8s trisberg$ k logs -f po/topic-gateway-4270803346-l85qx
2017-09-27 18:19:59.314 INFO [topic-gateway,,,] 1 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@182decdb: startup date [Wed Sep 27 18:19:59 GMT 2017]; root of context hierarchy
2017-09-27 18:20:00.353 INFO [topic-gateway,,,] 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'configurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$26ff84e8] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/