svrc

Spark on Cloud Foundry

This document describes one means of running a simple Apache Spark cluster on Cloud Foundry. It makes heavy use of Cloud Foundry's container networking features.

You can see an example running at http://spark-ui-proxy.184.73.108.92.xip.io.

Deploy BOSH-Lite on AWS

This cluster was deployed using BOSH-Lite on AWS. Note, this Director cannot be targetted with the new BOSH CLI (see cloudfoundry-attic/bosh-lite#424), but you can use the "old" Ruby CLI just fine. You can use the new CLI for local workflows like manifest interpolation, and then the "old" CLI for remote workflows like deploying and SSH.

svrc

Spark on Cloud Foundry

This document describes one means of running a simple Apache Spark cluster on Cloud Foundry. It makes heavy use of Cloud Foundry's container networking features.

You can see an example running at http://spark-ui-proxy.184.73.108.92.xip.io.

Deploy BOSH-Lite on AWS

This cluster was deployed using BOSH-Lite on AWS. Note, this Director cannot be targetted with the new BOSH CLI (see cloudfoundry-attic/bosh-lite#424), but you can use the "old" Ruby CLI just fine. You can use the new CLI for local workflows like manifest interpolation, and then the "old" CLI for remote workflows like deploying and SSH.

svrc

Spark on Cloud Foundry

This document describes one means of running a simple Apache Spark cluster on Cloud Foundry. It makes heavy use of Cloud Foundry's container networking features.

You can see an example running at http://spark-ui-proxy.184.73.108.92.xip.io.

Deploy BOSH-Lite on AWS

This cluster was deployed using BOSH-Lite on AWS. Note, this Director cannot be targetted with the new BOSH CLI (see cloudfoundry-attic/bosh-lite#424), but you can use the "old" Ruby CLI just fine. You can use the new CLI for local workflows like manifest interpolation, and then the "old" CLI for remote workflows like deploying and SSH.

svrc

apiVersion: v1
kind: Pod
metadata:
  name: testclient
  namespace: kafka
spec:
  containers:
  - name: kafka
    image: solsson/kafka:0.11.0.0
    command:

svrc

apiVersion: v1
kind: Pod
metadata:
  name: testclient
  namespace: kafka
spec:
  containers:
  - name: kafka
    image: solsson/kafka:0.11.0.0
    command:

svrc

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: normal
provisioner: kubernetes.io/vsphere-volume

svrc

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: normal
provisioner: kubernetes.io/vsphere-volume

svrc

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: datadir-my-kafka-kafka-0
  labels:
    app: kafka
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: normal

svrc

max_connections: 50
shared_buffers: 1792MB
effective_cache_size: 5376MB
maintenance_work_mem: 448MB
checkpoint_completion_target: 0.9
wal_buffers: 16MB
default_statistics_target: 100
random_page_cost: 1.1
effective_io_concurrency: 200
work_mem: 36700kB

svrc

---
opsman-configuration:
  vsphere:
    vcenter:
      url: vcenter.example.com
      username: admin
      password: password
      datastore: exmple-ds-1
      ca_cert: certificate
      host: example-host                     # vCenter host to deploy Ops Manager in