Ceph experiment

ceph_setup.md

Ubuntu install of Ceph:

sudo apt-get install ceph

ceph.conf (based off the sample config):

;
; This file defines cluster membership, the various locations
; that Ceph stores data, and any other runtime options.

; If a 'host' is defined for a daemon, the start/stop script will
; verify that it matches the hostname (or else ignore it).  If it is
; not defined, it is assumed that the daemon is intended to start on
; the current host (e.g., in a setup with a startup.conf on each
; node).

; global
[global]
	; enable secure authentication
	auth supported = cephx

; monitors
;  You need at least one.  You need at least three if you want to
;  tolerate any node failures.  Always create an odd number.
[mon]
	mon data = /home/tcoffin/ceph/mon$id

	; logging, for debugging monitor crashes, in order of
	; their likelihood of being helpful :)
	;debug ms = 1
	;debug mon = 20
	;debug paxos = 20
	;debug auth = 20

[mon0]
	host = localhost
	mon addr = 127.0.0.1:6789

;[mon1]
;	host = beta
;	mon addr = 192.168.0.11:6789
;
;[mon2]
;	host = gamma
;	mon addr = 192.168.0.12:6789

; mds
;  You need at least one.  Define two to get a standby.
[mds]
	; where the mds keeps it's secret encryption keys
	keyring = /home/tcoffin/ceph/keyring.$name

	; mds logging to debug issues.
	;debug ms = 1
	;debug mds = 20

[mds.localhost]
	host = localhost

;[mds.beta]
;	host = beta

; osd
;  You need at least one.  Two if you want data to be replicated.
;  Define as many as you like.
[osd]
	; This is where the btrfs volume will be mounted.
	osd data = /home/tcoffin/ceph/osd$id

	; Ideally, make this a separate disk or partition.  A few
 	; hundred MB should be enough; more if you have fast or many
 	; disks.  You can use a file under the osd data dir if need be
 	; (e.g. /data/osd$id/journal), but it will be slower than a
 	; separate disk or partition.

        ; This is an example of a file-based journal.
	osd journal = /home/tcoffin/ceph/osd$id/journal
	osd journal size = 100 ; journal size, in megabytes

	; osd logging to debug osd issues, in order of likelihood of being
	; helpful
	;debug ms = 1
	;debug osd = 20
	;debug filestore = 20
	;debug journal = 20

[osd0]
	host = localhost

	; if 'btrfs devs' is not specified, you're responsible for
	; setting up the 'osd data' dir.  if it is not btrfs, things
	; will behave up until you try to recover from a crash (which
	; usually fine for basic testing).
	btrfs devs = /home/tcoffin/ceph/dev/sdx

;[osd1]
;	host = epsilon
;	btrfs devs = /dev/sdy
;
;[osd2]
;	host = zeta
;	btrfs devs = /dev/sdx
;
;[osd3]
;	host = eta
;	btrfs devs = /dev/sdy

Will need to add user_xattr to the options mount point for the osd (in this case I'm just using my root disk)

Make a fake btrfs disk:

dd if=/dev/zero of=/home/tcoffin/ceph/dev/sdx bs=1G count=1

initialize

mkcephfs -c /etc/ceph/ceph.conf -v -a -k /home/tcoffin/ceph/keyring