Skip to content

Instantly share code, notes, and snippets.

@lukmdo
Created March 9, 2010 14:07
Show Gist options
  • Save lukmdo/326600 to your computer and use it in GitHub Desktop.
Save lukmdo/326600 to your computer and use it in GitHub Desktop.
Gearman4Perl and in general
1. Start gearman `server` on some machine (here `worker` machine):
$ gearmand --port 8888
2. Connect a `worker` to the server
$ gearman -h IP -p 8888 -w -f test wc
$ gearman -h 127.0.0.1 -p 8888 -w -f test executable_script
3. Push a job to queue by calling:
$ gearman -h IP -p 8888 -f test "SOME DATA TO PROCESS OR ARGS"
Notes:
* Gearman does NOT spawn any `worker` processes, therefore if a number of `workers`
becomes exceeded the `client` has to wait to be served.
* Gearmand debug-info problem none of --debug=1 or -vv options of gearmand seems to work
* Gearmand server can go down and up without the need to re-attach workers to it
* If the `worker` crushes (or machine it runs on) there is the `client` that has to decide what
should be done
* If the `client` crushes then the way to manage that is:
- use a monitor service like launchd, or daemntools/supervise
http://cr.yp.to/daemontools/supervise.html
http://launchd.macosforge.org/ AND http://developer.apple.com/macosx/launchd.html
- run those `clients` periodically repeatable with check if completed in previous run
* `client` can't pass complex data structures to the `worker`.
Passed data needs to be scalar or eventually a scalar reference. In case application
forces passing some input data structure the serialization will make it possible:
- Text format:
+ at `client` JSON::XS::encode_json and than on `worker` JSON::XS::decode_json
- Bin format:
+ at `client` Storable::nfreeze (for portable image) and than Storable::thaw on `worker`
and then it can be passed as a scalar. Alternatively consider the possibility of getting the input
data by the `worker` by itself.
Additional info:
* http://assets.en.oreilly.com/1/event/27/Gearman_%20Bringing%20the%20Power%20of%20Map_Reduce%20to%20Everyday%20Applications%20Presentation.pdf
* http://www.slideshare.net/andy.sh/gearman-and-perl
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment