Skip to content

Instantly share code, notes, and snippets.

View daf's full-sized avatar
✴️
halp

Dave Foster daf

✴️
halp
View GitHub Profile
@daf
daf / gist:6072978
Created July 24, 2013 18:06
HADS text
NESDIS ID 5530A592
NWSLI SOGM5
Location CANNON RIVER NEAR SOGN
Latitude 44 24 23
Longitude -92 55 51
HSA MPX
State MN
Owner NWSMMN
Manufacturer HA
Channel 38
@daf
daf / managed_endpoint.markdown
Last active December 16, 2015 16:19
managed endpoint writeup

Managed Endpoint API

A few months ago, we added a method (add_endpoint) to BaseService (see here) that allows you to add any additional endpoints your business logic requires to the process itself. The advantages are:

  • If the listen greenlet fails, the failure is noticed and reported up the chain to the process manager. Usually, this will mean your process will terminate.
  • You don't need to write any code for cleaning your endpoint up on service/process quit, it happens automatically.

History

Back in the day, we added start and stop methods to Subscriber endpoints. The goal of which was to make them easier to work with - just call start, and it automatically spawns a greenlet. The issue here is that if this greenlet fails, nothing ever takes notice of it - you just see the error on stdout. This leads to many problems such as consumers not terminating properly, leading to massive queue backup, and stopping bu

@daf
daf / gist:4723738
Last active December 12, 2015 05:39
backbone model attr transform
//IONUX.Models.UserRegistrationModel = Backbone.Model.extend({
IONUX.Models.UserRegistrationModel = IONUX.Models.EditableResource.extend({
url: "/userprofile/",
idAttribute: "_id",
schema: {
name: 'Text',
description: 'Text',
@daf
daf / traceback
Created February 1, 2013 17:02
runaway PD heartbeat queue
>o> traceback.print_stack(p.backend.beat_subscriber._cbthread.gr_frame)
File "./eggs/gevent-0.13.7-py2.7-linux-x86_64.egg/gevent/greenlet.py", line 390, in run
result = self._run(*self.args, **self.kwargs)
File "./extern/pyon/pyon/net/endpoint.py", line 487, in listen
m.route() # call default handler
File "./extern/pyon/pyon/net/endpoint.py", line 429, in route
self.endpoint._message_received(self.body, self.headers)
File "./extern/pyon/pyon/net/endpoint.py", line 88, in _message_received
return self.message_received(msg, headers)
File "./extern/pyon/pyon/net/endpoint.py", line 708, in message_received
http://buildbot.oceanobservatories.org:8010/builders/coi_smoke/builds/1078/steps/nose%20smoke%20tests/logs/stdio
ion.agents.instrument.test.test_gateway_to_instrument_agent.TestInstrumentAgentViaGateway.test_autosample: start time: 07:01:55 elapsed: 427.9639
ion.services.ans.test.test_workflow.TestWorkflowManagementIntegration.test_transform_workflow: start time: 07:09:03 elapsed: 274.2018
ion.services.coi.test.test_service_gateway_service.TestServiceGatewayServiceInt.test_resource_registry_operations_through_gateway: start time: 07:15:39 elapsed: 86.4866
ion.services.dm.test.test_dm_end_2_end.TestDMEnd2End.test_dm_end_2_end: start time: 07:17:06 elapsed: 369.9278
ion.services.sa.test.test_activate_instrument.TestActivateInstrumentIntegration.test_activateInstrumentSample: start time: 07:23:16 elapsed: 374.7367
http://buildbot.oceanobservatories.org:8010/builders/coi_smoke/builds/1077/steps/nose%20smoke%20tests/logs/stdio
@daf
daf / README.md
Created November 6, 2012 19:20
PL point gap chart

Hi hello

@daf
daf / log.txt
Created October 19, 2012 15:44
greenlet leak report
Greenlet leak report
Test Greenlet
=============================================================== ====================================================================================================================================================================================================================================
TestWorkflowManagementIntegration.test_transform_workflow <Greenlet at 0x103e1f910: <bound method RPCServer.listen of <pyon.net.endpoint.RPCServer object at 0x108d81710>>>
self: RPCServer: recv_name: NP (ion_test_7ddae1,notification
daf@ansible:~/dev/nitrogen$ ./src/nitrogen --help
Usage:
--head=[arg]
Select xinerama/multihead display in GUI, 0..n, -1 for full
--help, -h
Prints this help text.
--no-recurse
Do not recurse into subdirectories
--restore
Restore saved backgrounds
Queue blame report (purge: None)
Queue Test # Msg
==================================== = ===================================================================================== =====
ion_test_032024.ion.xs.ioncore.bazam + pyon.ion.test.test_exchange.TestExchangeObjectsInt.test_aa_first 0
+B ex: ion_test_032024.ion.xs.ioncore key: #
pyon.ion.test.test_exchange.TestExchangeObjectsInt.test_consume_one_message_at_a_time 36
pyon.ion.test.test_exchange.TestExchangeObjectsInt.test_middle 51
-B ex: ion_test_032024.ion.xs.ioncore key: #
pyon.ion.test.test_exchange.TestExc
@daf
daf / qb_coi_29may.txt
Created May 29, 2012 20:07
--with-queueblame report on all coi INT, tue 29 may
Queue blame report (DB: queueblame-c1e478, full: None, by_queue: False)
If 'PREV' in accesses column, indicates queue was not accessed during this test and could indicate bleed between tests.
*** The following queues still have messages or consumers! ***
Test Queue # Acc >1 Ex? # Msgs # Cnsmrs
====================================================================================================================== =================================== ======== === ====== ========
ion.services.dm.distribution.test.test_pubsub.PubSubIntTest.test_bind_exchange_subscription hi.science_data.ingestion_queue PREV T 2 0
ion.services.dm.distribution.test.test_pubsub.PubSubIntTest.test_bind_stream_subscription hi.science_data.ingestion_queue PREV T 2 0