Skip to content

Instantly share code, notes, and snippets.

@mattio
Created September 10, 2011 12:12
Show Gist options
  • Save mattio/1208237 to your computer and use it in GitHub Desktop.
Save mattio/1208237 to your computer and use it in GitHub Desktop.
Deadlock with queue representation
(: The basic problem I'm trying to solve is that of a queue that can also updated
for priority overrides. I thought about using map:map, but this approach seemed
easier for a first cut. :)
(: If I call queue-lib:retrieve-queue() first, everything works fine. If I try to
call queue-lib:retrieve-queue() from within queue-lib:enqueue(), I'm creating
an update within an update condition, as I understand it, and I end up with a
deadlock condition. I haven't resolved this yet, so any suggestions are
welcome! :)
xquery version "1.0-ml";
module namespace queue-lib = "http://me.com/queue-lib";
declare namespace queue = "http://me.com/queue";
declare private variable $QUEUE-URI := "/queue/queue.xml";
(: Retrieves the entire queue. :)
declare function queue-lib:retrieve-queue() as element(queue:queue) {
let $queue-document := queue-lib:initialize()
return $queue-document/queue:queue
};
(: Adds a new queue:item if it does not already exist, which is checked by id. :)
declare function queue-lib:enqueue($item as element(queue:item)) {
(: REVIEW: I'm not clear why re-using retrieve-queue() is a transactional problem when no queue exists ahead of time. :)
(: But if I can't do this, I *have* to initialize the queue first, then call this second. :)
(:let $queue-document := queue-lib:retrieve-queue():)
(: This won't work if I don't call queue-lib:retrieve-queue() beforehand, outside of this function. :)
let $queue-document := fn:doc($QUEUE-URI)/element()
let $id := $item/queue:id/text()
return
if( fn:exists($queue-document/descendant::queue:item[queue:id = $id]) ) then
fn:error( xs:QName("ERROR"), fn:concat($id, " already exists in the queue. ") )
else
xdmp:node-insert-child( $queue-document, $item )
};
(: If a queue document does not already exist, create one. Return the existing or newly created document. :)
declare private function queue-lib:initialize() {
(: Transactions + functions http://developer.marklogic.com/pipermail/general/2010-July/005852.html :)
if(fn:not(fn:exists(fn:doc($QUEUE-URI)))) then
xdmp:eval(
fn:concat(
(: REVIEW: Seems like I can't pass an element in as a variable because of concat(). :)
'xdmp:document-insert("' , $QUEUE-URI , '", <queue xmlns="http://me.com/queue"/>, (), "queue")'
),
(),
<options xmlns="xdmp:eval">
<isolation>different-transaction</isolation>
<prevent-deadlocks>true</prevent-deadlocks>
</options>
)
else ()
,
xdmp:eval(
'fn:doc("/queue/queue.xml")',
(),
<options xmlns="xdmp:eval">
<isolation>different-transaction</isolation>
<prevent-deadlocks>true</prevent-deadlocks>
</options>
)
};
@damonfeldman
Copy link

This is a pretty typical problem. The short answer is to use a directory of many queue-item documents rather than a single queue-document avoid lock contention on the single queue document.

What's happening is that MarkLogic uses standard Readers/Writer locks. So any transaction can have many readers locks or one writer lock. This access pattern: [get a resource, update that resource] creates contention when two transactions come in. T1 and T2 both get read locks (no problem). T1 then tries to upgrade to a write lock and blocks because T2 also has a read lock. T2 similarly blocks waiting on T1. This is a detectable deadlock so MarkLogic rolls back and retries one transaction. You'll see these deadlocks in ErrorLog.txt if your logging level is at Debug or lower.

Using evals inside updates that lock the same document can also create an undetectable deadlock, as described in MarkLogic documentation.

@mattio
Copy link
Author

mattio commented Sep 10, 2011

Ah, I wish I had thought of directories. If my thinking is correct, I can even use the search API. Excellent. Thanks so much for the suggestion!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment