public
Last active

The CouchDB replicator database

  • Download Gist
gistfile1.md
Markdown

1. Introduction to the replicator database

A database where you PUT/POST documents to trigger replications and you DELETE to cancel ongoing replications. These documents have exactly the same content as the JSON objects we used to POST to /_replicate/ (fields "source", "target", "create_target", "continuous", "doc_ids", "filter", "query_params".

Replication documents can have a user defined _id. Design documents (and _local documents) added to the replicator database are ignored.

The default name of this database is _replicator. The name can be changed in the .ini configuration, section [replicator], parameter db.

2. Basics

Let's say you PUT the following document into _replicator:

{
    "_id": "my_rep",
    "source":  "http://myserver.com:5984/foo",
    "target":  "bar",
    "create_target":  true
}

In the couch log you'll see 2 entries like these:

[Thu, 17 Feb 2011 19:43:59 GMT] [info] [<0.291.0>] Document `my_rep` triggered replication `c0ebe9256695ff083347cbf95f93e280+create_target`
[Thu, 17 Feb 2011 19:44:37 GMT] [info] [<0.124.0>] Replication `c0ebe9256695ff083347cbf95f93e280+create_target` finished (triggered by document `my_rep`)

As soon as the replication is triggered, the document will be updated by CouchDB with 3 new fields:

{
    "_id": "my_rep",
    "source":  "http://myserver.com:5984/foo",
    "target":  "bar",
    "create_target":  true,
    "_replication_id":  "c0ebe9256695ff083347cbf95f93e280",
    "_replication_state":  "triggered",
    "_replication_state_time":  1297974122
}

Note: special fields set by the replicator start with the prefix "_replication_".

  • _replication_id: the ID internally assigned to the replication. This is the ID exposed by the output from /_active_tasks/.
  • _replication_state: the current state of the replication;
  • _replication_state_time: a Unix timestamp (number of seconds since 1 Jan 1970) that tells us when the current replication state (defined in _replication_state) was set

When the replication finishes, it will update the _replication_state field (and _replication_state_time) with the value "completed", so the document will look like:

{
    "_id": "my_rep",
    "source":  "http://myserver.com:5984/foo",
    "target":  "bar",
    "create_target":  true,
    "_replication_id":  "c0ebe9256695ff083347cbf95f93e280",
    "_replication_state":  "completed",
    "_replication_state_time":  1297974122
}

When an error happens during replication, the _replication_state field is set to "error" (and _replication_state gets updated of course).

When you PUT/POST a document to the _replicator database, CouchDB will attempt to start the replication up to 10 times (configurable under [replicator], parameter max_replication_retry_count). If it fails on the first attempt, it waits 5 seconds before doing a second attempt. If the second attempt fails, it waits 10 seconds before doing a third attempt. If the third attempt fails, it waits 20 seconds before doing a fourth attempt (each attempt doubles the previous wait period). When an attempt fails, the Couch log will show you something like:

[error] [<0.149.0>] Error starting replication `67c1bb92010e7abe35d7d629635f18b6+create_target` (document `my_rep_2`): {db_not_found,<<"could not open http://myserver:5986/foo/">>

Note: the _replication_state field is only set to "error" when all the attempts were unsuccessful.

There are only 3 possible values for the _replication_state field: "triggered", "completed" and "error". Continuous replications never get their state to "completed".

3. Documents describing the same replication

Lets suppose 2 documents are added to the _replicator database in the following order:

{
    "_id": "doc_A",
    "source":  "http://myserver.com:5984/foo",
    "target":  "bar"
}

and

{
    "_id": "doc_B",
    "source":  "http://myserver.com:5984/foo",
    "target":  "bar"
}

Both describe exactly the same replication (only their _ids differ). In this case document "doc_A" triggers the replication, getting updated by CouchDB with the fields _replication_state, _replication_state_time and _replication_id, just like it was described before. Document "doc_B" however, is only updated with one field, the _replication_id so it will look like this:

{
    "_id": "doc_B",
    "source":  "http://myserver.com:5984/foo",
    "target":  "bar",
    "_replication_id":  "c0ebe9256695ff083347cbf95f93e280"
}

While document "doc_A" will look like this:

{
    "_id": "doc_A",
    "source":  "http://myserver.com:5984/foo",
    "target":  "bar",
    "_replication_id":  "c0ebe9256695ff083347cbf95f93e280",
    "_replication_state":  "triggered",
    "_replication_state_time":  1297974122
}

Note that both document get exactly the same value for the _replication_id field. This way you can identify which documents refer to the same replication - you can for example define a view which maps replication IDs to document IDs.

4. Canceling replications

To cancel a replication simply DELETE the document which triggered the replication. The Couch log will show you an entry like the following:

[Thu, 17 Feb 2011 20:16:29 GMT] [info] [<0.125.0>] Stopped replication `c0ebe9256695ff083347cbf95f93e280+continuous+create_target` because replication document `doc_A` was deleted

Note: You need to DELETE the document that triggered the replication. DELETEing another document that describes the same replication but it did not triggered it, will not cancel the replication.

5. Server restart

When CouchDB is restarted, it checks its _replicator database and restarts any replication that is described by a document that either has its _replication_state field set to "triggered" or it doesn't have yet the _replication_state field set.

Note: Continuous replications always have a _replication_state field with the value "triggered", therefore they're always restarted when CouchDB is restarted.

6. Changing the replicator database

Imagine your replicator database (default name is _replicator) has the two following documents that represent pull replications from servers A and B:

{
    "_id": "rep_from_A",
    "source":  "http://aserver.com:5984/foo",
    "target":  "foo_a",
    "continuous":  true,
    "_replication_id":  "c0ebe9256695ff083347cbf95f93e280",
    "_replication_state":  "triggered",
    "_replication_state_time":  1297971311
}
{
    "_id": "rep_from_B",
    "source":  "http://bserver.com:5984/foo",
    "target":  "foo_b",
    "continuous":  true,
    "_replication_id":  "231bb3cf9d48314eaa8d48a9170570d1",
    "_replication_state":  "triggered",
    "_replication_state_time":  1297974122
}

Now without stopping and restarting CouchDB, you change the name of the replicator database to another_replicator_db:

$ curl -vX PUT http://localhost:5984/_config/replicator/db -d '"another_replicator_db"'
"_replicator"

As soon as this is done, both pull replications defined before, are stopped. This is explicitly mentioned in CouchDB's log:

[Fri, 11 Mar 2011 07:44:20 GMT] [info] [<0.104.0>] Stopping all ongoing replications because the replicator database was deleted or changed
[Fri, 11 Mar 2011 07:44:20 GMT] [info] [<0.127.0>] 127.0.0.1 - - PUT /_config/replicator/db 200

Imagine now you add a replication document to the new replicator database named another_replicator_db:

{
    "_id": "rep_from_X",
    "source":  "http://xserver.com:5984/foo",
    "target":  "foo_x",
    "continuous":  true
}

From now own you have a single replication going on in your system: a pull replication pulling from server X. Now you change back the replicator database to the original one _replicator:

$ curl -X PUT http://localhost:5984/_config/replicator/db -d '"_replicator"'
"another_replicator_db"

Immediately after this operation, the replication pulling from server X will be stopped and the replications defined in the _replicator database (pulling from servers A and B) will be resumed.

Changing again the replicator database to another_replicator_db will stop the pull replications pulling from servers A and B, and resume the pull replication pulling from server X.

7. Replicating the replicator database

Imagine you have in server C a replicator database with the two following pull replication documents in it:

{
    "_id": "rep_from_A",
    "source":  "http://aserver.com:5984/foo",
    "target":  "foo_a",
    "continuous":  true,
    "_replication_id":  "c0ebe9256695ff083347cbf95f93e280",
    "_replication_state":  "triggered",
    "_replication_state_time":  1297971311
}
{
    "_id": "rep_from_B",
    "source":  "http://bserver.com:5984/foo",
    "target":  "foo_b",
    "continuous":  true,
    "_replication_id":  "231bb3cf9d48314eaa8d48a9170570d1",
    "_replication_state":  "triggered",
    "_replication_state_time":  1297974122
}

Now you would like to have the same pull replications going on in server D, that is, you would like to have server D pull replicating from servers A and B. You have two options:

  • Explicitly add two documents to server's D replicator database
  • Replicate server's C replicator database into server's D replicator database

Both alternatives accomplish exactly the same goal.

Hi there,

Running 1.1.0 in OSX. After deleting a replicator document couchdb crashed and now it won't start. Below is the complete trace when it crashed and after a typical start attempt.

Thanks,
-Nestor

[info] [] 127.0.0.1 - - 'GET' / 200
[info] [] 127.0.0.1 - - 'DELETE' /_replicator/by_clientId?revs_info=true 409
[info] [] 127.0.0.1 - - 'GET' /_replicator/by_clientId 200
[info] [] 127.0.0.1 - - 'DELETE' /_replicator/by_clientId?rev=2-f39c18c522709854dcc19c6456565c0a 200
[info] [] Stopped replication 1df345a6d2f41bc9849f61b51395df03+continuous+create_target because replication document by_clientId was deleted
[info] [] 127.0.0.1 - - 'POST' /_replicator 201
[info] [] Stopping all ongoing replications because the replicator database was deleted or changed
[error] [] ** Generic server terminating
** Last message in was {'EXIT',,
{shutdown,
{gen_server,call,
[,next_missing_revs,infinity]}}}
** When Server state == {state,,
{db,,,nil,
<<"1319075213275955">>,,,
{db_header,5,43,0,
{2774506,{21,0}},
{2775454,21},
{2768987,[]},
0,nil,nil,1000},
43,
{btree,,
{2774506,{21,0}},
#Fun,
#Fun,
#Fun,
#Fun},
{btree,,
{2775454,21},
#Fun,
#Fun,
#Fun,
#Fun},
{btree,,
{2768987,[]},
#Fun,
#Fun,
#Fun,nil},
43,<<"dms4">>,
"/usr/local/var/lib/couchdb/dms4.couch",[],[],
nil,
{user_ctx,null,[],undefined},
nil,1000,
[before_header,after_header,on_file_open],
false},
,,[],0,
{[],[]},
{,#Ref},
false,0,nil,[],[]}
** Reason for termination ==
** {function_clause,
[{couch_rep_reader,handle_info,
[{'EXIT',,
{shutdown,
{gen_server,call,
[,next_missing_revs,infinity]}}},
{state,,
{db,,,nil,<<"1319075213275955">>,,
,
{db_header,5,43,0,
{2774506,{21,0}},
{2775454,21},
{2768987,[]},
0,nil,nil,1000},
43,
{btree,,
{2774506,{21,0}},
#Fun,
#Fun,
#Fun,
#Fun},
{btree,,
{2775454,21},
#Fun,
#Fun,
#Fun,
#Fun},
{btree,,
{2768987,[]},
#Fun,
#Fun,
#Fun,nil},
43,<<"dms4">>,"/usr/local/var/lib/couchdb/dms4.couch",[],
[],nil,
{user_ctx,null,[],undefined},
nil,1000,
[before_header,after_header,on_file_open],
false},
,,[],0,
{[],[]},
{,#Ref},
false,0,nil,[],[]}]},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}

...

$ couchdb
Apache CouchDB 1.1.0 (LogLevel=info) is starting.
[info] [] Stopping all ongoing replications because the replicator database was deleted or changed
[error] [] ** Generic server couch_replication_manager terminating
** Last message in was {rep_db_update,
{[{<<"seq">>,3},
{<<"id">>,<<"3115e06cdcaade38e0e748d509000196">>},
{<<"changes">>,
[{[{<<"rev">>,
<<"2-b87fdc78d7e149c8027d17aef29ff279">>}]}]},
{doc,
{[{<<"_id">>,
<<"3115e06cdcaade38e0e748d509000196">>},
{<<"_rev">>,
<<"2-b87fdc78d7e149c8027d17aef29ff279">>},
{<<"id">>,<<"by_clientId">>},
{<<"source">>,<<"dms4">>},
{<<"target">>,
<<"http://bhubint.krfs.com:5984/dms4">>},
{<<"create_target">>,true},
{<<"continuous">>,true},
{<<"filter">>,
<<"replicateFilter/clientFilter">>},
{<<"_replication_state">>,<<"triggered">>},
{<<"_replication_state_time">>,
<<"2011-10-19T21:46:53-04:00">>},
{<<"_replication_id">>,
<<"0bef042c859d21c639c26b8cea151297">>}]}}]}}
** When Server state == {state,,,<<"_replicator">>,[],10}
** Reason for termination ==
** {noproc,{gen_server,call,[couch_httpd,{get,port}]}}

=ERROR REPORT==== 19-Oct-2011::22:03:17 ===
** Generic server couch_replication_manager terminating
** Last message in was {rep_db_update,
{[{<<"seq">>,3},
{<<"id">>,<<"3115e06cdcaade38e0e748d509000196">>},
{<<"changes">>,
[{[{<<"rev">>,
<<"2-b87fdc78d7e149c8027d17aef29ff279">>}]}]},
{doc,
{[{<<"_id">>,
<<"3115e06cdcaade38e0e748d509000196">>},
{<<"_rev">>,
<<"2-b87fdc78d7e149c8027d17aef29ff279">>},
{<<"id">>,<<"by_clientId">>},
{<<"source">>,<<"dms4">>},
{<<"target">>,
<<"http://bhubint.krfs.com:5984/dms4">>},
{<<"create_target">>,true},
{<<"continuous">>,true},
{<<"filter">>,
<<"replicateFilter/clientFilter">>},
{<<"_replication_state">>,<<"triggered">>},
{<<"_replication_state_time">>,
<<"2011-10-19T21:46:53-04:00">>},
{<<"_replication_id">>,
<<"0bef042c859d21c639c26b8cea151297">>}]}}]}}
** When Server state == {state,,,<<"_replicator">>,[],10}
** Reason for termination ==
** {noproc,{gen_server,call,[couch_httpd,{get,port}]}}
[error] [] {error_report,,
{,crash_report,
[[{initial_call,
{couch_replication_manager,init,['Argument__1']}},
{pid,},
{registered_name,couch_replication_manager},
{error_info,
{exit,
{noproc,{gen_server,call,[couch_httpd,{get,port}]}},
[{gen_server,terminate,6},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,
[couch_secondary_services,couch_server_sup,]},
{messages,[]},
{links,[,]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,987},
{stack_size,24},
{reductions,1513}],
[{neighbour,
[{pid,},
{registered_name,[]},
{initial_call,{couch_event_sup,init,['Argument__1']}},
{current_function,{gen_server,loop,6}},
{ancestors,
[couch_replication_manager,couch_secondary_services,
couch_server_sup,]},
{messages,[{'$gen_cast',stop}]},
{links,[,]},
{dictionary,[]},
{trap_exit,false},
{status,runnable},
{heap_size,233},
{stack_size,9},
{reductions,32}]}]]}}

=CRASH REPORT==== 19-Oct-2011::22:03:17 ===
crasher:
initial call: couch_replication_manager:init/1
pid:
registered_name: couch_replication_manager
exception exit: {noproc,{gen_server,call,[couch_httpd,{get,port}]}}
in function gen_server:terminate/6
ancestors: [couch_secondary_services,couch_server_sup,]
messages: []
links: [,]
dictionary: []
trap_exit: true
status: running
heap_size: 987
stack_size: 24
reductions: 1513
neighbours:
neighbour: [{pid,},
{registered_name,[]},
{initial_call,{couch_event_sup,init,['Argument__1']}},
{current_function,{gen_server,loop,6}},
{ancestors,[couch_replication_manager,
couch_secondary_services,couch_server_sup,
]},
{messages,[{'$gen_cast',stop}]},
{links,[,]},
{dictionary,[]},
{trap_exit,false},
{status,runnable},
{heap_size,233},
{stack_size,9},
{reductions,32}]
[error] [] {error_report,,
{,supervisor_report,
[{supervisor,{local,couch_secondary_services}},
{errorContext,child_terminated},
{reason,
{noproc,
{gen_server,call,[couch_httpd,{get,port}]}}},
{offender,
[{pid,},
{name,replication_manager},
{mfargs,
{couch_replication_manager,start_link,[]}},
{restart_type,permanent},
{shutdown,1000},
{child_type,worker}]}]}}
Apache CouchDB has started. Time to relax.

=SUPERVISOR REPORT==== 19-Oct-2011::22:03:17 ===
Supervisor: {local,couch_secondary_services}
Context: child_terminated
Reason: {noproc,{gen_server,call,[couch_httpd,{get,port}]}}
Offender: [{pid,},
{name,replication_manager},
{mfargs,{couch_replication_manager,start_link,[]}},
{restart_type,permanent},
{shutdown,1000},
{child_type,worker}]

[info] [] Apache CouchDB has started on http://127.0.0.1:5984/
[info] [] Stopping all ongoing replications because the replicator database was deleted or changed
[error] [] ** Generic server couch_replication_manager terminating
** Last message in was {rep_db_update,
{[{<<"seq">>,3},
{<<"id">>,<<"3115e06cdcaade38e0e748d509000196">>},
{<<"changes">>,
[{[{<<"rev">>,
<<"2-b87fdc78d7e149c8027d17aef29ff279">>}]}]},
{doc,
{[{<<"_id">>,
<<"3115e06cdcaade38e0e748d509000196">>},
{<<"_rev">>,
<<"2-b87fdc78d7e149c8027d17aef29ff279">>},
{<<"id">>,<<"by_clientId">>},
{<<"source">>,<<"dms4">>},
{<<"target">>,
<<"http://bhubint.krfs.com:5984/dms4">>},
{<<"create_target">>,true},
{<<"continuous">>,true},
{<<"filter">>,
<<"replicateFilter/clientFilter">>},
{<<"_replication_state">>,<<"triggered">>},
{<<"_replication_state_time">>,
<<"2011-10-19T21:46:53-04:00">>},
{<<"_replication_id">>,
<<"0bef042c859d21c639c26b8cea151297">>}]}}]}}
** When Server state == {state,,,<<"_replicator">>,[],10}
** Reason for termination ==
** {bad_return_value,{not_found,json_mismatch}}

=ERROR REPORT==== 19-Oct-2011::22:03:17 ===
** Generic server couch_replication_manager terminating
** Last message in was {rep_db_update,
{[{<<"seq">>,3},
{<<"id">>,<<"3115e06cdcaade38e0e748d509000196">>},
{<<"changes">>,
[{[{<<"rev">>,
<<"2-b87fdc78d7e149c8027d17aef29ff279">>}]}]},
{doc,
{[{<<"_id">>,
<<"3115e06cdcaade38e0e748d509000196">>},
{<<"_rev">>,
<<"2-b87fdc78d7e149c8027d17aef29ff279">>},
{<<"id">>,<<"by_clientId">>},
{<<"source">>,<<"dms4">>},
{<<"target">>,
<<"http://bhubint.krfs.com:5984/dms4">>},
{<<"create_target">>,true},
{<<"continuous">>,true},
{<<"filter">>,
<<"replicateFilter/clientFilter">>},
{<<"_replication_state">>,<<"triggered">>},
{<<"_replication_state_time">>,
<<"2011-10-19T21:46:53-04:00">>},
{<<"_replication_id">>,
<<"0bef042c859d21c639c26b8cea151297">>}]}}]}}
** When Server state == {state,,,<<"_replicator">>,[],10}
** Reason for termination ==
** {bad_return_value,{not_found,json_mismatch}}
....

Hi Nestor,

Your problem was very likely fixed by https://issues.apache.org/jira/browse/COUCHDB-1199 (it's not exclusive about filtered replications btw, it fixes several other issues).
These fixes are in CouchDB 1.1.1, which is currently in vote phase in the Apache CouchDB development mailing list, meaning it's going to be released very soon (like a week or so). You can grab the 1.1.1 release candidate and give it a try?

Hi Filipe,

I get the below error when trying to build:

nestor:couchdb nestor$ autoconf
configure.ac:61: error: possibly undefined macro: AC_DEFINE
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
configure.ac:246: error: possibly undefined macro: AC_MSG_WARN
configure.ac:313: error: possibly undefined macro: AC_CHECK_ICU
configure.ac:330: error: possibly undefined macro: AC_CHECK_CURL

Thanks,
-Nestor

Instead of running autoconf directly, try ./configure

Hi Filipe,

I was cloning from git (http://git-wip-us.apache.org/repos/asf/couchdb.git) instead of checking out from SVN as I have documented here http://thinkinginsoftware.blogspot.com/2011/10/upgrading-couchdb-in-osx.html and as that route gave me a version 1.3.0a-7c989ec-git instead of 1.1.1 I decided to checkout from the svn branch (http://svn.apache.org/repos/asf/couchdb/branches/1.1.x/)

In both cases I have errors when starting from executable but it works when invoking ./utils/run. Any idea why?

$ couchdb

=INFO REPORT==== 20-Oct-2011::12:35:12 ===
alarm_handler: {set,{{disk_almost_full,"/"},[]}}

=INFO REPORT==== 20-Oct-2011::12:35:12 ===
alarm_handler: {set,{system_memory_high_watermark,[]}}
Apache CouchDB 1.3.0a-7c989ec-git (LogLevel=info) is starting.
[error] [] {error_report,,
{,supervisor_report,
[{supervisor,{local,couch_secondary_services}},
{errorContext,start_error},
{reason,{already_started,}},
{offender,
[{pid,undefined},
{name,db_update_notifier},
{mfargs,
{couch_db_update_notifier_sup,start_link,
[]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]}]}}

=SUPERVISOR REPORT==== 20-Oct-2011::12:35:12 ===
Supervisor: {local,couch_secondary_services}
Context: start_error
Reason: {already_started,}
Offender: [{pid,undefined},
{name,db_update_notifier},
{mfargs,{couch_db_update_notifier_sup,start_link,[]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]

[error] [] {error_report,,
{,supervisor_report,
[{supervisor,{local,couch_server_sup}},
{errorContext,start_error},
{reason,shutdown},
{offender,
[{pid,undefined},
{name,couch_secondary_services},
{mfargs,{couch_secondary_sup,start_link,[]}},
{restart_type,permanent},
{shutdown,infinity},
{child_type,supervisor}]}]}}

=SUPERVISOR REPORT==== 20-Oct-2011::12:35:12 ===
Supervisor: {local,couch_server_sup}
Context: start_error
Reason: shutdown
Offender: [{pid,undefined},
{name,couch_secondary_services},
{mfargs,{couch_secondary_sup,start_link,[]}},
{restart_type,permanent},
{shutdown,infinity},
{child_type,supervisor}]

=CRASH REPORT==== 20-Oct-2011::12:35:12 ===
crasher:
initial call: application_master:init/4
pid:
registered_name: []
exception exit: {bad_return,
{{couch_app,start,
[normal,
["/usr/local/etc/couchdb/default.ini",
"/usr/local/etc/couchdb/local.ini"]]},
{'EXIT',
{{badmatch,{error,shutdown}},
[{couch_server_sup,start_server,1},
{application_master,start_it_old,4}]}}}}
in function application_master:init/4
ancestors: []
messages: [{'EXIT',,normal}]
links: [,]
dictionary: []
trap_exit: true
status: running
heap_size: 610
stack_size: 24
reductions: 146
neighbours:

=INFO REPORT==== 20-Oct-2011::12:35:12 ===
application: couch
exited: {bad_return,{{couch_app,start,
[normal,
["/usr/local/etc/couchdb/default.ini",
"/usr/local/etc/couchdb/local.ini"]]},
{'EXIT',{{badmatch,{error,shutdown}},
[{couch_server_sup,start_server,1},
{application_master,start_it_old,4}]}}}}
type: temporary
^C
nestor-krfs:couchdb1.1.x nestor$
nestor-krfs:couchdb1.1.x nestor$ ./utils/run
Apache CouchDB 1.1.1a1186848 (LogLevel=info) is starting.
Apache CouchDB has started. Time to relax.
[info] [] Apache CouchDB has started on http://127.0.0.1:5984/

Thanks!
-Nestor

On a side note I have tested replication with 1.1.1a1186848 including filters and it works great as advertised. I think I am missing a little need we currently have:

How can I ensure that documents deleted in the source are indeed deleted in the destination? That is not currently happening.

Thanks!
-Nestor

Hi,

I got my questions responded in the mailing list and I have blogged about them in http://thinkinginsoftware.blogspot.com/2011/10/upgrading-couchdb-in-osx.html and http://thinkinginsoftware.blogspot.com/2011/10/couchdb-filtered-replication.html

Thank you very much for this post. It made me get up to speed with the new replicator.

-Nestor

Hi Nestor, sorry for not replying earlier.
Glad to see all is working for you now :)

regards

Thank you Filipe!

On Sat, Oct 22, 2011 at 7:08 AM, Filipe Manana
reply@reply.github.com
wrote:

Hi Nestor, sorry for not replying earlier.
Glad to see all is working for you now :)

regards

Reply to this email directly or view it on GitHub:
https://gist.github.com/832610

hi all,

I am trying use CouchDB to implement such use scenario as below and just don't know how to do ...

Use scenario : 1) one mobile phone, one pc both running CouchDB. there is an app on the mobile phone which initiate a pull replication to get a copy of database on the PC, and there is another app on the PC that can add some filter to the database to control which part of the database can be pulled to the mobile phone. seems only the replication initiator can specify the filter , not the database source, right? or I just missed something? 2) when the replication is done, the app on the mobile phone can get a notification or callback, is it supported by current couchDB? or is there other way to do it ?

thanks.
whatsthatjn

Thank you Filipe, I think this is a great addition!

I wonder one thing though: lets say I started a continuous replication:

{
   "_id": "continuous test",
   "source": "db1",
   "target": "db2",
   "continuous": true
}

That works. But when I delete ever db1 or db2, the replication itself gets stopped, but the document in _replicator database remains. Wouldn't it be a desired behavior to have the continuous test doc be removed from _replicator database when I destroy the source or target database?

I think there is a bug with _design/doc documents. They do not get replicated when using the _replicator db, it works fine the traditional way. Is this a bug or a feature?

@gr2m: It's intentional. If there's an error (like deleting one of the endpoint databases), the replication document is not deleted. Instead it gets its "_replition_state" field set to "error". This is to allow users to know that a replication failed (eventually add views on the replicator database, etc).

As for the replication of design documents, read section 8 "The user_ctx property and delegations". The default user_ctx doesn't have an _admin role, which means design documents are not replicated - this is the principle of the least privileges by default. Automatically adding user_ctx, based on the user's user_ctx who adds the document, is not out of question (but only for a major CouchDB version, like 2.0).

So just add the following to the doc:

{
...
"user_ctx": {
"name": username, // you can omit this field if you want
"roles": ["_admin"]
}
}

What is the recommended way to restart a replication that is currently in the error state?

On Thu, Dec 1, 2011 at 6:49 PM, Don Park
reply@reply.github.com
wrote:

What is the recommended way to restart a replication that is currently in the error state?

Edit the doc and remove the field "_replication_state".
Or create another identical doc (same source, target, etc).


Reply to this email directly or view it on GitHub:
https://gist.github.com/832610

Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

it seems as if any continuous replication saved in _replicator counts against the all_dbs_active limit, is this correct? Is there anything I can do about it, beside raising the all_dbs_active limit?

My setup is that each user has an own account db with continuous replication to a central db, I thought the _replicator is awesome to assure that the replications stay alive, but no I'm not sure if this setup will work out. What do you think?

Felipe,

So far this works great. I could not believe how fast the replication took place.

I have a couple of questions.

How can a continuous replication be edited or paused for editing? When I try I get an alert that only the replicator can modify a triggered replication.

How are credentials passed to target database? I had tested with https://"user":"password"@"irisuser".iriscouch.com, but I realize that this is not secure.

How are filters used with replicator. I want to filter a design doc from replication.

Thank you.

Regards,
Brian

On Mon, Feb 13, 2012 at 12:51 AM, bsquared
reply@reply.github.com
wrote:

Felipe,

Filipe :)

So far this works great.  I could not believe how fast the replication took place.

Great!

I have a couple of questions.

How can a continuous replication be edited or paused for editing?  When I try I get an alert that only the replicator can modify a triggered replication.

The recommended way is really to create a new document and delete the
former. You can remove that restriction by editing the
validate_doc_update function in the design document of _replicator,
but the results can be unpredictable based on which fields you update
(I don't recommend it).

How are credentials passed to target database?  I had tested with https://"user":"password"@"irisuser".iriscouch.com, but I realize that this is not secure.

Correct, with 1.1.1 and all previous release it's not very secure.
This changes with the upcoming release 1.2.0. See
http://wiki.apache.org/couchdb/Replication, Replicator database
section, which mentions the changes for 1.2.0.

How are filters used with replicator.  I want to filter a design doc from replication.

Same way as with replicator. Design document with the filter function
lives in the source database. See
http://wiki.apache.org/couchdb/Replication for documentation.

Thank you.

You're welcome,

Regards,
Brian


Reply to this email directly or view it on GitHub:
https://gist.github.com/832610

Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Filipe Manana
reply@reply.github.com
writes:

How are credentials passed to target database?  I had tested with https://"user":"password"@"irisuser".iriscouch.com, but I realize that this is not secure.

Correct, with 1.1.1 and all previous release it's not very secure.
This changes with the upcoming release 1.2.0. See
http://wiki.apache.org/couchdb/Replication, Replicator database
section, which mentions the changes for 1.2.0.

I thought I was running 1.3. Must be something with ArchLinux
package. I'll have to check it out.

      couchdb version
      $ couchdb -V
      couchdb - Apache CouchDB 1.3.0a-

Thanks for your reply. It was helpful.

One more question. How does one restrict replication into a database?

Regards,
Brian

You can restrict replication to a database via the _security object
and creating users in the _users database. You'll get information and
examples for both in the wiki.

On Mon, Feb 13, 2012 at 6:09 PM, bsquared
reply@reply.github.com
wrote:

Filipe Manana
reply@reply.github.com
writes:

How are credentials passed to target database?  I had tested with https://"user":"password"@"irisuser".iriscouch.com, but I realize that this is not secure.

Correct, with 1.1.1 and all previous release it's not very secure.
This changes with the upcoming release 1.2.0. See
http://wiki.apache.org/couchdb/Replication, Replicator database
section, which mentions the changes for 1.2.0.

I thought I was running 1.3.  Must be something with ArchLinux
package.  I'll have to check it out.

         couchdb version
         $ couchdb -V
         couchdb - Apache CouchDB 1.3.0a-

Thanks for your reply.  It was helpful.

One more question.  How does one restrict replication into a database?

Regards,
Brian


Reply to this email directly or view it on GitHub:
https://gist.github.com/832610

Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Thanks for the work and docs on _replicator. I understand that the docs sent to _replicator are almost exactly like the ones you'd send to _replicate, (including query_params and filter), but it might be nice to include an example doc or two that show setting filter and query_params explicitly.

Hi,
Thanks for the great feature.
We are encountering a strange issue with this. The records in _replicator database are getting updated continously. one of the recod looks like this
{
"_id": "flight-cd0694-20130720-nbe-ams2cloud",
"_rev": "2082-41aa7aeed4404be340e3c76f8c402014",
"source": "flight-cd0694-20130720-nbe-ams",
"target": "https://xxxxx:yyyyy@CD.sync.api.abc.com/mi-master",
"continuous": true,
"filter": "Purchase/subscriptionsAndPurchases",
"_replication_state": "triggered",
"_replication_state_time": "2013-07-23T13:13:41+02:00",
"_replication_id": "0c37f5634a83b99575a182b690ff0d05"
}

The size of the db stands at 300Mb when it has only 9 records.Please help

I think it would be better to move the automatically updated fields out of the replication document (so that only the user modifies it). Instead the same information would be made available via the _active stats API.

Perhaps if the replicator encounters a failure so bad that it stops retrying, it should touch the document with an error report.

Thanks for this detailed write-up Filipe!

However, we have a tricky situation at hand in one of our cool android-apps which relies on CouchDB for peer-to-peer 'sync'.

We set up a replication record in the _replicator DB. But then, it takes a while before CouchDB comes round to actually 'triggering' our replication [which is between the CouchDB's of 2 peers!].

The issue is that we see there are other replication-records as well...(that try to replicate from the Internet to the local Couch...& vice-versa). But for our peer-to-peer replication, we rely on just WiFi.

Based on the logs, we therefore see that when we are in a WiFi-only(no Internet!) network, the Internet-based replications keep getting 'timeout's, and hence, our (more critical!) Local-to-Local p2p-Couch replication-records take a long while to get their replication_state to get changed to triggered. Even otherwise, we want our peer-to-peer replications to get 'triggered' first over the others that the app may want to trigger.

Is there any way to prioritise our replications so that CouchDB picks them up first? (Name of replication-id, etc...?)

Also, we tried using Ektorp to set-up & start our continuous Local-to-Local replications using a StdCouchDbInstance, and found that it internally POST's to /_replicate--which implies we can't track them via the standard _replicator system database.

Do replications triggered by POST'ing to the /_replicate URL take priority over those that put a record in _replicator DB?

Given the constraint that we can't go change the underlying CouchDB version now, how could we go about tweaking this..?

Would be great if you could please provide some pointers here...

BTW, we are using CouchBase Mobile version 1.20a  (maven version: 2.0.0-beta)

Thanks & regards,
~raghu

Please sign in to comment on this gist.

Something went wrong with that request. Please try again.