Skip to content

Instantly share code, notes, and snippets.

@marianoguerra

marianoguerra/README.md

Created May 27, 2018 18:53
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save marianoguerra/fca40072f9b6788b46ffd8ba6e9481af to your computer and use it in GitHub Desktop.
Save marianoguerra/fca40072f9b6788b46ffd8ba6e9481af to your computer and use it in GitHub Desktop.
Download ZIP
riak_core_lite metadata refactor notes
Raw
README.md

Files to delete:

git rm src/hashtree.erl
git rm src/hashtree_tree.erl
git rm src/riak_core_aae_vnode.erl
git rm src/riak_core_broadcast.erl
git rm src/riak_core_broadcast_handler.erl
git rm src/riak_core_entropy_info.erl
git rm src/riak_core_entropy_manager.erl
git rm src/riak_core_exchange_fsm.erl
git rm src/riak_core_index_hashtree.erl
git rm src/riak_core_metadata.erl
git rm src/riak_core_metadata_evt_sup.erl
git rm src/riak_core_metadata_exchange_fsm.erl
git rm src/riak_core_metadata_hashtree.erl
git rm src/riak_core_metadata_manager.erl
git rm src/riak_core_metadata_object.erl
git rm test/bprops_eqc.erl
git rm test/bucket_eqc_utils.erl
git rm test/bucket_fixup_test.erl

Changes:

  • remove riak_core_metadata*, riak_core_broadcast, riak_core_gossip from riak_core_sup

  • remove riak_core_console:timezone

  • remove eleveldb dep on rebar.config

  • remove eleveldb override on rebar.config

  • remove eleveldb on riak_core.app.src

Remaining modules referencing riak_core_metadata related modules

egrep -n "(hashtree|hashtree_tree|riak_core_aae_vnode|riak_core_broadcast|riak_core_broadcast_handler|riak_core_entropy_info|riak_core_entropy_manager|riak_core_exchange_fsm|riak_core_index_hashtree|riak_core_metadata|riak_core_metadata_evt_sup|riak_core_metadata_exchange_fsm|riak_core_metadata_hashtree|riak_core_metadata_manager|riak_core_metadata_object):" src/*

By Function:

  • riak_core_metadata:fold/4, 1

  • riak_core_metadata:get/3, 2

  • riak_core_metadata:put/3, 4

  • riak_core_metadata:delete/2, 1

  • riak_core_metadata:iterator/0, 2

  • riak_core_metadata:iterator/3, 1

  • riak_core_metadata:itr_close/1, 1

  • riak_core_metadata:itr_done/1, 1

  • riak_core_metadata:itr_key_values/1, 1

  • riak_core_metadata:itr_next/1, 1

By Caller

  • riak_core_bucket, 4
  • riak_core_bucket_type, 11
  • riak_core_claimant, 6

Referenced Types

spec iterator() -> riak_core_metadata:iterator().
spec itr_next(riak_core_metadata:iterator()) ->
spec itr_done(riak_core_metadata:iterator()) -> boolean().
spec itr_value(riak_core_metadata:iterator()) ->
spec itr_close(riak_core_metadata:iterator()) -> ok.
spec bucket_type_iterator() -> riak_core_metadata:iterator().

Raw:

src/riak_core_bucket.erl:92:    riak_core_metadata:put(?METADATA_PREFIX, bucket_key(Bucket), BucketMeta).
src/riak_core_bucket.erl:124:    BucketMeta = riak_core_metadata:get(?METADATA_PREFIX, bucket_key(Bucket),
src/riak_core_bucket.erl:167:    riak_core_metadata:delete(?METADATA_PREFIX, bucket_key(Bucket));
src/riak_core_bucket.erl:182:    MetadataBuckets = riak_core_metadata:fold(fun({_, undefined}, Acc) ->
src/riak_core_bucket_type.erl:275:    riak_core_metadata:iterator(),
src/riak_core_bucket_type.erl:292:-spec iterator() -> riak_core_metadata:iterator().
src/riak_core_bucket_type.erl:298:-spec itr_next(riak_core_metadata:iterator()) ->
src/riak_core_bucket_type.erl:299:                      riak_core_metadata:iterator().
src/riak_core_bucket_type.erl:301:    riak_core_metadata:itr_next(It).
src/riak_core_bucket_type.erl:304:-spec itr_done(riak_core_metadata:iterator()) -> boolean().
src/riak_core_bucket_type.erl:306:    riak_core_metadata:itr_done(It).
src/riak_core_bucket_type.erl:310:-spec itr_value(riak_core_metadata:iterator()) ->
src/riak_core_bucket_type.erl:313:    {BucketType, Props} = riak_core_metadata:itr_key_values(It),
src/riak_core_bucket_type.erl:317:-spec itr_close(riak_core_metadata:iterator()) -> ok.
src/riak_core_bucket_type.erl:319:    riak_core_metadata:itr_close(It).
src/riak_core_claimant.erl:203:    case riak_core_metadata:get(?BUCKET_TYPE_PREFIX, BucketType,
src/riak_core_claimant.erl:216:-spec bucket_type_iterator() -> riak_core_metadata:iterator().
src/riak_core_claimant.erl:218:    riak_core_metadata:iterator(?BUCKET_TYPE_PREFIX, [{default, undefined},
src/riak_core_claimant.erl:285:            riak_core_metadata:put(?BUCKET_TYPE_PREFIX, BucketType, ClaimedProps),
src/riak_core_claimant.erl:296:            riak_core_metadata:put(?BUCKET_TYPE_PREFIX, BucketType, MergedProps),
src/riak_core_claimant.erl:770:    riak_core_metadata:put(?BUCKET_TYPE_PREFIX, BucketType, ActiveProps).

Remaining modules referencing caller modules that call riak_core_metadata related modules

egrep -n "(riak_core_bucket|riak_core_bucket_type|riak_core_claimant):" src/*

By Caller

      1 riak_core_app % riak_core_bucket:append_bucket_defaults
      1 riak_core_handoff_manager % riak_core_bucket:default_object_nval
      1 riak_core_ring % riak_core_claimant:ring_changed
      3 riak_core_apl % riak_core_bucket:get_bucket/1 (and 2 types)
      3 riak_core_util % riak_core_bucket:get_bucket/1
      4 riak_core_ring_manager %riak_core_bucket:get_bucket,merge_props

	  % modules to refactor
      7 riak_core_bucket
      1 riak_core_bucket_props
     18 riak_core_bucket_type
     12 riak_core_claimant
      1 riak_core_claim_sim % riak_core_claimant:reassign_indices

	  % easy to refactor
      9 riak_core_console

Notes

The pluggable interface for buckets would have to implement

  • on_start: called by riak_core_app

    • riak_core_bucket:append_bucket_defaults would be calledhere

  • get_bucket

  • get_default_val(AttrName, DefaultValue): returns default values for attributes (should always receive a default to allow dummy implementations that just return that)

  • on_ring_changed: called by riak_core_ring

Raw

src/riak_core_apl.erl:109:-spec get_apl_ann(riak_core_bucket:bucket(), [node()]) -> preflist_ann().
src/riak_core_apl.erl:111:    BucketProps = riak_core_bucket:get_bucket(Bucket),
src/riak_core_apl.erl:119:-spec get_apl_ann_with_pnum(riak_core_bucket:bucket()) -> preflist_with_pnum_ann().
src/riak_core_app.erl:81:    riak_core_bucket:append_bucket_defaults(riak_core_bucket_type:defaults(default_type)),
src/riak_core_bucket.erl:47:-type bucket_type()  :: riak_core_bucket_type:bucket_type().
src/riak_core_bucket.erl:71:    case riak_core_bucket_type:get(Type) of
src/riak_core_bucket.erl:123:    TypeMeta = riak_core_bucket_type:get(Type),
src/riak_core_bucket.erl:195:    [{riak_core_bucket:name(B), riak_core_bucket:n_val(B)} ||
src/riak_core_bucket.erl:196:        B <- riak_core_bucket:get_buckets(Ring)].
src/riak_core_bucket.erl:201:    riak_core_bucket:n_val(riak_core_bucket_props:defaults()).
src/riak_core_bucket.erl:206:    BucketTypeNVals = riak_core_bucket_type:all_n(),
src/riak_core_bucket_props.erl:39:               {riak_core_bucket_type:bucket_type(), undefined | binary()} | binary(),
src/riak_core_bucket_type.erl:113:-type bucket_type_props() :: riak_core_bucket:properties().
src/riak_core_bucket_type.erl:197:    ?IF_CAPABLE(riak_core_claimant:create_bucket_type(BucketType, Props1),
src/riak_core_bucket_type.erl:205:    ?IF_CAPABLE(riak_core_claimant:bucket_type_status(BucketType), undefined).
src/riak_core_bucket_type.erl:213:    ?IF_CAPABLE(riak_core_claimant:activate_bucket_type(BucketType), {error, undefined}).
src/riak_core_bucket_type.erl:222:    ?IF_CAPABLE(riak_core_claimant:update_bucket_type(BucketType, Props), {error, not_capable}).
src/riak_core_bucket_type.erl:229:    riak_core_claimant:get_bucket_type(BucketType, undefined).
src/riak_core_bucket_type.erl:242:-spec all_n() -> riak_core_bucket:nval_set().
src/riak_core_bucket_type.erl:244:    riak_core_bucket_type:fold(fun bucket_type_prop_nval_fold/2, ordsets:new()).
src/riak_core_bucket_type.erl:247:-spec bucket_type_prop_nval_fold({bucket_type(), riak_core_bucket:properties()},
src/riak_core_bucket_type.erl:248:        riak_core_bucket:nval_set()) -> riak_core_bucket:nval_set().
src/riak_core_bucket_type.erl:250:    case riak_core_bucket:get_value(active, BProps) of
src/riak_core_bucket_type.erl:257:-spec bucket_prop_nval_fold(riak_core_bucket:properties(), riak_core_bucket:nval_set()) ->
src/riak_core_bucket_type.erl:258:    riak_core_bucket:nval_set().
src/riak_core_bucket_type.erl:260:    case riak_core_bucket:get_value(n_val, BProps) of
src/riak_core_bucket_type.erl:281:    case riak_core_bucket_type:itr_done(It) of
src/riak_core_bucket_type.erl:283:            riak_core_bucket_type:itr_close(It),
src/riak_core_bucket_type.erl:287:            fold(riak_core_bucket_type:itr_next(It), Fun, NewAccum)
src/riak_core_bucket_type.erl:294:    riak_core_claimant:bucket_type_iterator().
src/riak_core_claimant.erl:164:%%          --------> riak_core_claimant:ring_changed/2
src/riak_core_claimant.erl:168:%% @see riak_core_bucket_type:create/2
src/riak_core_claimant.erl:169:-spec create_bucket_type(riak_core_bucket_type:bucket_type(), [{atom(), any()}]) ->
src/riak_core_claimant.erl:174:%% @see riak_core_bucket_type:status/1
src/riak_core_claimant.erl:175:-spec bucket_type_status(riak_core_bucket_type:bucket_type()) ->
src/riak_core_claimant.erl:180:%% @see riak_core_bucket_type:activate/1
src/riak_core_claimant.erl:181:-spec activate_bucket_type(riak_core_bucket_type:bucket_type()) ->
src/riak_core_claimant.erl:188:-spec get_bucket_type(riak_core_bucket_type:bucket_type(), X) -> [{atom(), any()}] | X.
src/riak_core_claimant.erl:195:-spec get_bucket_type(riak_core_bucket_type:bucket_type(), X, boolean()) ->
src/riak_core_claimant.erl:209:%% @see riak_core_bucket_type:update/2
src/riak_core_claimant.erl:210:-spec update_bucket_type(riak_core_bucket_type:bucket_type(), [{atom(), any()}]) ->
src/riak_core_claimant.erl:215:%% @see riak_core_bucket_type:iterator/0
src/riak_core_claim_sim.erl:302:              {_, NewRing2} = riak_core_claimant:reassign_indices(NewRing),
src/riak_core_console.erl:390:        case riak_core_claimant:leave_member(Node) of
src/riak_core_console.erl:417:        case riak_core_claimant:remove_member(Node) of
src/riak_core_console.erl:443:        case riak_core_claimant:replace(Node1, Node2) of
src/riak_core_console.erl:477:        case riak_core_claimant:force_replace(Node1, Node2) of
src/riak_core_console.erl:509:        case riak_core_claimant:abort_resize() of
src/riak_core_console.erl:533:        case riak_core_claimant:resize_ring(NewRingSize) of
src/riak_core_console.erl:565:        case riak_core_claimant:clear() of
src/riak_core_console.erl:589:    case riak_core_claimant:plan() of
src/riak_core_console.erl:724:    case riak_core_claimant:commit() of
src/riak_core_handoff_manager.erl:442:            DefaultN = riak_core_bucket:default_object_nval()
src/riak_core_ring.erl:1438:            riak_core_claimant:ring_changed(Node, CState)
src/riak_core_ring_manager.erl:192:    riak_core_bucket:get_bucket(Bucket);
src/riak_core_ring_manager.erl:575:                        BucketProps = riak_core_bucket:get_bucket(Bucket, Ring),
src/riak_core_ring_manager.erl:580:                        MergedProps = riak_core_bucket:merge_props(
src/riak_core_ring_manager.erl:600:    %% special meaning in `riak_core_bucket:get_bucket_props/2`. We then
src/riak_core_util.erl:271:    BucketProps = riak_core_bucket:get_bucket(Bucket),
src/riak_core_util.erl:884:    BucketProps = riak_core_bucket:get_bucket(Bucket),
src/riak_core_util.erl:948:    BucketProps = [riak_core_bucket:get_bucket(Bucket, Ring) || Bucket <- Buckets],

Active projects using riak_core

To grep around for riak_core API usage

https://github.com/basho/riak_kv.git https://gitlab.com/Project-FiFo/DalmatinerDB/dalmatinerdb https://gitlab.com/Project-FiFo/FiFo/sniffle https://gitlab.com/Project-FiFo/FiFo/snarl https://gitlab.com/Project-FiFo/FiFo/howl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment