pficommonのJubatus移植

pficommon4jubatus.md

pficommonのjubatus移植

• pficommonのコードはjubatusリポジトリの jubatus/util 以下へ持ってくる

• utilがくるディレクトリはcore, serverと同じレベルになる

• jubatus_coreに並んで、jubatus_utilモジュールができて、coreとserverから依存されることになる

• Jubatusで参照されてないコード(network, databaseなど)は含めない

• pfi/pficommonリベース済みリポジトリ:

• https://github.com/gwtnb/pficommon/tree/jubatus

• 議事録 7/1

• https://github.com/jubatus/jubatus/wiki/Meeting%3A2013_07_01#discussion

作業の流れ

1, 2, 3は1人もしくは少人数で実行。2,4は機械的な作業が多いため、sedで実行して、あとはcpplintのための作業になる見込み。 3は選択項目が若干あるかもしれない（どのファイルを削除すべきか、等）。 README作成は地味に手間がかかりそうなので、誰か1名割り当てると作業はスムーズになりそう。

1. (1名) pficommonをjubatusリポジトリへ持ってくる
2. (1〜若干名) 本家pficommonと混ざらないように名前空間やインクルードガードを修正
3. (1〜若干名) jubatus用に準備（不要な依存の削除、wscript等の整理）
4. (1〜人海戦術で) pficommonの依存を削除して、jubatus::utilへ移行

他のブランチを気にせず実行できる取り込み作業 (1,2,3)

pficommonをjubatus/utilへ移動

• https://github.com/gwtnb/pficommon/tree/jubatus からコピーしてくる
• 上記でもjubatusから参照してない、不要と思われるモジュールの列挙・削除
• コミットは次の作業が終わってからかな？

移動したpficommon(jubatus/util)の修正

• pficommon本家と衝突しないよう名前を変更
• 名前空間の変更 pfi:: → jubatus::util へ
• ヘッダのインクルードガード
• ディレクトリ構成(src不要かも), wscriptの修正（インストール先）
• README修正
• Jubatus向け移植にあたっての変更点を書く（上記の変更）
• pficommonの元のライセンスから変更はない
• 変更点は、上の作業のように名前空間とインクルードガードの変更だけであり、追加・改善はないためCopyrightを追記しない旨
• forkした段階と、最後に更新したときの上流pfi/pficommonバージョン(ハッシュ・日時)
• 書きかけ: https://gist.github.com/suma/5b2e511587dc7adae387

他のブランチとの衝突を避けて実行したい作業 (4)

pficommonの依存は大きいので、手動マージになるとつらいのでタイミングはチームであわせた方がよい。

• pficommonからjubatus::utilへの移行作業
• pficommon依存の削除・jubatus::utilへの移行
• pficommonを使用しているコードの修正（モジュールの参照、ヘッダのパス、名前空間やwscriptの修正）

影響のあるヘッダとソース

次のコマンド・実行結果が変更範囲となる（たぶん）。

$ egrep "pfi::|pficommon" -l find . -name "*pp"

./jubatus/jubatus/server/jubavisor/main.cpp
./jubatus/jubatus/server/jubavisor/jubavisor.hpp
./jubatus/jubatus/server/jubavisor/jubavisor.cpp
./jubatus/jubatus/server/jubavisor/process.cpp
./jubatus/jubatus/server/server/recommender_impl.cpp
./jubatus/jubatus/server/server/graph_server.hpp
./jubatus/jubatus/server/server/regression_impl.cpp
./jubatus/jubatus/server/server/nearest_neighbor_serv.hpp
./jubatus/jubatus/server/server/stat_impl.cpp
./jubatus/jubatus/server/server/classifier_impl.cpp
./jubatus/jubatus/server/server/regression_serv.cpp
./jubatus/jubatus/server/server/regression_proxy.cpp
./jubatus/jubatus/server/server/graph_impl.cpp
./jubatus/jubatus/server/server/nearest_neighbor_serv.cpp
./jubatus/jubatus/server/server/classifier_serv.cpp
./jubatus/jubatus/server/server/graph_proxy.cpp
./jubatus/jubatus/server/server/regression_server.hpp
./jubatus/jubatus/server/server/stat_serv.cpp
./jubatus/jubatus/server/server/stat_server.hpp
./jubatus/jubatus/server/server/anomaly_impl.cpp
./jubatus/jubatus/server/server/nearest_neighbor_server.hpp
./jubatus/jubatus/server/server/classifier_server.hpp
./jubatus/jubatus/server/server/anomaly_serv.cpp
./jubatus/jubatus/server/server/classifier_proxy.cpp
./jubatus/jubatus/server/server/anomaly_proxy.cpp
./jubatus/jubatus/server/server/nearest_neighbor_proxy.cpp
./jubatus/jubatus/server/server/classifier_serv.hpp
./jubatus/jubatus/server/server/graph_serv.hpp
./jubatus/jubatus/server/server/stat_serv.hpp
./jubatus/jubatus/server/server/graph_serv.cpp
./jubatus/jubatus/server/server/recommender_serv.cpp
./jubatus/jubatus/server/server/recommender_server.hpp
./jubatus/jubatus/server/server/stat_proxy.cpp
./jubatus/jubatus/server/server/nearest_neighbor_impl.cpp
./jubatus/jubatus/server/server/recommender_proxy.cpp
./jubatus/jubatus/server/server/anomaly_serv.hpp
./jubatus/jubatus/server/server/recommender_serv.hpp
./jubatus/jubatus/server/server/regression_serv.hpp
./jubatus/jubatus/server/server/anomaly_server.hpp
./jubatus/jubatus/server/cmd/jubactl.cpp
./jubatus/jubatus/server/cmd/jubaconfig.cpp
./jubatus/jubatus/server/cmd/jubaconv.cpp
./jubatus/jubatus/server/common/cached_zk.cpp
./jubatus/jubatus/server/common/cht.cpp
./jubatus/jubatus/server/common/config_test.cpp
./jubatus/jubatus/server/common/config.hpp
./jubatus/jubatus/server/common/signals.hpp
./jubatus/jubatus/server/common/unique_lock.hpp
./jubatus/jubatus/server/common/membership.cpp
./jubatus/jubatus/server/common/unique_lock_test.cpp
./jubatus/jubatus/server/common/signals.cpp
./jubatus/jubatus/server/common/global_id_generator_zk.cpp
./jubatus/jubatus/server/common/zk.cpp
./jubatus/jubatus/server/common/global_id_generator_standalone.hpp
./jubatus/jubatus/server/common/mprpc/rpc_util.hpp
./jubatus/jubatus/server/common/mprpc/rpc_server.cpp
./jubatus/jubatus/server/common/mprpc/rpc_client_test.cpp
./jubatus/jubatus/server/common/mprpc/rpc_result.hpp
./jubatus/jubatus/server/common/mprpc/rpc_error.hpp
./jubatus/jubatus/server/common/mprpc/rpc_mclient.hpp
./jubatus/jubatus/server/common/mprpc/rpc_server.hpp
./jubatus/jubatus/server/common/mprpc/rpc_response.hpp
./jubatus/jubatus/server/common/network_test.cpp
./jubatus/jubatus/server/common/util.cpp
./jubatus/jubatus/server/common/cached_zk_test.cpp
./jubatus/jubatus/server/common/global_id_generator_standalone.cpp
./jubatus/jubatus/server/common/network.cpp
./jubatus/jubatus/server/common/cht.hpp
./jubatus/jubatus/server/common/config.cpp
./jubatus/jubatus/server/common/lock_service.hpp
./jubatus/jubatus/server/common/zk.hpp
./jubatus/jubatus/server/common/network.hpp
./jubatus/jubatus/server/common/zk_test.cpp
./jubatus/jubatus/server/common/global_id_generator_zk.hpp
./jubatus/jubatus/server/framework/proxy.hpp
./jubatus/jubatus/server/framework/server_base.hpp
./jubatus/jubatus/server/framework/proxy.cpp
./jubatus/jubatus/server/framework/server_util.cpp
./jubatus/jubatus/server/framework/proxy_common.hpp
./jubatus/jubatus/server/framework/server_helper.cpp
./jubatus/jubatus/server/framework/proxy_common.cpp
./jubatus/jubatus/server/framework/mixer/dummy_mixer.hpp
./jubatus/jubatus/server/framework/mixer/linear_mixer_test.cpp
./jubatus/jubatus/server/framework/mixer/skip_mixer.hpp
./jubatus/jubatus/server/framework/mixer/linear_mixer.hpp
./jubatus/jubatus/server/framework/mixer/mixer_factory.cpp
./jubatus/jubatus/server/framework/mixer/push_mixer.cpp
./jubatus/jubatus/server/framework/mixer/linear_mixer.cpp
./jubatus/jubatus/server/framework/mixer/mixer.hpp
./jubatus/jubatus/server/framework/mixer/push_mixer.hpp
./jubatus/jubatus/server/framework/mixer/random_mixer.hpp
./jubatus/jubatus/server/framework/mixer/mixer_factory.hpp
./jubatus/jubatus/server/framework/mixer/broadcast_mixer.hpp
./jubatus/jubatus/server/framework/server_util.hpp
./jubatus/jubatus/server/framework/server_helper.hpp
./jubatus/jubatus/core/nearest_neighbor/lsh.cpp
./jubatus/jubatus/core/nearest_neighbor/nearest_neighbor_base_test.cpp
./jubatus/jubatus/core/nearest_neighbor/nearest_neighbor_test.cpp
./jubatus/jubatus/core/nearest_neighbor/bit_vector_nearest_neighbor_base.cpp
./jubatus/jubatus/core/nearest_neighbor/bit_vector_nearest_neighbor_base_test.cpp
./jubatus/jubatus/core/nearest_neighbor/minhash.hpp
./jubatus/jubatus/core/nearest_neighbor/nearest_neighbor_factory.hpp
./jubatus/jubatus/core/nearest_neighbor/nearest_neighbor_base.cpp
./jubatus/jubatus/core/nearest_neighbor/nearest_neighbor_base.hpp
./jubatus/jubatus/core/nearest_neighbor/euclid_lsh.cpp
./jubatus/jubatus/core/nearest_neighbor/nearest_neighbor_factory.cpp
./jubatus/jubatus/core/nearest_neighbor/bit_vector_nearest_neighbor_base.hpp
./jubatus/jubatus/core/nearest_neighbor/euclid_lsh.hpp
./jubatus/jubatus/core/nearest_neighbor/minhash.cpp
./jubatus/jubatus/core/nearest_neighbor/lsh.hpp
./jubatus/jubatus/core/nearest_neighbor/lsh_function.cpp
./jubatus/jubatus/core/driver/nearest_neighbor.hpp
./jubatus/jubatus/core/driver/stat.hpp
./jubatus/jubatus/core/driver/anomaly.hpp
./jubatus/jubatus/core/driver/graph.cpp
./jubatus/jubatus/core/driver/recommender.cpp
./jubatus/jubatus/core/driver/recommender_test.cpp
./jubatus/jubatus/core/driver/regression_test.cpp
./jubatus/jubatus/core/driver/regression.cpp
./jubatus/jubatus/core/driver/graph.hpp
./jubatus/jubatus/core/driver/classifier.cpp
./jubatus/jubatus/core/driver/test_util.hpp
./jubatus/jubatus/core/driver/classifier.hpp
./jubatus/jubatus/core/driver/classifier_test.cpp
./jubatus/jubatus/core/driver/stat_test.cpp
./jubatus/jubatus/core/driver/anomaly_test.cpp
./jubatus/jubatus/core/driver/nearest_neighbor.cpp
./jubatus/jubatus/core/driver/graph_test.cpp
./jubatus/jubatus/core/driver/recommender.hpp
./jubatus/jubatus/core/driver/anomaly.cpp
./jubatus/jubatus/core/driver/regression.hpp
./jubatus/jubatus/core/driver/linear_function_mixer.cpp
./jubatus/jubatus/core/classifier/classifier_factory.hpp
./jubatus/jubatus/core/classifier/classifier_base.hpp
./jubatus/jubatus/core/classifier/classifier_factory.cpp
./jubatus/jubatus/core/classifier/classifier_test_util.hpp
./jubatus/jubatus/core/classifier/classifier_test.cpp
./jubatus/jubatus/core/classifier/classifier_config.hpp
./jubatus/jubatus/core/stat/stat.hpp
./jubatus/jubatus/core/stat/stat.cpp
./jubatus/jubatus/core/stat/stat_test.cpp
./jubatus/jubatus/core/graph/graph_wo_index.hpp
./jubatus/jubatus/core/graph/graph_factory.hpp
./jubatus/jubatus/core/graph/graph_wo_index_test.cpp
./jubatus/jubatus/core/graph/graph_type.hpp
./jubatus/jubatus/core/graph/graph_wo_index.cpp
./jubatus/jubatus/core/graph/graph_factory.cpp
./jubatus/jubatus/core/fv_converter/num_filter_factory.hpp
./jubatus/jubatus/core/fv_converter/datum_to_fv_converter_test.cpp
./jubatus/jubatus/core/fv_converter/splitter_factory.cpp
./jubatus/jubatus/core/fv_converter/key_matcher_factory.cpp
./jubatus/jubatus/core/fv_converter/key_matcher_factory.hpp
./jubatus/jubatus/core/fv_converter/json_converter.cpp
./jubatus/jubatus/core/fv_converter/num_feature_factory_test.cpp
./jubatus/jubatus/core/fv_converter/datum_to_fv_converter.hpp
./jubatus/jubatus/core/fv_converter/keyword_weights.hpp
./jubatus/jubatus/core/fv_converter/num_filter_factory.cpp
./jubatus/jubatus/core/fv_converter/num_feature_factory.cpp
./jubatus/jubatus/core/fv_converter/dynamic_num_feature_test.cpp
./jubatus/jubatus/core/fv_converter/dynamic_splitter.hpp
./jubatus/jubatus/core/fv_converter/util.cpp
./jubatus/jubatus/core/fv_converter/feature_hasher.cpp
./jubatus/jubatus/core/fv_converter/json_converter.hpp
./jubatus/jubatus/core/fv_converter/string_filter_factory.hpp
./jubatus/jubatus/core/fv_converter/datum_to_fv_converter.cpp
./jubatus/jubatus/core/fv_converter/datum.hpp
./jubatus/jubatus/core/fv_converter/num_feature_factory.hpp
./jubatus/jubatus/core/fv_converter/string_filter_factory.cpp
./jubatus/jubatus/core/fv_converter/except_match_test.cpp
./jubatus/jubatus/core/fv_converter/splitter_factory.hpp
./jubatus/jubatus/core/fv_converter/splitter_factory_test.cpp
./jubatus/jubatus/core/fv_converter/mixable_weight_manager.cpp
./jubatus/jubatus/core/fv_converter/weight_manager.hpp
./jubatus/jubatus/core/fv_converter/revert.cpp
./jubatus/jubatus/core/fv_converter/dynamic_string_filter.hpp
./jubatus/jubatus/core/fv_converter/prefix_match.hpp
./jubatus/jubatus/core/fv_converter/string_filter_factory_test.cpp
./jubatus/jubatus/core/fv_converter/num_filter_factory_test.cpp
./jubatus/jubatus/core/fv_converter/counter.hpp
./jubatus/jubatus/core/fv_converter/dynamic_num_filter.hpp
./jubatus/jubatus/core/fv_converter/dynamic_num_feature.hpp
./jubatus/jubatus/core/fv_converter/suffix_match.hpp
./jubatus/jubatus/core/fv_converter/json_converter_test.cpp
./jubatus/jubatus/core/fv_converter/except_match.hpp
./jubatus/jubatus/core/fv_converter/libsvm_converter.cpp
./jubatus/jubatus/core/fv_converter/converter_config_test.cpp
./jubatus/jubatus/core/fv_converter/converter_config.cpp
./jubatus/jubatus/core/fv_converter/converter_config.hpp
./jubatus/jubatus/core/fv_converter/key_matcher_factory_test.cpp
./jubatus/jubatus/core/recommender/lsh_util.hpp
./jubatus/jubatus/core/recommender/recommender_mock.cpp
./jubatus/jubatus/core/recommender/lsh_util_test.cpp
./jubatus/jubatus/core/recommender/lsh.cpp
./jubatus/jubatus/core/recommender/recommender_mock.hpp
./jubatus/jubatus/core/recommender/recommender_mock_storage.cpp
./jubatus/jubatus/core/recommender/euclid_lsh_test.cpp
./jubatus/jubatus/core/recommender/minhash.hpp
./jubatus/jubatus/core/recommender/recommender_base.cpp
./jubatus/jubatus/core/recommender/recommender_mock_storage.hpp
./jubatus/jubatus/core/recommender/nearest_neighbor_recommender.cpp
./jubatus/jubatus/core/recommender/lsh_util.cpp
./jubatus/jubatus/core/recommender/euclid_lsh.cpp
./jubatus/jubatus/core/recommender/recommender_factory.cpp
./jubatus/jubatus/core/recommender/recommender_random_test.cpp
./jubatus/jubatus/core/recommender/recommender_mock_test.cpp
./jubatus/jubatus/core/recommender/euclid_lsh.hpp
./jubatus/jubatus/core/recommender/recommender_base.hpp
./jubatus/jubatus/core/recommender/minhash.cpp
./jubatus/jubatus/core/recommender/recommender_mock_storage_test.cpp
./jubatus/jubatus/core/recommender/lsh.hpp
./jubatus/jubatus/core/recommender/inverted_index.hpp
./jubatus/jubatus/core/recommender/nearest_neighbor_recommender.hpp
./jubatus/jubatus/core/recommender/recommender_factory.hpp
./jubatus/jubatus/core/recommender/recommender_mock_util.cpp
./jubatus/jubatus/core/recommender/inverted_index.cpp
./jubatus/jubatus/core/common/key_manager.cpp
./jubatus/jubatus/core/common/unordered_map.hpp
./jubatus/jubatus/core/common/exception.cpp
./jubatus/jubatus/core/common/portable_mixer.hpp
./jubatus/jubatus/core/common/byte_buffer_test.cpp
./jubatus/jubatus/core/common/exception.hpp
./jubatus/jubatus/core/common/jsonconfig/config.hpp
./jubatus/jubatus/core/common/jsonconfig/exception.cpp
./jubatus/jubatus/core/common/jsonconfig/exception.hpp
./jubatus/jubatus/core/common/jsonconfig/cast.hpp
./jubatus/jubatus/core/common/jsonconfig/config.cpp
./jubatus/jubatus/core/common/byte_buffer.hpp
./jubatus/jubatus/core/common/exception_test.cpp
./jubatus/jubatus/core/common/key_manager.hpp
./jubatus/jubatus/core/common/exception_info.hpp
./jubatus/jubatus/core/common/jsonconfig_test.cpp
./jubatus/jubatus/core/regression/regression_factory_test.cpp
./jubatus/jubatus/core/regression/regression_test.cpp
./jubatus/jubatus/core/regression/regression_factory.cpp
./jubatus/jubatus/core/regression/passive_aggressive.hpp
./jubatus/jubatus/core/regression/regression_factory.hpp
./jubatus/jubatus/core/regression/regression_test_util.hpp
./jubatus/jubatus/core/regression/regression_base.hpp
./jubatus/jubatus/core/storage/local_storage_mixture.hpp
./jubatus/jubatus/core/storage/norm_l2.cpp
./jubatus/jubatus/core/storage/norm_none.hpp
./jubatus/jubatus/core/storage/norm_l1.cpp
./jubatus/jubatus/core/storage/local_storage_mixture.cpp
./jubatus/jubatus/core/storage/lsh_index_storage.hpp
./jubatus/jubatus/core/storage/inverted_index_storage.cpp
./jubatus/jubatus/core/storage/storage_test.cpp
./jubatus/jubatus/core/storage/lsh_vector.hpp
./jubatus/jubatus/core/storage/norm_l1.hpp
./jubatus/jubatus/core/storage/inverted_index_storage.hpp
./jubatus/jubatus/core/storage/sparse_matrix_storage.cpp
./jubatus/jubatus/core/storage/local_storage.cpp
./jubatus/jubatus/core/storage/local_storage.hpp
./jubatus/jubatus/core/storage/bit_vector.hpp
./jubatus/jubatus/core/storage/norm_factory.hpp
./jubatus/jubatus/core/storage/lsh_index_storage.cpp
./jubatus/jubatus/core/storage/local_storage_mixture_test.cpp
./jubatus/jubatus/core/storage/storage_factory_test.cpp
./jubatus/jubatus/core/storage/bit_index_storage.cpp
./jubatus/jubatus/core/storage/norm_l2.hpp
./jubatus/jubatus/core/storage/storage_factory.hpp
./jubatus/jubatus/core/storage/storage_base.cpp
./jubatus/jubatus/core/storage/bit_index_storage.hpp
./jubatus/jubatus/core/storage/sparse_matrix_storage.hpp
./jubatus/jubatus/core/storage/storage_factory.cpp
./jubatus/jubatus/core/storage/norm_factory.cpp
./jubatus/jubatus/core/storage/storage_type.hpp
./jubatus/jubatus/core/table/column/column_type.hpp
./jubatus/jubatus/core/table/column/column_table.cpp
./jubatus/jubatus/core/table/column/column_table.hpp
./jubatus/jubatus/core/table/column/column_table_test.cpp
./jubatus/jubatus/core/table/column/bit_vector.hpp
./jubatus/jubatus/core/table/column/abstract_column.hpp
./jubatus/jubatus/core/table/column/owner.hpp
./jubatus/jubatus/core/anomaly/light_lof_test.cpp
./jubatus/jubatus/core/anomaly/anomaly_factory.cpp
./jubatus/jubatus/core/anomaly/anomaly_factory.hpp
./jubatus/jubatus/core/anomaly/lof.hpp
./jubatus/jubatus/core/anomaly/anomaly_base.hpp
./jubatus/jubatus/core/anomaly/lof.cpp
./jubatus/jubatus/core/anomaly/lof_storage_test.cpp
./jubatus/jubatus/core/anomaly/lof_test.cpp
./jubatus/jubatus/core/anomaly/lof_storage.cpp
./jubatus/jubatus/core/anomaly/light_lof.hpp
./jubatus/jubatus/core/anomaly/lof_storage.hpp
./jubatus/jubatus/core/anomaly/light_lof.cpp
./jubatus/jubatus/core/framework/mixable.hpp

jubatus/core や jubatus/server と同レベルに jubatus/util っていうのを作ってそこに移動するという認識です．
名前空間は jubatus::util::lang とかですね

ありがとうございます！

👍