Created
November 8, 2022 10:03
-
-
Save mhash1m/dbf89266d294801fe94437628a076265 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Checking whether there is an H2O instance running at http://localhost:54321 ..... not found. | |
Attempting to start a local H2O server... | |
Java Version: openjdk version "1.8.0_342"; OpenJDK Runtime Environment (build 1.8.0_342-8u342-b07-0ubuntu1~20.04-b07); OpenJDK 64-Bit Server VM (build 25.342-b07, mixed mode) | |
Starting server from /home/hash1m/miniconda3/envs/zen_ml/lib/python3.7/site-packages/h2o/backend/bin/h2o.jar | |
Ice root: /tmp/tmpjslvnzy5 | |
JVM stdout: /tmp/tmpjslvnzy5/h2o_hash1m_started_from_python.out | |
JVM stderr: /tmp/tmpjslvnzy5/h2o_hash1m_started_from_python.err | |
Server is running at http://127.0.0.1:54321 | |
Connecting to H2O server at http://127.0.0.1:54321 ... successful. | |
-------------------------- ----------------------------- | |
H2O_cluster_uptime: 01 secs | |
H2O_cluster_timezone: Asia/Karachi | |
H2O_data_parsing_timezone: UTC | |
H2O_cluster_version: 3.36.1.5 | |
H2O_cluster_version_age: 1 month and 24 days | |
H2O_cluster_name: H2O_from_python_hash1m_kzwvlu | |
H2O_cluster_total_nodes: 1 | |
H2O_cluster_free_memory: 3.424 Gb | |
H2O_cluster_total_cores: 8 | |
H2O_cluster_allowed_cores: 8 | |
H2O_cluster_status: locked, healthy | |
H2O_connection_url: http://127.0.0.1:54321 | |
H2O_connection_proxy: {"http": null, "https": null} | |
H2O_internal_security: False | |
Python_version: 3.7.13 final | |
-------------------------- ----------------------------- | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
You specified a plain-text value for the sensitive attribute tracking_token for a MLFlowExperimentTrackerConfig stack component. This is currently only a warning, but future versions of ZenML will require you to pass in sensitive information as secrets. Check out the documentation on how to configure your stack components with secrets here: https://docs.zenml.io/advanced-guide/practical/secrets-management | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
Running pipeline rekog_training_pipeline on stack mlflow_s3 (caching disabled) | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
You specified a plain-text value for the sensitive attribute tracking_token for a MLFlowExperimentTrackerConfig stack component. This is currently only a warning, but future versions of ZenML will require you to pass in sensitive information as secrets. Check out the documentation on how to configure your stack components with secrets here: https://docs.zenml.io/advanced-guide/practical/secrets-management | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
Step ingest_data has started. | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
You specified a plain-text value for the sensitive attribute tracking_token for a MLFlowExperimentTrackerConfig stack component. This is currently only a warning, but future versions of ZenML will require you to pass in sensitive information as secrets. Check out the documentation on how to configure your stack components with secrets here: https://docs.zenml.io/advanced-guide/practical/secrets-management | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials | |
Step ingest_data has finished in 22.338s. | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
You specified a plain-text value for the sensitive attribute tracking_token for a MLFlowExperimentTrackerConfig stack component. This is currently only a warning, but future versions of ZenML will require you to pass in sensitive information as secrets. Check out the documentation on how to configure your stack components with secrets here: https://docs.zenml.io/advanced-guide/practical/secrets-management | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
You are running with a non-default project 'pascalle1'. Any stacks, components, pipelines and pipeline runs produced in this project will currently not be accessible through the dashboard. However, this will be possible in the near future. | |
╭───────────────────── Traceback (most recent call last) ──────────────────────╮ | |
│ /home/hash1m/zenml_SM/AutoML-User-Interface-master/pipeline_run.py:86 in │ | |
│ <module> │ | |
│ │ | |
│ 83 │ # rekog_pipeline_run('./wiz_rekog.csv', 'likes_count', │ | |
│ 84 │ # aug_pipeline_run('./wiz_rekog.csv', 'likes_count', ta │ | |
│ 85 │ # time_pipeline_run('./frank.csv', 'likes_count', "regression", "3" │ | |
│ ❱ 86 │ rekog_pipeline_run('./frank.csv', 'likes_count', "regression", "3", │ | |
│ 87 │ # aug_pipeline_run('./sns312.csv', 'likes_count', "classification", │ | |
│ 88 │ | |
│ │ | |
│ /home/hash1m/zenml_SM/AutoML-User-Interface-master/pipeline_run.py:73 in │ | |
│ rekog_pipeline_run │ | |
│ │ | |
│ 70 │ │ feature_selection = feature_selection(), │ | |
│ 71 │ │ run_h2o = run_h2o() │ | |
│ 72 │ ) │ | |
│ ❱ 73 │ train_pipeline.run() │ | |
│ 74 │ | |
│ 75 if __name__ == "__main__": │ | |
│ 76 │ # bin_opts = ["0", "1"] │ | |
│ │ | |
│ /home/hash1m/miniconda3/envs/zen_ml/lib/python3.7/site-packages/zenml/pipeli │ | |
│ nes/base_pipeline.py:538 in run │ | |
│ │ | |
│ 535 │ │ # behavior │ | |
│ 536 │ │ constants.SHOULD_PREVENT_PIPELINE_EXECUTION = True │ | |
│ 537 │ │ try: │ | |
│ ❱ 538 │ │ │ return_value = stack.deploy_pipeline(pipeline_deployment) │ | |
│ 539 │ │ finally: │ | |
│ 540 │ │ │ constants.SHOULD_PREVENT_PIPELINE_EXECUTION = False │ | |
│ 541 │ | |
│ │ | |
│ /home/hash1m/miniconda3/envs/zen_ml/lib/python3.7/site-packages/zenml/stack/ │ | |
│ stack.py:698 in deploy_pipeline │ | |
│ │ | |
│ 695 │ │ Returns: │ | |
│ 696 │ │ │ The return value of the call to `orchestrator.run_pipeline │ | |
│ 697 │ │ """ │ | |
│ ❱ 698 │ │ return self.orchestrator.run(deployment=deployment, stack=self │ | |
│ 699 │ │ | |
│ 700 │ def _get_active_components_for_step( │ | |
│ 701 │ │ self, step_config: "StepConfiguration" │ | |
│ │ | |
│ /home/hash1m/miniconda3/envs/zen_ml/lib/python3.7/site-packages/zenml/orches │ | |
│ trators/base_orchestrator.py:283 in run │ | |
│ │ | |
│ 280 │ │ self._prepare_run(deployment=deployment) │ | |
│ 281 │ │ │ | |
│ 282 │ │ result = self.prepare_or_run_pipeline( │ | |
│ ❱ 283 │ │ │ deployment=deployment, stack=stack │ | |
│ 284 │ │ ) │ | |
│ 285 │ │ │ | |
│ 286 │ │ self._cleanup_run() │ | |
│ │ | |
│ /home/hash1m/miniconda3/envs/zen_ml/lib/python3.7/site-packages/zenml/orches │ | |
│ trators/local/local_orchestrator.py:77 in prepare_or_run_pipeline │ | |
│ │ | |
│ 74 │ │ │ │ ) │ | |
│ 75 │ │ │ │ | |
│ 76 │ │ │ self.run_step( │ | |
│ ❱ 77 │ │ │ │ step=step, │ | |
│ 78 │ │ │ ) │ | |
│ 79 │ │ │ | |
│ 80 │ │ run_duration = time.time() - start_time │ | |
│ │ | |
│ /home/hash1m/miniconda3/envs/zen_ml/lib/python3.7/site-packages/zenml/orches │ | |
│ trators/base_orchestrator.py:370 in run_step │ | |
│ │ | |
│ 367 │ │ if step.config.step_operator: │ | |
│ 368 │ │ │ execution_info = self._execute_step(component_launcher) │ | |
│ 369 │ │ else: │ | |
│ ❱ 370 │ │ │ stack.prepare_step_run(info=step_run_info) │ | |
│ 371 │ │ │ try: │ | |
│ 372 │ │ │ │ execution_info = self._execute_step(component_launcher │ | |
│ 373 │ │ │ except: # noqa: E722 │ | |
│ │ | |
│ /home/hash1m/miniconda3/envs/zen_ml/lib/python3.7/site-packages/zenml/stack/ │ | |
│ stack.py:745 in prepare_step_run │ | |
│ │ | |
│ 742 │ │ for component in self._get_active_components_for_step( │ | |
│ 743 │ │ │ info.config │ | |
│ 744 │ │ ).values(): │ | |
│ ❱ 745 │ │ │ component.prepare_step_run(info=info) │ | |
│ 746 │ │ | |
│ 747 │ def cleanup_step_run(self, info: "StepRunInfo") -> None: │ | |
│ 748 │ │ """Cleans up resources after the step run is finished. │ | |
│ │ | |
│ /home/hash1m/miniconda3/envs/zen_ml/lib/python3.7/site-packages/zenml/integr │ | |
│ ations/mlflow/experiment_trackers/mlflow_experiment_tracker.py:192 in │ | |
│ prepare_step_run │ | |
│ │ | |
│ 189 │ │ │ run_id=run_id, │ | |
│ 190 │ │ │ run_name=info.run_name, │ | |
│ 191 │ │ │ experiment_id=experiment.experiment_id, │ | |
│ ❱ 192 │ │ │ tags=tags, │ | |
│ 193 │ │ ) │ | |
│ 194 │ │ │ | |
│ 195 │ │ if settings.nested: │ | |
│ │ | |
│ /home/hash1m/miniconda3/envs/zen_ml/lib/python3.7/site-packages/mlflow/track │ | |
│ ing/fluent.py:318 in start_run │ | |
│ │ | |
│ 315 │ │ │ | |
│ 316 │ │ resolved_tags = context_registry.resolve_tags(user_specified_ │ | |
│ 317 │ │ │ | |
│ ❱ 318 │ │ active_run_obj = client.create_run(experiment_id=exp_id_for_r │ | |
│ 319 │ │ | |
│ 320 │ _active_run_stack.append(ActiveRun(active_run_obj)) │ | |
│ 321 │ return _active_run_stack[-1] │ | |
│ │ | |
│ /home/hash1m/miniconda3/envs/zen_ml/lib/python3.7/site-packages/mlflow/track │ | |
│ ing/client.py:265 in create_run │ | |
│ │ | |
│ 262 │ │ │ lifecycle_stage: active │ | |
│ 263 │ │ │ status: RUNNING │ | |
│ 264 │ │ """ │ | |
│ ❱ 265 │ │ return self._tracking_client.create_run(experiment_id, start_ │ | |
│ 266 │ │ | |
│ 267 │ def list_run_infos( │ | |
│ 268 │ │ self, │ | |
│ │ | |
│ /home/hash1m/miniconda3/envs/zen_ml/lib/python3.7/site-packages/mlflow/track │ | |
│ ing/_tracking_service/client.py:103 in create_run │ | |
│ │ | |
│ 100 │ │ │ experiment_id=experiment_id, │ | |
│ 101 │ │ │ user_id=user_id, │ | |
│ 102 │ │ │ start_time=start_time or int(time.time() * 1000), │ | |
│ ❱ 103 │ │ │ tags=[RunTag(key, value) for (key, value) in tags.items()] │ | |
│ 104 │ │ ) │ | |
│ 105 │ │ | |
│ 106 │ def list_run_infos( │ | |
│ │ | |
│ /home/hash1m/miniconda3/envs/zen_ml/lib/python3.7/site-packages/mlflow/store │ | |
│ /tracking/rest_store.py:170 in create_run │ | |
│ │ | |
│ 167 │ │ │ │ tags=tag_protos, │ | |
│ 168 │ │ │ ) │ | |
│ 169 │ │ ) │ | |
│ ❱ 170 │ │ response_proto = self._call_endpoint(CreateRun, req_body) │ | |
│ 171 │ │ run = Run.from_proto(response_proto.run) │ | |
│ 172 │ │ return run │ | |
│ 173 │ | |
│ │ | |
│ /home/hash1m/miniconda3/envs/zen_ml/lib/python3.7/site-packages/mlflow/store │ | |
│ /tracking/rest_store.py:56 in _call_endpoint │ | |
│ │ | |
│ 53 │ def _call_endpoint(self, api, json_body): │ | |
│ 54 │ │ endpoint, method = _METHOD_TO_INFO[api] │ | |
│ 55 │ │ response_proto = api.Response() │ | |
│ ❱ 56 │ │ return call_endpoint(self.get_host_creds(), endpoint, method, │ | |
│ 57 │ │ | |
│ 58 │ def list_experiments( │ | |
│ 59 │ │ self, │ | |
│ │ | |
│ /home/hash1m/miniconda3/envs/zen_ml/lib/python3.7/site-packages/mlflow/utils │ | |
│ /rest_utils.py:256 in call_endpoint │ | |
│ │ | |
│ 253 │ │ response = http_request( │ | |
│ 254 │ │ │ host_creds=host_creds, endpoint=endpoint, method=method, j │ | |
│ 255 │ │ ) │ | |
│ ❱ 256 │ response = verify_rest_response(response, endpoint) │ | |
│ 257 │ js_dict = json.loads(response.text) │ | |
│ 258 │ parse_dict(js_dict=js_dict, message=response_proto) │ | |
│ 259 │ return response_proto │ | |
│ │ | |
│ /home/hash1m/miniconda3/envs/zen_ml/lib/python3.7/site-packages/mlflow/utils │ | |
│ /rest_utils.py:185 in verify_rest_response │ | |
│ │ | |
│ 182 │ """Verify the return code and format, raise exception if the reque │ | |
│ 183 │ if response.status_code != 200: │ | |
│ 184 │ │ if _can_parse_as_json_object(response.text): │ | |
│ ❱ 185 │ │ │ raise RestException(json.loads(response.text)) │ | |
│ 186 │ │ else: │ | |
│ 187 │ │ │ base_msg = "API request to endpoint %s failed with error c │ | |
│ 188 │ │ │ │ endpoint, │ | |
╰──────────────────────────────────────────────────────────────────────────────╯ | |
RestException: BAD_REQUEST: (psycopg2.errors.UniqueViolation) duplicate key | |
value violates unique constraint "tag_pk" | |
DETAIL: Key (key, run_uuid)=(mlflow.runName, 8bb915bab15842bc989a9868c978b5e0) | |
already exists. | |
[SQL: INSERT INTO tags (key, value, run_uuid) VALUES (%(key)s, %(value)s, | |
%(run_uuid)s)] | |
[parameters: ({'key': 'mlflow.user', 'value': 'hash1m', 'run_uuid': | |
'8bb915bab15842bc989a9868c978b5e0'}, {'key': 'mlflow.source.name', 'value': | |
'pipeline_run.py', 'run_uuid': '8bb915bab15842bc989a9868c978b5e0'}, {'key': | |
'mlflow.source.type', 'value': 'LOCAL', 'run_uuid': | |
'8bb915bab15842bc989a9868c978b5e0'}, {'key': 'mlflow.source.git.commit', | |
'value': '48ba36873316a75f4a1e1489459db2debdc74085', 'run_uuid': | |
'8bb915bab15842bc989a9868c978b5e0'}, {'key': 'zenml', 'value': '0.21.1', | |
'run_uuid': '8bb915bab15842bc989a9868c978b5e0'}, {'key': 'mlflow.runName', | |
'value': 'rekog_training_pipeline-2022_11_08-14_48_30_122312', 'run_uuid': | |
'8bb915bab15842bc989a9868c978b5e0'}, {'key': 'mlflow.runName', 'value': | |
'legendary-shrike-198', 'run_uuid': '8bb915bab15842bc989a9868c978b5e0'})] | |
(Background on this error at: https://sqlalche.me/e/14/gkpj) | |
Closing connection _sid_abb7 at exit | |
H2O session _sid_abb7 closed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment