Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
pe-orchestration-services sql ssl error after restore PE 2017.2.2

pe-orchestration-services sql ssl error after restore PE 2017.2.2

2017-07-06 13:02:36,858 ERROR [p.o.api] Failed to make a connection to the database
java.sql.SQLTransientConnectionException: Timeout waiting for the database pool to become ready.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at clojure.lang.Reflector.invokeConstructor(Reflector.java:180)
        at puppetlabs.jdbc_util.pool$wrap_with_delayed_init$reify__35232.getConnection(pool.clj:160)
        at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:307)
        at clojure.java.jdbc$get_connection.invoke(jdbc.clj:197)
        at clojure.java.jdbc$db_query_with_resultset.invokeStatic(jdbc.clj:878)
        at clojure.java.jdbc$db_query_with_resultset.invoke(jdbc.clj:847)
        at clojure.java.jdbc$query.invokeStatic(jdbc.clj:915)
        at clojure.java.jdbc$query.invoke(jdbc.clj:884)
        at clojure.java.jdbc$query.invokeStatic(jdbc.clj:898)
        at clojure.java.jdbc$query.invoke(jdbc.clj:884)
        at puppetlabs.orchestrator.job$list_jobs.invokeStatic(job.clj:94)
        at puppetlabs.orchestrator.job$list_jobs.invoke(job.clj:87)
        at puppetlabs.orchestrator.api.jobs$list_jobs.invokeStatic(jobs.clj:106)
        at puppetlabs.orchestrator.api.jobs$list_jobs.invoke(jobs.clj:92)
        at puppetlabs.orchestrator.api.jobs$app$fn__37263.invoke(jobs.clj:196)
        at ring.middleware.keyword_params$wrap_keyword_params$fn__17935.invoke(keyword_params.clj:35)
        at ring.middleware.params$wrap_params$fn__16133.invoke(params.clj:64)
        at bidi.ring$fn__18750.invokeStatic(ring.clj:21)
        at bidi.ring$fn__18750.invoke(ring.clj:18)
        at bidi.ring$fn__18735$G__18730__18744.invoke(ring.clj:12)
        at puppetlabs.comidi$make_handler$fn__20441.invoke(comidi.clj:249)
        at puppetlabs.orchestrator.api$wrap_base_url$fn__37352.invoke(api.clj:145)
        at puppetlabs.metrics.http$fn__31068$wrap_with_request_metrics__31073$fn__31077$fn__31079$fn__31080$fn__31081.invoke(http.clj:152)
        at puppetlabs.metrics.http.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
        at com.codahale.metrics.Timer.time(Timer.java:101)
        at puppetlabs.metrics.http$fn__31068$wrap_with_request_metrics__31073$fn__31077$fn__31079$fn__31080.invoke(http.clj:152)
        at puppetlabs.metrics.http.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
        at com.codahale.metrics.Timer.time(Timer.java:101)
        at puppetlabs.metrics.http$fn__31068$wrap_with_request_metrics__31073$fn__31077$fn__31079.invoke(http.clj:148)
        at puppetlabs.comidi$fn__20498$wrap_with_route_metadata__20503$fn__20504$fn__20506.invoke(comidi.clj:336)
        at puppetlabs.orchestrator.api$permission_check$fn__37344.invoke(api.clj:118)
        at puppetlabs.rbac_client.middleware.authentication$wrap_block_anonymous_access$fn__37315.invoke(authentication.clj:131)
        at puppetlabs.rbac_client.middleware.authentication$wrap_token_access_STAR_$fn__37300.invoke(authentication.clj:69)
        at puppetlabs.orchestrator.api$wrap_exception_handling$fn__37336.invoke(api.clj:102)
        at puppetlabs.orchestrator.api$wrap_catch_500s$fn__37347.invoke(api.clj:130)
        at puppetlabs.i18n.core$locale_negotiator$fn__124.invoke(core.clj:357)
        at puppetlabs.trapperkeeper.services.webserver.jetty9_core$ring_handler$fn__24031.invoke(jetty9_core.clj:423)
        at puppetlabs.trapperkeeper.services.webserver.jetty9_core.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1129)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1065)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
        at org.eclipse.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:299)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:497)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:748)

And this all the time:

2017-07-06 13:49:46,872 WARN  [p.j.pool] orchestrator - Error while attempting to connect to database, retrying.
java.sql.SQLTransientConnectionException: orchestrator - Connection is not available, request timed out after 30000ms.
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:195)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:147)
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83)
        at puppetlabs.jdbc_util.pool$wrap_with_delayed_init$fn__35210$fn__35211.invoke(pool.clj:126)
        at puppetlabs.jdbc_util.pool$wrap_with_delayed_init$fn__35210.invoke(pool.clj:124)
        at clojure.core$binding_conveyor_fn$fn__4676.invoke(core.clj:1938)
        at clojure.lang.AFn.call(AFn.java:18)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.postgresql.util.PSQLException: SSL error: Received fatal alert: decrypt_error
        at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:67)
        at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:351)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:137)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:215)
        at org.postgresql.Driver.makeConnection(Driver.java:406)
        at org.postgresql.Driver.connect(Driver.java:274)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:314)
        at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:171)
        at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:436)
        at com.zaxxer.hikari.pool.HikariPool.access$500(HikariPool.java:65)
        at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:567)
        at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:560)
        ... 4 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: decrypt_error
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
        at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2023)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
        at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:62)
        ... 18 common frames omitted
2017-07-06 13:50:05,344 WARN  [p.j.core] Database status check timed out after 4 seconds.
2017-07-06 13:50:05,344 WARN  [p.j.core] Status check of db failed with error:
java.lang.InterruptedException: null
        at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
        at java.util.concurrent.FutureTask.get(FutureTask.java:204)
        at clojure.core$deref_future.invokeStatic(core.clj:2210)
        at clojure.core$future_call$reify__6962.deref(core.clj:6688)
        at clojure.core$deref.invokeStatic(core.clj:2232)
        at clojure.core$deref.invoke(core.clj:2214)
        at puppetlabs.jdbc_util.pool$wrap_with_delayed_init$reify__35232.getConnection(pool.clj:159)
        at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:307)
        at clojure.java.jdbc$get_connection.invoke(jdbc.clj:197)
        at clojure.java.jdbc$db_query_with_resultset.invokeStatic(jdbc.clj:878)
        at clojure.java.jdbc$db_query_with_resultset.invoke(jdbc.clj:847)
        at clojure.java.jdbc$query.invokeStatic(jdbc.clj:915)
        at clojure.java.jdbc$query.invoke(jdbc.clj:884)
        at clojure.java.jdbc$query.invokeStatic(jdbc.clj:898)
        at clojure.java.jdbc$query.invoke(jdbc.clj:884)
        at puppetlabs.jdbc_util.core$db_up_QMARK_$fn__31952.invoke(core.clj:57)
        at clojure.core$binding_conveyor_fn$fn__4676.invoke(core.clj:1938)
        at clojure.lang.AFn.call(AFn.java:18)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
2017-07-06 13:50:10,107 WARN  [p.j.core] Database status check timed out after 4 seconds.
2017-07-06 13:50:10,107 WARN  [p.j.core] Status check of db failed with error:

and so on and so forth seemingly forever

If you try and use the 'run puppet' function from the console you get the following error message in a red bar:

Running Puppet failed to start: Request for foo.example failed. Message: Could not make a connection to the database

Fix:

  • Stop pe-orchestration-services
  • Stop pxp-agent
  • rm /etc/puppetlabs/orchestration-services/ssl/*
  • Start pe-orchestration-services
  • Start pxp-agent
@jtmfam

This comment has been minimized.

Copy link

commented Jul 3, 2019

The solution is right (meaning removing the certs.) I had to stop and start more than just pe-orchestration-services and pxp-agent for version 2018.1.8. I stopped all puppet related services, remove the certificates, and restarted them all--problem solved. Thanks for sharing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.