Skip to content

Instantly share code, notes, and snippets.

@jessereynolds

jessereynolds/gist:5082b82216989d9e0bba7c1ca1828963

Last active July 3, 2019 17:38
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jessereynolds/5082b82216989d9e0bba7c1ca1828963 to your computer and use it in GitHub Desktop.
Save jessereynolds/5082b82216989d9e0bba7c1ca1828963 to your computer and use it in GitHub Desktop.
Download ZIP
pe-orchestration-services sql ssl error after restore PE 2017.2.2
Raw
gistfile1.md

pe-orchestration-services sql ssl error after restore PE 2017.2.2

2017-07-06 13:02:36,858 ERROR [p.o.api] Failed to make a connection to the database
java.sql.SQLTransientConnectionException: Timeout waiting for the database pool to become ready.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at clojure.lang.Reflector.invokeConstructor(Reflector.java:180)
        at puppetlabs.jdbc_util.pool$wrap_with_delayed_init$reify__35232.getConnection(pool.clj:160)
        at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:307)
        at clojure.java.jdbc$get_connection.invoke(jdbc.clj:197)
        at clojure.java.jdbc$db_query_with_resultset.invokeStatic(jdbc.clj:878)
        at clojure.java.jdbc$db_query_with_resultset.invoke(jdbc.clj:847)
        at clojure.java.jdbc$query.invokeStatic(jdbc.clj:915)
        at clojure.java.jdbc$query.invoke(jdbc.clj:884)
        at clojure.java.jdbc$query.invokeStatic(jdbc.clj:898)
        at clojure.java.jdbc$query.invoke(jdbc.clj:884)
        at puppetlabs.orchestrator.job$list_jobs.invokeStatic(job.clj:94)
        at puppetlabs.orchestrator.job$list_jobs.invoke(job.clj:87)
        at puppetlabs.orchestrator.api.jobs$list_jobs.invokeStatic(jobs.clj:106)
        at puppetlabs.orchestrator.api.jobs$list_jobs.invoke(jobs.clj:92)
        at puppetlabs.orchestrator.api.jobs$app$fn__37263.invoke(jobs.clj:196)
        at ring.middleware.keyword_params$wrap_keyword_params$fn__17935.invoke(keyword_params.clj:35)
        at ring.middleware.params$wrap_params$fn__16133.invoke(params.clj:64)
        at bidi.ring$fn__18750.invokeStatic(ring.clj:21)
        at bidi.ring$fn__18750.invoke(ring.clj:18)
        at bidi.ring$fn__18735$G__18730__18744.invoke(ring.clj:12)
        at puppetlabs.comidi$make_handler$fn__20441.invoke(comidi.clj:249)
        at puppetlabs.orchestrator.api$wrap_base_url$fn__37352.invoke(api.clj:145)
        at puppetlabs.metrics.http$fn__31068$wrap_with_request_metrics__31073$fn__31077$fn__31079$fn__31080$fn__31081.invoke(http.clj:152)
        at puppetlabs.metrics.http.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
        at com.codahale.metrics.Timer.time(Timer.java:101)
        at puppetlabs.metrics.http$fn__31068$wrap_with_request_metrics__31073$fn__31077$fn__31079$fn__31080.invoke(http.clj:152)
        at puppetlabs.metrics.http.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
        at com.codahale.metrics.Timer.time(Timer.java:101)
        at puppetlabs.metrics.http$fn__31068$wrap_with_request_metrics__31073$fn__31077$fn__31079.invoke(http.clj:148)
        at puppetlabs.comidi$fn__20498$wrap_with_route_metadata__20503$fn__20504$fn__20506.invoke(comidi.clj:336)
        at puppetlabs.orchestrator.api$permission_check$fn__37344.invoke(api.clj:118)
        at puppetlabs.rbac_client.middleware.authentication$wrap_block_anonymous_access$fn__37315.invoke(authentication.clj:131)
        at puppetlabs.rbac_client.middleware.authentication$wrap_token_access_STAR_$fn__37300.invoke(authentication.clj:69)
        at puppetlabs.orchestrator.api$wrap_exception_handling$fn__37336.invoke(api.clj:102)
        at puppetlabs.orchestrator.api$wrap_catch_500s$fn__37347.invoke(api.clj:130)
        at puppetlabs.i18n.core$locale_negotiator$fn__124.invoke(core.clj:357)
        at puppetlabs.trapperkeeper.services.webserver.jetty9_core$ring_handler$fn__24031.invoke(jetty9_core.clj:423)
        at puppetlabs.trapperkeeper.services.webserver.jetty9_core.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1129)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1065)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
        at org.eclipse.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:299)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:497)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:748)

And this all the time:

2017-07-06 13:49:46,872 WARN  [p.j.pool] orchestrator - Error while attempting to connect to database, retrying.
java.sql.SQLTransientConnectionException: orchestrator - Connection is not available, request timed out after 30000ms.
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:195)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:147)
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83)
        at puppetlabs.jdbc_util.pool$wrap_with_delayed_init$fn__35210$fn__35211.invoke(pool.clj:126)
        at puppetlabs.jdbc_util.pool$wrap_with_delayed_init$fn__35210.invoke(pool.clj:124)
        at clojure.core$binding_conveyor_fn$fn__4676.invoke(core.clj:1938)
        at clojure.lang.AFn.call(AFn.java:18)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.postgresql.util.PSQLException: SSL error: Received fatal alert: decrypt_error
        at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:67)
        at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:351)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:137)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:215)
        at org.postgresql.Driver.makeConnection(Driver.java:406)
        at org.postgresql.Driver.connect(Driver.java:274)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:314)
        at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:171)
        at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:436)
        at com.zaxxer.hikari.pool.HikariPool.access$500(HikariPool.java:65)
        at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:567)
        at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:560)
        ... 4 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: decrypt_error
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
        at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2023)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
        at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:62)
        ... 18 common frames omitted
2017-07-06 13:50:05,344 WARN  [p.j.core] Database status check timed out after 4 seconds.
2017-07-06 13:50:05,344 WARN  [p.j.core] Status check of db failed with error:
java.lang.InterruptedException: null
        at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
        at java.util.concurrent.FutureTask.get(FutureTask.java:204)
        at clojure.core$deref_future.invokeStatic(core.clj:2210)
        at clojure.core$future_call$reify__6962.deref(core.clj:6688)
        at clojure.core$deref.invokeStatic(core.clj:2232)
        at clojure.core$deref.invoke(core.clj:2214)
        at puppetlabs.jdbc_util.pool$wrap_with_delayed_init$reify__35232.getConnection(pool.clj:159)
        at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:307)
        at clojure.java.jdbc$get_connection.invoke(jdbc.clj:197)
        at clojure.java.jdbc$db_query_with_resultset.invokeStatic(jdbc.clj:878)
        at clojure.java.jdbc$db_query_with_resultset.invoke(jdbc.clj:847)
        at clojure.java.jdbc$query.invokeStatic(jdbc.clj:915)
        at clojure.java.jdbc$query.invoke(jdbc.clj:884)
        at clojure.java.jdbc$query.invokeStatic(jdbc.clj:898)
        at clojure.java.jdbc$query.invoke(jdbc.clj:884)
        at puppetlabs.jdbc_util.core$db_up_QMARK_$fn__31952.invoke(core.clj:57)
        at clojure.core$binding_conveyor_fn$fn__4676.invoke(core.clj:1938)
        at clojure.lang.AFn.call(AFn.java:18)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
2017-07-06 13:50:10,107 WARN  [p.j.core] Database status check timed out after 4 seconds.
2017-07-06 13:50:10,107 WARN  [p.j.core] Status check of db failed with error:

and so on and so forth seemingly forever

If you try and use the 'run puppet' function from the console you get the following error message in a red bar:

Running Puppet failed to start: Request for foo.example failed. Message: Could not make a connection to the database

Fix:

  • Stop pe-orchestration-services
  • Stop pxp-agent
  • rm /etc/puppetlabs/orchestration-services/ssl/*
  • Start pe-orchestration-services
  • Start pxp-agent
@jtmfam
Copy link

jtmfam commented Jul 3, 2019

The solution is right (meaning removing the certs.) I had to stop and start more than just pe-orchestration-services and pxp-agent for version 2018.1.8. I stopped all puppet related services, remove the certificates, and restarted them all--problem solved. Thanks for sharing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment