In general, the logging situation on this service needs improvement, since logs cannot be accessed via AWS UI or directly via SSH. It can only be accessed by copying it out of the running docker instance.
This error occurs a few time everyday, and seems to be related to weird request bodies/parameters.
Stacktrace below:
2019-10-19 14:28:55,190 - [ERROR] - from application in application-akka.actor.default-dispatcher-10
! @7dh2lcidd - Internal server error, for (GET) [/api/branches/-1/parcels/unallocated?token=1c3c2f8b771c25b32371d34bcf340630&expToken=1c3c2f8b771c25b32371d34bcf340630] ->
play.api.UnexpectedException: Unexpected exception[MismatchedInputException: No content to map due to end-of-input
at [Source: (String)""; line: 1, column: 0]]
at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:347)
at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:267)
at play.core.server.AkkaHttpServer$$anonfun$1.applyOrElse(AkkaHttpServer.scala:448)
at play.core.server.AkkaHttpServer$$anonfun$1.applyOrElse(AkkaHttpServer.scala:446)
at scala.concurrent.Future.$anonfun$recoverWith$1(Future.scala:417)
at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input
at [Source: (String)""; line: 1, column: 0]
at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4145)
at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3971)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2276)
at play.api.libs.json.jackson.JacksonJson$.parseJsValue(JacksonJson.scala:255)
at play.api.libs.json.StaticBinding$.parseJsValue(StaticBinding.scala:16)
at play.api.libs.json.Json$.parse(Json.scala:171)
at controllers.Branches.$anonfun$getBranchCodeFromId$2(Branches.scala:116)
at scala.util.Success.$anonfun$map$1(Try.scala:255)
at scala.util.Success.map(Try.scala:213)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
... 12 common frames omitted
Happens in spikes of up to ~50 every few days. There's nothing obvious in the code that should cause this. The logs also didn't reveal much.
Clive claims that use of this endpoint will be phased out in future, so perhaps we can just exclude it from our stats.
Happens in spikes of ~30 every few days. Seems to happen at the same time as token check spikes. Wouod be interesting to see if the 2 directly related.
Nothing in logs. The most obvious point of failure in the code is a ather large SQL query that may have been written under assumptions that are no longer valid.
Seems to have been solved by an earlier release.