martinklepsch [12:05 PM] I have a blocking function that I want to wrap in a timeout — is there something in Java I could easily reach for?
martinklepsch [12:11 PM] Currently reaching for core.async but I'm thinking there must be something simple than this:
(defmacro with-timeout [ms & body]
(let [c (async/thread-call #(do ~@body))]
(let [[v ch] (alts!! [c (timeout 100)])]
(if-not (= ch c)
(throw (ex-info "Operation timed out" {}))
v))))
hiredman [12:14 PM] well, that macro is definitely not correct, because it is doing the call instead of making the call
[12:14] well, that macro is definitely not correct, because it is doing the call instead of emitting code that does it
[12:15] you can just use future, and deref, as of I dunno, some clojure release awhile back, deref of futures can take extra arguments for timing out
dialelo [12:16 PM] i'd go with future + deref with timeout as suggested by @hiredman
martinklepsch [12:16 PM] @hiredman: yeah that macro is "wip" 😉
hiredman [12:16 PM] previous to that, you could just use future and call the java .get with a time out
hans [12:24 PM] @martinklepsch: what happens to the thread that you're starting when a timeout occurs?
martinklepsch [12:24 PM] @hiredman: @dialelo thanks that sounds much better indeed. Just for my own macro-fu:
(defmacro with-timeout [ms & body]
`(let [c# (async/thread-call #(do ~@body))]
(let [[v# ch#] (async/alts!! [c# (async/timeout ~ms)])]
(if-not (= ch# c#)
(throw (ex-info "Operation timed out" {}))
v#))))
this is better?
[12:28] @hans: not sure I understand?
hans [12:29 PM] @martinklepsch: I was just wondering whether you're prepared to let the thread continue to run, or whether you could run into the problem that you're starting more and more threads facing timeout conditions. @martinklepsch: It could be completely okay with your use case, but it could also be preferable to handle the timeout situation at a lower level rather than using threads.
martinklepsch [12:35 PM] @hans: in my case on timeout I'll throw an exception which will be caught & invalidate the resource that has caused the timeout
hans [12:36 PM] @martinklepsch: so you're okay with the fact that the thread may be running forever?
martinklepsch [12:41 PM] @hans well that sounds bad... :simple_smile: I guess I should terminate the thread? fwiw this should probably done in the normal case as well no? (sorry my knowledge about thread handling is ... lacking)
hans [12:41 PM] @martinklepsch: it gets worse, because if you think that you can just terminate a thread, you're wrong :simple_smile: @martinklepsch: so effectively, you need to either accept that your threads are leaking, or you need to find a way to detect the timeout in a synchronous fashion, without creating a thread just for the purpose of being able to wait. modern i/o subsystems usually support safe timeout methods, but it won't be as simple as wrapping some arbitrary call in a with-timeout form. @martinklepsch: for me, it was a hard to accept fact that i cannot terminate a thread from another thread in a safe fashion, but i eventually gave in. @martinklepsch: (i'll stop after this) the reason why you cannot safely terminate a thread from another thread is that the system does not perform its resource control on a per-thread fashion. consider that the thread might hold a lock when you terminate it - how would you ever get the lock released?
martinklepsch [12:55 PM] @hans: thanks, makes sense (internet broke) so what did you end up with? also wondering, given these issues, how do @dialelo and @hiredman deal with leaking threads? is the deref with a timeout/.get magically cleaning things up correctly?
dm3 [12:57 PM] usually you use a thread pool for this kind of work
hans [12:57 PM] if i need timeouts, i ask myself how to implement them in a synchronous fashion first. i don't usually think that i could have a with-timeout macro that'd help me avoid going into the details.
dm3 [12:57 PM] this way you control the number of threads that get created
hans [12:58 PM] @dm3: right, a thread pool with a fixed number of threads may be helpful, as one can monitor its utilization and, say, issue a warning or just die if the pool is exhausted.
dm3 [12:58 PM]
e.g. ztellman/manifold allows passing in a custom executor service
to a deferred/future
hans [12:59 PM] but essentially, using a thread pool is just pushing the problem to another level. i'd begin by thinking about how the timeout can be detected synchronously. @martinklepsch: what kind of operation are you dealing with that can take too long?
martinklepsch [1:00 PM] @hans: @dm3 right so with the core.async approach there would already be a threadpool if I understand correctly
dm3 [1:00 PM] it would be global to the whole application though IIRC
martinklepsch [1:02 PM] @hans: database operations. Sometimes connections just become stale and won't do anything anymore (I think this is caused by an issue in the DB driver which will be fixed soon) so after thats solved mostly to make sure things break instead of taking forever :simple_smile:
hans [1:02 PM] as i said, the thread pool only pushes the problem out, it does not actually solve it. if you could run into the situation that a thread could hang forever, the thread pool will just push the detection of that issue to a later point in time, or prevent you from detecting it at all. that may be a viable strategy, but it'd better be a conscious decision :simple_smile:
dm3 [1:03 PM] totally agree - you need to have a timeout strategy defined just trying to point out that understanding the execution model is essential for any serious work
hans [1:03 PM] @martinklepsch: for that kind of thing, it may make sense as an intermediate measure. just make sure that with-timeout is not promoted to your utils.clj and becomes a regular thing that people use to avoid thinking about timeouts properly.
[1:04] @martinklepsch: i'd really be interested in knowing why a database connection becomes stale and can't be used anymore. for me, that'd be a perfect procrastination issue.
martinklepsch [1:05 PM] @hans: There's some bogus core.async code. It's not really the connection but rather the driver code. Not much interesting..
hans [1:05 PM] burp :simple_smile: