h query syntax discussion

gistfile1.txt

11:58:31 AM	judell  nickstenn: thoughts on h/issues/2095 re: 1) syntax normalization across /stream and /api, 2) behavior normalization (search works differently)
11:59:11 AM	nickstenn  h#2095
11:59:12 AM	vannevar  Normalize URL syntax and query behavior across /api and /stream: https://github.com/hypothesis/h/issues/2095
11:59:23 AM	 GitHub64 joined the channel.
11:59:23 AM	-GitHub64-  [h] nickstenning force-pushed 1916-stream-cards from 1862529 to fe5cd6c: https://github.com/hypothesis/h/commits/1916-stream-cards
11:59:23 AM	-GitHub64-  h/1916-stream-cards fe5cd6c Gergely Ujvari: Fix controller tests
11:59:23 AM	 GitHub64 left the channel.
11:59:49 AM	tilgovi  judell: I'd like to see that come as a natural consequence of unifying those code paths completely
12:00:15 PM	tilgovi  there's some interesting query parsing stuff going on, client and server, that can likely be vastly simplified
12:00:35 PM	judell  so now it's split between python and js?
12:00:49 PM	tilgovi  yeah
12:01:03 PM	tilgovi  and in the back end, too, it's split into different code paths
12:01:08 PM	nickstenn  tilgovi: hahahahahaha
12:01:12 PM	nickstenn  https://github.com/hypothesis/h/issues/2095#issuecomment-87056763
12:01:29 PM	nickstenn  in summary, judell: YES.
12:01:45 PM	tilgovi  Wow
12:02:09 PM	tilgovi  Next time someone asks me whether it's challenging to work with people on the other side of the world I'll just explain we're telepathic.
12:02:21 PM	nickstenn  that is really uncannily similar
12:02:50 PM	nickstenn  is there any particular reason we're parsing the query on the client-side at the moment?
12:02:52 PM	nickstenn  historical accident?
12:03:12 PM	nickstenn  like, the bit I *really* don't get is where we parse the query on the client side
12:03:20 PM	nickstenn  send it down the websocket as a data structure
12:03:31 PM	nickstenn  and then transform it into a different data structure at the other end
12:05:43 PM	judell  fwiw, in every conversation we've had with prospective users i've presumed simple tag search, possibly for compound tags, that would behave the same in /stream and /api. i get that unification is the right long-term thing, but if that one use case could be normalized it'd be nice.
12:06:25 PM	nickstenn  are you talking with users that are using the API?
12:06:50 PM	tilgovi  csillag: the whole way we're doing this UI sync business is crazy
12:06:57 PM	judell  users that presume the query capabilities i'm describing
12:07:20 PM	nickstenn  right, but that's about being able to search for tags and conjunctions of tags, yes?
12:07:24 PM	tilgovi  I feel like we should just have the AnnotationUI API and during the application configuration, if we're in an iframe we should decorate the service so sysnc
12:07:33 PM	nickstenn  not actually about unification between API and the stream search?
12:07:40 PM	judell  btw can you show me /api and /search urls that do an exact-match query for tag AND tag?
12:08:33 PM	nickstenn  that is a great question
12:08:37 PM	nickstenn  let me have a look
12:09:12 PM	judell  nickstenn: you're right that unification is not an ask, i'm just expressing my sense, as a user of many apis, that learning them goes much better when the user-accessible URL-line and the API-accessible URL match
12:09:16 PM	tilgovi  so, I just looked at the front-end code
12:09:18 PM	tilgovi  and tags is an AND search
12:09:24 PM	judell  syntax?
12:09:33 PM	tilgovi  not sure
12:09:36 PM	nickstenn  judell: yep, totally agreed on that -- I'm just checking what the priority is
12:09:44 PM	csillag  Tilgovi: I'm off for weekend. Later.
12:09:53 PM	tilgovi  csillag: have a nice weekend
12:10:06 PM	nickstenn  judell: I think "tag:hypothesisrelated tag:api" is what you're looking for
12:10:21 PM	tilgovi  https://hypothes.is/stream?q=tag:hypothesisrelated%20tag:api
12:11:07 PM	judell  ok that helps for now, thx.
12:11:14 PM	judell  and the /api equivalent?
12:11:16 PM	tilgovi  this seems to work too
12:11:16 PM	tilgovi  https://hypothes.is/api/search?tags=hypothesisrelated&tags=api
12:11:27 PM	nickstenn  yeesh
12:11:29 PM	nickstenn  that's horrible
12:11:36 PM	 Sebastien-L has quit: Ping timeout: 272 seconds.
12:11:37 PM	nickstenn  who wrote *that* API
12:11:40 PM	 nickstenn ducks
12:11:41 PM	tilgovi  lol
12:12:28 PM	nickstenn  tilgovi: are we agreed that lucene key:value is the direction we should go for both?
12:12:40 PM	tilgovi  probably
12:12:48 PM	tilgovi  It's a common syntax
12:13:08 PM	nickstenn  we can always build UI on top that generates that
12:13:10 PM	tilgovi  in both APIs and user input
12:13:26 PM	tilgovi  you're talking about the query parser that understands AND and OR, too, yeah?
12:13:47 PM	nickstenn  yep
12:14:04 PM	tilgovi  yes
12:14:36 PM	nickstenn  the nice thing about that is that it's basically elasticsearch's native query syntax: http://www.elastic.co/guide/en/elasticsearch/reference/1.x/query-dsl-query-string-query.html
12:14:49 PM	judell  AND matters mostly. OR almost never
12:15:22 PM	 badon joined the channel.
12:15:31 PM	judell  any consistent syntax is fine
12:15:53 PM	judell  needs to be clear how to quote and conjoin params
12:16:02 PM	nickstenn  right
12:16:07 PM	nickstenn  well I think we all know we should go that way
12:16:19 PM	nickstenn  it's just a question of priority
12:16:27 PM	nickstenn  luckily, we have someone to worry about that now :)
12:16:35 PM	judell  :-)
12:17:41 PM	judell  last thing for now, /api equivalent to https://hypothes.is/stream?q=tag:hypothesisrelated%20tag:api is ... ?
12:18:02 PM	nickstenn  judell: scroll up -- tilgovi gave an example up there ^^
12:18:31 PM	judell  sorry, missed, that, got it, thanks, see ya later
12:19:55 PM	 travis-ci joined the channel.
12:19:56 PM	travis-ci  hypothesis/h#5655 (1916-stream-cards - 1862529 : Gergely Ujvari): The build has errored.
12:19:56 PM	travis-ci  Change view : https://github.com/hypothesis/h/compare/3cf998454f29...18625297ff1d
12:19:56 PM	travis-ci  Build details : http://travis-ci.org/hypothesis/h/builds/56135909
12:19:56 PM	 travis-ci left the channel.
12:21:12 PM	judell  actually 1 more thing (ghost of steve jobs...) https://hypothes.is/api/search?tags=%27h-examples%27 is defaulting to some kind of lucene fuzziness?
12:22:22 PM	nickstenn  hmm
12:22:31 PM	nickstenn  not sure there's anything we can do about that right this second, I'm afraid
12:22:43 PM	nickstenn  we'd probably have to change the tokenizer for the tags field
12:23:01 PM	judell  ok, just want to understand what's going on
12:23:24 PM	nickstenn  so your query (and/or the field when indexed) is broken into tokens
12:23:34 PM	nickstenn  i suspect the tags field is breaking on "-"
12:23:40 PM	judell  even if quoted, apparently
12:24:08 PM	nickstenn  so your search ends up being a search for "h OR examples"
12:24:25 PM	nickstenn  yes, it's not great
12:24:47 PM	judell  is there another way to quote the term?
12:25:25 PM	judell  put it a different way, should lucene docs be the guide for now?
12:25:41 PM	nickstenn  no, I'm afraid it's much worse than that
12:26:01 PM	nickstenn  if this were a direct lucene API quoting would (I'm pretty sure) do what you want
12:26:27 PM	 GitHub18 joined the channel.
12:26:27 PM	-GitHub18-  [h] tilgovi force-pushed 800-fancy-selected-annotations from 932ae26 to 3459915: https://github.com/hypothesis/h/commits/800-fancy-selected-annotations
12:26:27 PM	-GitHub18-  h/800-fancy-selected-annotations 4d0d820 csillag: Add emphasis to selected annotations...
12:26:28 PM	-GitHub18-  h/800-fancy-selected-annotations 3459915 csillag: Tweak color
12:26:28 PM	 GitHub18 left the channel.
12:26:57 PM	judell  ok, enough for now, i'll add this to the issue and move on, thx
12:27:01 PM	nickstenn  but I don't think there's any way of getting what you want here short of "tags=h&tags=examples", which I freely confess is absurd
12:27:19 PM	tilgovi  multiple query parameters totally legit!
12:27:31 PM	tilgovi  MuliDict ftw
12:27:46 PM	nickstenn  :(