gistfile1.txt

    18:29:19   karissa | noffle: relatedly, i want to get your thoughts on how to think about kappa-cores/hypercores with media attached. we did some brainstorming on it in
                       | the cabal meeting
    18:29:34    noffle | like, media in the hypercore?
    18:29:46   karissa | https://hackmd.io/BeZ49Q2ARZavH1v2JbbJKw#
    18:30:04   karissa | under 'how do we do binary stuff in cabal? ideas below '  
    18:30:06         * | noffle reads
    18:30:35    noffle | wow, really good notes
    18:32:02   karissa | yeah it was a fun meeting!
    18:32:32   karissa | seeing a pattern here that could overlap between mapeo and cabal --> media attached to messages/media attached to observations
    18:32:52    noffle | nod
    18:32:58   karissa | with also some user control around 'sparseness' of it
    18:33:10    noffle | I like the idea of keeping blobs in seprate hypercores
    18:33:59   karissa | wondering if message -> blob store or device(user) -> blob store makes more sense. i think mapeo uses the latter right now
    18:33:59    noffle | one cost that can seem invisible is that you need to manage a swarm around EACH hypercore, so with N blobs, if you do 1 blob per hypercore that's N
                       | swarms. but if it was, say, per-user, that's M swarms (assuming M users)
    18:34:26    noffle | maintaining an ongoing membership in a swarm costs network connections
    18:34:27   karissa | yeah that's a limitation i was thinking of too.
    18:34:37   karissa | which makes me lean towards device/user-> blob store
    18:34:54    noffle | what we could do is: one *multifeed* of blobs per cabal
    18:34:59    noffle | so it's still M hypercores
    18:35:03    noffle | but one swarm for blobs
    18:35:08   karissa | oh nice
    18:35:08    noffle | one swarm for chat data
    18:35:38    noffle | it means expsoing sparse replication on multifeed
    18:35:43    noffle | at some point
    18:35:52   karissa | will putting blob data in a hypercore get slow after a while? i think hyperdrive writes it to an external blob store right?
    18:36:08    noffle | slow how?
    18:36:12    noffle | like, appending locally? syncing?
    18:36:40    noffle | the hyperdb-based hyperdrive writes to a hypercore
    18:36:47   karissa | not sure exactly where the slowdown happens, but I remember this being a key point of how hyperdrive can handle upwards of GB/TB
    18:36:47    noffle | not sure about the single-writer version
    18:36:58    noffle | oh interesting, hadn't heard about this
    18:37:06   karissa | i think it writes the ranges to the hypercore but not the blob data
    18:37:11   karissa | kinda like sqlite blobs
    18:37:26    noffle | is this a hypercore issue or hyperdrive issue?
    18:38:09   karissa | exactly thats the question kinda raised in the cabal meeting, whether to have a 'media message' link to a hyperdrive
    18:38:14   karissa | or to a hypercore
    18:38:33   karissa | cause you can imagine wanting to share like, a folder or bunch of images
    18:38:40   karissa | or having a user account or cabal have the 'media'
    18:39:12    noffle | oh sorry I meant is that scaling issue on hypercore or hyperdrive?
    18:40:03   karissa | ah i think it's a limitation of a merkle dag, which is why git is slow, iirc
    18:40:31    noffle | huh. wouldn't that be a limit on # of entries then, not on total amount of data stored?
    18:40:36    noffle | one merkle entry per node
    18:40:47    noffle | node/file
    18:40:54    noffle | would be interested in more bg on that   
    18:41:09   karissa | yeah it'd be great to have this documentation on hypercore
    18:41:16   karissa | perhaps that's something you and i could work on one day
    18:41:23    noffle | it sounds like something that hyper{core,drive} wants to fix, so maybe it's OK to rely on it & expect it to be fixed upstream as dat grows
    18:41:23   karissa | 'book sprint'
    18:41:31    noffle | I like docs :)
    18:42:16   karissa | noffle: sorry i meant its not really a limitation in hyperdrive as it's architected, and it can handle lots of data -- it does currently have an
                       | issue with tons of *files* though, (like, thousands/hundreds of thousands in the same level)
    18:42:19    noffle | yeah maybe we could allowing one-blob links like "KEY@SEQ" but also directory shares, which are maybe just "KEY/HASH"
    18:42:31         * | noffle nod
    18:42:42    noffle | "we could allow"
    18:43:17    noffle | hyperblob://<KEY|HASH> and dat://<KEY/HASH> maybe
    18:43:18   karissa | noffle: it would be cool to imagine media messages as just links to the outside world, and if a client supports reading it, that's great. so we can
                       | evolve this and try different things
    18:43:22    noffle | yeah!
    18:43:30    noffle | unixy
    18:43:47    noffle | ha we should've had this chat in #cabal-club
    18:43:48   karissa | yeah. also keep messages as tiny as possible in case a particular cabal gets super popular
    18:43:54    noffle | yes
    18:44:12   karissa | haha. well anyway, it relates to Mapeo because I *think* we want to do similar kinds of stuff. and probably other apps..
    18:45:05    noffle | nod
    18:45:06    noffle | good chat
    18:45:30    noffle | I'm going to go over these meeting minutes & will include these notes
    18:45:36   karissa | rad