Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
18:02 <stodge> Does write performance degrade as the size of the database increases?
I'm talking millions of writes.
18:03 <justinsheehy> stodge: generally, no. bitcask is designed to have roughly constant
write time with regard to database size.
18:03 <stodge> Thanks
18:04 <stodge> I'm researching nosql solutions for a data warehouse. I want to
dump data into it and then use map/reduce to retrieve data later for statistics etc
18:05 <stodge> I need something that will happily whirr away in the background
storing data with writes every second
18:07 <justinsheehy> so far, sounds like a plausibly riak-shaped kind of thing --
though there are some analytics systems that may be better optimized for the data warehouse
style querying, riak can do some of that and can certainly consumer lots of continuous writes.
18:08 <stodge> ok
18:09 <stodge> I also want to provide 1..n web servers to let users access and query
historical data, so I need a distributed db
18:10 <stodge> I have a requirement that may not be suitable to nosql solutions.
A lot of my data is essentially a snapshot; a customer can wind the clock back and retrieve
data every second starting from say 7-Feb-2011 08:00:00 and every second the browser will
request the current snapshot at that time.
18:11 <stodge> The customer can recreate the 'system' state at 08:00 on the 7th and
can playback the system state at 1s increments.
18:11 <justinsheehy> well, there's nothing built-in to riak (or most systems I know)
to do exactly that, but there are different ways to build it
18:12 <justinsheehy> I have to run, hit up the mailing list if you don't find people
here sometime when you need help.
18:12 <stodge> Ok thanks, will do
18:19 <Wuher> stodge: No idea if it would really be suited for your needs,
but have a look at logstash
18:19 <stodge> Will do thanks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.