elasticsearch - Cost effective and timely system to do Business Logic queries (batch processing) on weekly snapshots of a 15 gb MySQL database? -
we want batch operations on series of mysql snapshots, unsure of architecture purseu. have on 100 snapshots of 15 gb database.
we debating architecture use wouldn't embarrassing slow, not huge overhead (time-wise or monetarily). here have considered:
loading each db open stack vm doing our query saving data found off somewhere in redis instance can use python dump reports. (this slow no?)
loading elastic search , perform queries via elastic search (this not allow robustness/logic of querying want.)
we have looked @ hive or prestodb require large amounts of hardware?
on small scale of our tables this:
client: client_id (primary key) name address city device: device_id (primary key) name location service: service_id (primary key) name price client_id device_id (there no relationships between tables)
here example of want do:
look @ services have increased in price on last month of snapshots (4 snapshots). client's location of client , device name each service increased. dump csv or other visualizing software kibana.
Comments
Post a Comment