python - Appropriate data structure for social networks -
for college project crawling social networks perform data analysis on datasets afterwards.
now crawlers work (more or less), wondering best way store data is. crawling personal information profiles network graph whole able perform network analysis.
first thought save data in regular database. but, since have save friends of every profile , number of friends varies couple of friends 5,000, thought saving in database using orm (e.g. sqlalchemy), lead redundant connections between profiles though. idea save normal profile data (interests, number of friends, city, age etc.) in regular database , save network graph in separate data structure, e.g. in networkx graph, or using network graph database everything.
what kind of database should use crawling social networks? there standard way of saving such data?
Comments
Post a Comment