by Quade (Posted Sun, 21 Sep 2014 08:45:40 GMT)
I don't think I qualify as a "team".
2 threads might help. Or it might just bottleneck in a different place. I'll have to try. 2 threads will probably only be better if the problem is CPU load. If the problem is disk IO then more threads won't help. One thought might be to split the work into a "reading and turning into files" part and another into the "compacting and writing to DB part. A new version of Sqlite, the database seems to have improved import speed significantly in 6.6.
I'm not sure you all have a sense for how much data we're talking about. Pick one group, a.b.hdtv for instance. It's probably got 2 billion posts in it. It's:
2 * 10^9 * 400 bytes = 800 Gigs of data. That's just one group. With Compression that's probably only an 80 gig download but, it's still 800 gigs of data that needs to processed.
Boneless has > 6 billion posts in it.
It's valuable data that represents 100's of gigs of downloads. You might want to back it up to external storage from time to time.
Read Main Topic
The NB team missed an important use case when they designed this.
I don't think I qualify as a "team".
2 threads might help. Or it might just bottleneck in a different place. I'll have to try. 2 threads will probably only be better if the problem is CPU load. If the problem is disk IO then more threads won't help. One thought might be to split the work into a "reading and turning into files" part and another into the "compacting and writing to DB part. A new version of Sqlite, the database seems to have improved import speed significantly in 6.6.
I'm not sure you all have a sense for how much data we're talking about. Pick one group, a.b.hdtv for instance. It's probably got 2 billion posts in it. It's:
2 * 10^9 * 400 bytes = 800 Gigs of data. That's just one group. With Compression that's probably only an 80 gig download but, it's still 800 gigs of data that needs to processed.
Boneless has > 6 billion posts in it.
But for example if someone had a drive die and has to reinstall NB and re-download all the headers, the user is basically SOL
It's valuable data that represents 100's of gigs of downloads. You might want to back it up to external storage from time to time.
Read Main Topic