by Quade (Posted Wed, 24 Sep 2014 16:38:38 GMT)
What kind of machine are you doing this on? Your results seem to be far worse than mine.
What kind of hard disk is the data folder on?
As a data point, my machine processed 800 megs of header in about 7 minutes. This is a different version than you're using with I think a faster version of the database.
More threads aren't going to help unless the bottleneck is CPU. If the bottleneck is the DB or disk IO in general then more threads might make it worse. If you have a machine with poor single core integer performance like an AMD, it could probably help. Might make sense to change the GZ format into files instead of headers too. In that way the downloader CPU's do more of the work. Have to think about that one.
Read Main Topic
What kind of machine are you doing this on? Your results seem to be far worse than mine.
What kind of hard disk is the data folder on?
As a data point, my machine processed 800 megs of header in about 7 minutes. This is a different version than you're using with I think a faster version of the database.
More threads aren't going to help unless the bottleneck is CPU. If the bottleneck is the DB or disk IO in general then more threads might make it worse. If you have a machine with poor single core integer performance like an AMD, it could probably help. Might make sense to change the GZ format into files instead of headers too. In that way the downloader CPU's do more of the work. Have to think about that one.
Read Main Topic
