time to process 100026 headers
   batch size | threads | time
   -----------+---------+-------
      1       |   1     | 3m 44s 
      5       |   2     | 4m 51s 
     25       |   2     | 3m 25s 
     50       |   2     | 3m  4s  
   1000       |   2     | 2m 29s 
   5000       |   2     | 2m 26s 
  25006       |   4     | 2m  3s (batch size = #headers/#threads)
   3125       |   8     | 1m 19s 
   6251       |  16     | 1m 18s (batch size = #headers/#threads)
   3125       |  16     | 1m  9s (batch size = #headers/#threads/2)
   1562       |  16     | 1m  8s (batch size = #headers/#threads/4)
    781       |  16     | 1m  6s (batch size = #headers/#threads/8)
    500       |  16     | 1m  5s
    390       |  16     | 1m  9s (batch size = #headers/#threads/16)
    125       |  32     | 1m  7s
    250       |  32     | 1m  3s
    500       |  32     | 1m  5s
    781       |  32     | 1m  5s
    781       |  50     | 1m  3s
   1000       | 100     | 1m 21s 
    781       | 200     | 1m 44s
 (this is on a 16 core machine)
 in any event - now that i've got that working - i should probably go back to either optimizing the database i have or just switch to pgsql or some shit
 or both