The New Environment That Performs Like "Wow!"

We are again stuck with hellish build time problems. Our ccache solution is suffering and I think it is because of the underlying infrastructure limitations. I hate to say "I think" because it kills me not to know what exactly is wrong with the infrastructure. I'm not an expert in network, storage, linux administration, and it made me handicapped. We have to resort the IT support for help all the time, and they are at my organization not up to our expectations.

We thought we are in trouble. Luckily, and strangely, the new cloud environment that we recently got, where we are run our build farm and evaluating our builds on RedHat 6.5 is performing very well. The first impression is that this new environment is 6 times faster! Very surprising results.

I verified a few basic differences between the old and new build farm, and the differences/improvements are:

1. RedHat 5.5 vs RedHat6.5


2. 16 cores hyperthreaded vs 16 cores hyperthreaded
   
   
   
   • I studied a few articles on if hyperthreading helps speed up compilation, and from what I understand, it does NOT help compilation. As far as my experience goes, hyperthreading will be a performance "loss" for compilation because compilers are not multi-threaded.
   
   
   • http://stackoverflow.com/questions/4430001/which-x86-c-compilers-are-multithreaded-by-itself
   
   
   • http://stackoverflow.com/questions/20640193/why-is-parallel-compilation-performance-with-ht-worse-than-without
   
   
   • That said, although the machines in our build farm are 8core machines(with hyperthreading, 16cores), and one may run make -j16, it slows down the compilation because of multi-threading. So, we default to running "make -j8" on a 16core.
   
   
   
   


3. 16GB RAM vs 80GB RAM
   
   
   
   • This seem to have improved the compile-times. I'm not sure to what extent.
   
   
   
   


4. Disk storage type:
   
   
   
   • The storage used by the build comes from epeheral. I don't know what benefits this gave.
   
   
   
   


5. Network Connectivity between machines in the build farm.
   
   
   
   • Our builds run using a comerical make flavor called "ElectricAccelerator" Builds are distributed, and they share the intermediate build results across the build farm.
   
   
   • That obviously needs connectivity across the build farm to be fast.
   
   
   • My observation is, the old and new infrastructure differ a lot with the way they are connected to each other.
   
   
   • ping showed some visible difference.
   
   
   
   

I'm happy with the preliminary results with the build times, and hopefully when we are in production with this setup, our developers will hopefully spend less time waiting, and we can spend more time on some quality topics than monitoring the environment all the time. Comments are welcome.