Compilation of large C++ projects takes patience so for a while now we have been provided automated builds to the community using a dedicated physical system we called IfcOpenBot. In reality this poor thing was a Mac Mini from 2014 in the least powerful configuration possible, sitting next to the meter box at the AECgeeks office. The reason for this setup is that compiling for macOS requires a macOS operating system and a running a macOS virtual machine on non-apple hardware is a violation of their terms of service. We filled the Mac Mini with VirtualBox virtual machines (including a cheap Win 8 Pro license that got a free upgrade to Win 10) and tied things together with a bunch of shell scripts (VBoxManage to start and stop the VMs) and SSH to connect into the hosts. We installed Cygwin into the windows box to allow SSH access. Then it was simply a matter of running the build scripts we have in the repository and collecting the output.
This approach worked well for several years, but began to face issues:
- The physical setup and MacOS host didn't really allow for full remote access, this meant we had to carry monitors around occasionally to connect and debug.
- The setup of VirtualBox incremental snapshots introduced a massive storage overhead. So more and more portable USB SSD drives were added.
- Build times were really slow, even with caching the dependencies and using ccache it took several days.
- In the end it wasn't really possible anymore to run the latest Win 10 guest anymore along the macOS host as they were fighting for the little 4GB of RAM.
New approach
The christmas holidays gave us some time to rework this solution.
We rent a hosted Mac Mini using one of the commercially available solutions. Monthly cost $60.
We have a digital ocean droplet doing the orchestration. Monthly cost $5.
Windows and Linux machines are spinned up using the AWS EC2 command line tools. Build on linux takes about 1 hour on t2.2xlarge so that's 30 dollar cents per build according to the current prices.
For windows we have configured the user-data scripts to install chocolatey and WinRM on boot and on the host we use ansible to control the build. For the other systems we just use plain SSH.
The new builds from this system have just arrived: please help test for issues. In addition to the changes discussed above, the linux distro is changed from Debian to Ubuntu 16.04 (so this changes glibc versions and all). Since EC2 does not host 32bit operating systems (anymore, it seems), we use cross-compiling now on 64bit Ubuntu. The visual studio version on Windows is now 2017 instead of 2015 (using the build tools package from chocolatey).
Support
Among others, this shows the complexity for open source projects to build for macOS. Initially it drove us to expensive hardware that we couldn't upgrade. In the current approach it takes up over 85% of the cost (assuming a weekly build on EC2).
If the IfcOpenBot builds are useful to you, please consider a small donation at https://opencollective.com/opensourcebim to cover these costs. OpenSourceBIM is a new foundation started by myself and Dion Moult that tries to organize and support the development of open source software for OpenBIM. We will provide more details later.
No comments:
Post a Comment