I run Linux for most of my development work (As would most college-level students for college work). I specifically prefer to use Manjaro (or any Arch-based distribution, to be honest) because of how well it has worked for me.
However, if you head over to the HPCC Systems download page, you may notice a stark absence of Manjaro and/or Arch Linux. The supported operating systems (as of writing this), is CentOS 7/8 and various flavours of Ubuntu. Thankfully, all is not lost, as HPCC Systems is open source, and one of the best outcomes of that is that it can be built from source.
Scouring the manual and github repository brought me to this build instructions page where instructions were provided for building from source for the client tools. This, gave me a great idea for how to set up the environment.
A development environment is simple enough where you have a client computer making requests to a single-node cluster (which can be your computer again).
This reminded me of how useful Docker can be here. I’ve been using Docker a lot as of recent, and I tend to use it a lot in cases where the resource overhead of VMs do not feel good to me, but it still makes sense to keep the processes of a server process contained.
So this was going to be my two part installation:
- The HPCC Systems Client tools – compiled from source
- The HPCC Systems Cluster – on Docker as a separate container
Client Tools
Getting client tools to work was simple. I installed all the dependencies, plus sccache (Really helpful as it caches your compilation results and helps when you recompile). I combined the instructions from sccache’s page and the build instructions, to build only the client tools, and to use sccache as a cache tool. I typed in make -j6 and then took a small break.
Once that was all done and dusted, it seemed weird that make package was trying to install the package to my system, skipping the package making that usually happens in other distros. I skipped straight ahead to the sudo make install and now I had a working Client Tools install.
HPCC Systems
Although native installs are the best, I have found that VMs are a good way to manage it in case something goes wrong (Botched configuration, resources overuse) and are a breeze to clean up. As of recent I have started to use Docker, and its a nice replacement for VMs where the extra level of virtualization is not required. So, I also found some resources on building HPCC Systems images for Docker, but it was for v7, although I was using v8. Using this, I ended up making some changes and setting up a nice Dockerfile and docker-compose.yml to create and use a single-node cluster. Adding a restart: always to the Compose configuration, allows the image to restart on machine restart (which is a really nice feature I’ve needed).
For me, Docker already had forwarded its ports to localhost(:8010,…). If not, there are environment variables ECL_WATCH_IP and ECL_WATCH_PORT that can be set for the client tools to connect to that specific cluster.
And all done!
With a development environment ready to go, I can start testing ECL and how translations between HSQL and ECL can work!