Setting Up Distributed Computing on BSD Systems in Order to Aid COVID-19 Research

Setting Up Distributed Computing on BSD Systems in Order to Aid COVID-19 Research

This will be a tutorial on setting up BOINC on FreeBSD (specifically, FreeNAS), in order to devote your extra computing power to aid The National Upcycled Computing Collective (NUCC, Inc.), a 501c non-profit organization, who is working on distributed computing projects with Rosetta@Home, which is focused solely on COVID-19 research at the time of this writing. Click here to read more about NUCC, and be sure to check out (and follow) @nucc_inc and @NUCC_STATS on Twitter. If you are fresh out of CPU cycles and would much rather prefer directly feeding cash to the cause, please feel free to do so here.

To get instructions for other operating systems that will have you processing workloads in less than 10 minutes, view the GitHub repo featuring quick super-quick setups scripts at https://github.com/phx/nucc.


Edit 3/28/2020: Many thanks to @iXsystems and @freenas for tweeting this out after it started gaining traction on /r/freenas and /r/freebsd!

That being said, I can barely take credit for any of this. My good buddy EvilMoFo is the one who helped me come up with the majority of these instructions and tweak the steps just right. Before he stepped into the picture, I had thrown up my hands and literally said, “I guess I can forget running this on Unix, because it just flat-out doesn’t work.” I then went on to set up a VM in FreeNAS to do the work, until EvilMoFo chimed in with his BSD genius and was like, “here, try this,” and bingo-bango I was up and running. You can visit his site at https://www.evilmofo.com/.

I have also removed a few steps from the documentation, which are not really necessary to get things up and running for this project in particular. I have gotten positive feedback on this article, that it has helped others get up and running successfully, but my goal is to make it as simple and straight-forward as possible. Please don’t hesitate to leave a comment if you have any issues or suggestions.


Edit 3/31/2020: Additional thanks to @MarcoFigueroa for featuring this project on The Hackers Mansion live podcast on March 28th. You can go here to see the full interview on Twitter.


Tuning the System

First we need to modify some configurations in /etc/rc.conf and /etc/sysctl.conf, which can easily be set as Tunables in FreeNAS by going to System > Tunables and setting the configuration values as seen below:

# linux_load="YES" (/etc/rc.conf):
# FreeNAS Tunable:
Variable: linux_load
Value: YES
Type: rc.conf
Description: Load Linux kernel modules
Enabled: (checked)

# security.bsd.unprivileged_idprio=1 (/etc/sysctl.conf)
# FreeNAS Tunable:
Variable: security.bsd.unprivileged_idprio
Value: 1
Type: sysctl
Enabled: (checked)

Now, open up a root shell on the host system, and run the following commands in order to go ahead and enable these options without needing a host reboot:

  • kldload linux
  • kldload linux64
  • sysctl security.bsd.unprivileged_idprio=1

Configuration on the Host

I’m currently running FreeNAS-11.3-U1, and it’s constantly giving me crap about setting up VNET jails via the web UI, so we’re going to do this the good ol’ fashioned way from the command line. Feel free to replace your release version with whatever you are running where I have 11.3-RELEASE, and obviously change your static IP and defaultrouter values as necessary:

iocage create -n "boinc" -r 11.3-RELEASE vnet="on" ip4_addr="vnet0|192.168.1.29/24" defaultrouter="192.168.1.1" boot="on"

Start the jail and exec into the console:

  • iocage start boinc
  • iocage console boinc

Configuration Inside the Jail

Now that you gotcha root shell inside ya chroot jail, let’s get choochin’.

# Enable the latest packages
echo 'FreeBSD: {url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest"}' > /usr/local/etc/pkg/repos/FreeBSD.conf
env ASSUME_ALWAYS_YES=YES pkg bootstrap
pkg update
pkg upgrade

# Enable ports:
portsnap fetch

# When running portsnap for the first time, extract the snapshot into /usr/ports:
portsnap extract

# Find our pot 'o gold:
cd /usr/ports/net/boinc-client

# Install the dependencies via pkg because ain't nobody got time to build all this mess from source:
pkg install curl dialog4ports gmake gettext pkgconf autoconf automake libtool freetype2 linux_base-c7

# Configure BOINC
make config

After running make config inside /usr/ports/net/boinc-client, you will be prompted with a dialog screen, and it’s very important to match the options below:

The following should be SELECTED/CHECKED:

  • CLIENT
  • LINUX
  • USER

The following should be UNSELECTED/UNCHECKED:

  • MANAGER
  • NLS
  • SKINS
  • X11
  • NO-GUI-RPC (very important that this is not selected)

Select OK to finish the configuration, and let’s get back to the commands.

Since this post is actually the BSD portion of the full documentation from https://github.com/phx/nucc, the NUCC United weak key is included in value of boinc_client_flags below. If you are already a member of another team, simply replace 2108683_fdd846588bee255b50901b8b678d52ec with your own weak key:

make
make install

# Enable BOINC on startup and attach to the NUCC Rosetta@home project
# These 2 commands just append lines to /etc/rc.conf:
sysrc boinc_client_enable=YES
sysrc boinc_client_flags="--allow_remote_gui_rpc --attach_project http://boinc.bakerlab.org/rosetta/ 2108683_fdd846588bee255b50901b8b678d52ec"

Please note: The weak key used with boinc_client_flags in /etc/rc.conf (2108683_fdd846588bee255b50901b8b678d52ec) is meant to get you up and running and processing workloads ASAP with the 501c non-profit National Upcycled Computing Collective (NUCC), as this post itself is actually the BSD portion of a larger set of documentation located at https://github.com/phx/nucc. The way this is set up and configured doesn’t require a BOINC account or a Rosetta@home account. It’s simply meant to get you immediately processing COVID-19 workloads as soon as possible. If you already have a user or team that you wish to work with, please feel free to substitute 2108683_fdd846588bee255b50901b8b678d52ec with your own weak key in /etc/rc.conf.

Set up the required authentication

You should modify /var/db/boinc/gui_rpc_auth.cfg to have a simple password on a single line. This will be used to authenticate with the boinc service in order to view and manage workloads.


Time To Put Those Work Boots On

Start the boinc-client service:

service boinc-client start

You can now install the boinc_curses package to view the workloads in a nice local TUI interface:

pkg install boinc_curses

To be able to execute boinccmd as users other than the boinc user, you will need to run chmod g+r /var/db/boinc/gui_rpc_auth.cfg.

If you are using a normal user instead of root and want to execute boinccmd commands, then run chgrp <your user group here> /var/db/boinc/gui_rpc_auth.cfg.

I run my jail as root, and the boinc-client process runs as the boinc user, so I would think that everything would work fine, but I still encountered some issues when running boinc_curses. That being said, I was able to get everything to work just fine by executing boinc_curses -p [password specified in /var/db/boinc/gui_rpc_auth.cfg].

I kind of got tired of specifying the password to boinc_curses on the command line every time, so I simply created myself a boincstats command by appending the following line to my ~/.profile:

alias boincstats="boinc_curses -p $(cat /var/db/boinc/gui_rpc_auth.cfg)"

Remotely View and Manage Workloads (show that work)

You can follow the normal BOINC installation instructions to download BOINC for another OS on your network, launching the BOINC Manager, and going to File > Select Computer. From there, you will get a pop-up requesting an IP/hostname and password.

Simply enter the IP address of your BSD jail and the password you specified in /var/db/boinc/gui_rpc_auth.cfg, and you will be able to remotely view and manage your tasks from elsewhere on the local network.

Alternatively, there is BOINCTASKS, which is only a task manager and doesn’t require you to download the client as well. The cool feature about BOINCTASKS is that if you have more than a single BOINC client on your local network, it will scan your network and add the clients it finds so that you can manage them all from a single interface without having to disconnect from one and reconnect to another like you would normally have to do with BOINC Manager. It’s a Windows program, but works flawlessly with Wine on Linux:

If you have any trouble or see any mistakes that I need to correct, please contact me on Discord (if you know me), or submit a comment below.


Edit 3/30/2020: I have heard from one person that they are having trouble with gui_rpc_auth. I followed these instructions directly, and have no trouble at all connecting to my client from another machine on the local network. That being said, if you have issues connecting remotely, please follow the additional instructions below.

When EvilMoFo and I set this up, his /var/db/boinc directory looked like this:

% ls -l /var/db/boinc
total 160
-rw-r--r--   1 boinc  boinc     1189 Mar 22 11:52 account_boinc.bakerlab.org_rosetta.xml
-rw-r--r--   1 boinc  boinc    54214 Mar 22 04:41 all_projects_list.xml
lrwxr-xr-x   1 boinc  boinc       38 Mar 22 04:58 ca-bundle.crt -> /usr/local/share/certs/ca-root-nss.crt
-rw-r--r--   1 boinc  boinc   104112 Mar 22 13:33 client_state.xml
-rw-r--r--   1 boinc  boinc   104017 Mar 22 13:33 client_state_prev.xml
-rw-r--r--   1 boinc  boinc      310 Mar 22 05:16 coproc_info.xml
-rw-r--r--   1 boinc  boinc      123 Mar 22 13:33 daily_xfer_history.xml
-rw-r--r--   1 boinc  boinc    11661 Mar 22 04:41 get_current_version.xml
-rw-r-----   1 boinc  <secret>    32 Mar 22 04:40 gui_rpc_auth.cfg
-rw-r--r--   1 boinc  boinc     2354 Mar 22 13:33 job_log_boinc.bakerlab.org_rosetta.txt
-rw-r--r--   1 boinc  boinc        0 Mar 22 04:40 lockfile
-rw-r--r--   1 boinc  boinc    23658 Mar 22 04:41 master_boinc.bakerlab.org_rosetta.xml
drwxrwx--x   2 boinc  boinc        6 Mar 22 05:16 notices
drwxrwx--x   3 boinc  boinc        3 Mar 22 04:40 projects
-rw-r--r--   1 boinc  boinc    28617 Mar 22 13:32 sched_reply_boinc.bakerlab.org_rosetta.xml
-rw-r--r--   1 boinc  boinc    38674 Mar 22 13:31 sched_request_boinc.bakerlab.org_rosetta.xml
drwxrwx--x  18 boinc  boinc       18 Mar 22 04:45 slots
-rw-r--r--   1 boinc  boinc      422 Mar 22 13:32 statistics_boinc.bakerlab.org_rosetta.xml
-rw-r--r--   1 boinc  boinc        0 Mar 22 05:16 stderrdae.txt
-rw-r--r--   1 boinc  boinc        0 Mar 22 05:16 stderrgpudetect.txt
-rw-r--r--   1 boinc  boinc    17679 Mar 22 13:33 stdoutdae.txt
-rw-r--r--   1 boinc  boinc       68 Mar 22 05:16 stdoutgpudetect.txt
-rw-r--r--   1 boinc  boinc      304 Mar 22 05:17 time_stats_log

Mine is the same, with the exception of the permissions not being as well-configured. He only manages his projects locally using boinc_curses. I, on the other hand, typically manage my projects remotely from the BOINC Manager on a Linux box and have no issues connecting to the FreeNAS jail. I can confirm that I do not currently have cc_config.xml or remote_hosts.cfg in /var/db/boinc, but if you are running into issues connecting remotely, try this:

Create /var/db/boinc/cc_config.xml and make the contents look like this:

<cc_config>
   <options>
       <allow_remote_gui_rpc>1</allow_remote_gui_rpc>
   </options>
</cc_config>

Create /var/db/remote_hosts.cfg, and make it look like this:

127.0.0.1
<INSERT IP ADDRESS OF REMOTE MONITORING MACHINE>

chown and chmod as necessary to make sure the files have the correct permissions, and then restart the service:

service boinc-client restart

Please let me know if you have any further issues.

Comments ( 7 )

  1. ReplyKenwoodfox
    Update `service boinc_client start` to `service boinc-client start` ?
    • ReplyJamey
      Thanks! Good catch. It has been updated.
  2. Valuable News – 2020/03/30 | 𝚟𝚎𝚛𝚖𝚊𝚍𝚎𝚗
    […] Setup Distributed Computing on BSD Systems in Order to Aid COVID-19 Research. https://bellebookandcode.com/tech/setting-up-distributed-computing-on-bsd-systems-in-order-to-aid-co… […]
  3. Replyjgrafton
    Great post, this worked for me in a FreeBSD 12.1 jail. I found that `curl` was also a required dependency for the boinc-client build. Also, the end of the `pkg install` line contains a period. `pkg install dialog4ports gmake gettext pkgconf autoconf automake libtool freetype2 linux_base-c7 curl`
    • ReplyJamey
      Thanks! Post has been updated.
  4. This Vicious Cure by Emily Suvada | Belle, Book & Code
    […] after seeing it gain traction over on Reddit at /r/freenas and /r/freebsd, and because of the BSD documentation that is hosted on this very blog. Additionally, I did a live interview with Marco Figueroa on his podcast, The Hackers Mansion, […]
  5. Replybrian
    Thanks for a great guide! On my system (FreeNAS 11.3) I had to use `linux_enable` instead of `linux_load` in the system tunables.

Leave a Reply