Gitlab Runners: Keeping MacOS Awake

Recently, I’ve been working to create a reliable set of MacBook Gitlab Runners for running Xcode builds. Confusingly, the jobs running on these machines would freeze or not get picked up at all; that is until I would SSH into them.

There is frustratingly little information on running Gitlab Runners on Macs and it seemed that no one else was encountering (or at least posting) about this particular issue.

I hope this helps someone else in need of a path. I will start with the solution first but if you’re interested I’ll also dive into a brief overview of my debugging process farther down.

Solution

Configure Power Settings to never sleep (some of this may be overkill and I’ve read that some of these settings are machine-specific):

# Disable Sleep
sudo pmset sleep 0
# Disable Standby
sudo pmset standby 0
# Disable Display Sleep
sudo pmset displaysleep 0
# Disable Auto Power Off
sudo pmset autopoweroff 0
# Disable Disk Sleep
sudo pmset disksleep 0
# Disable Powernap
sudo pmset powernap 0

If any of these error, run pmset -g to view your available settings.

Then, we’ll need to background a caffeinate command to prevent powerd from putting everything to sleep:

# caffeinate
#  -i  Prevent the system from idle sleeping.
#  -m  Prevent the disk from idle sleeping.
#  -s  Prevent the system from sleeping.
nohup caffeinate -ims &

You should find that the machine no longer sleeps when you aren’t logged in.

If you need to stop caffeinate run:

ps aux | grep caffeinate | grep -v grep | awk '{print $2}' | xargs kill

cron it (automate)

In order to ensure this process continues to restart on failure or after reboot, create a script /usr/local/bin/caffeinate_kick.sh (or wherever makes sense for you):

#!/bin/bash
pgrep caffeinate
if [ $? -ne 0 ]
then 
    nohup caffeinate -ims & 
fi

Run chmod +x /usr/local/bin/caffeinate_kick.sh

And add this entry to crontab (run crontab -e as the appropriate user):

* * * * * /usr/local/bin/caffeinate_kick.sh

This will ensure the caffeinate process is always running and will restart it after a system reboot.

Finding the Solution

The symptoms of my runners going to sleep were jobs becoming “frozen” or never being picked up at all. I found that SSH-ing in to the boxes would “unstuck” the job or get them to be picked up by a runner.

I didn’t know for sure initially that they were going to sleep, but I suspected it was a state issue based on the above.

I had configured the machines to never sleep as far as I could tell:

After some online digging, I was able to find the pmset command.

Running pmset -g everything looked good:

gitlabrunner-docker2-mac:~ gitlab-runner$ pmset -g
System-wide power settings:
Currently in use:
 lidwake              0
 autopoweroff         0
 standbydelayhigh     86400
 autopoweroffdelay    28800
 proximitywake        1
 standby              0
 standbydelaylow      10800
 ttyskeepawake        1
 hibernatemode        3
 powernap             0
 gpuswitch            2
 hibernatefile        /var/vm/sleepimage
 highstandbythreshold 50
 womp                 1
 displaysleep         10
 networkoversleep     0
 sleep                0
 tcpkeepalive         1
 halfdim              1
 acwake               0
 disksleep            0

But, running pmset -g log I could still see “Sleep” events:

2020-07-07 11:20:27 -0600 Sleep                 Entering Sleep state due to 'Maintenance Sleep':TCPKeepAlive=inactive Using AC (Charge:100%) 16 secs

The times it would go to sleep lined up with the freezes in my running jobs.

After much Googling, I finally discovered caffeinate. The easiest option of usage was to background this process indefinitely. After running the nohup command above, I was able to verify that my machines were no longer dropping out and all my jobs were completing reliably! 🎉