This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Server Configuration and Notes

Server Configuration and Notes.

1: VPS Migration

1.1: Going to jail
1.2: Creating Vultr Instance

2: Ghost Blog
3: Setup GitLab Repo
4: Build RKey Tech

This section is for Server configuration and deployment notes.

1 - VPS Migration

VPS migration from OnPrem and Digital Ocean Cloud to Vultr Cloud.

1.1 - Going to jail

Matching Blog Post

Getting thrown in jail

Install BastilleBSD:

doas pkg update
doas pkg upgrade
doas pkg install bastille

Setup Bastille to start at boot:

doas sysrc bastille_enable=YES

I only have one IP, so I use local loop back:

doas sysrc cloned_interfaces+=lo1
doas sysrc ifconfig_lo1_name="bastille0"
doas service netif cloneup

I use Packet Filter firewall so here is my initial /etc/pf.conf: Before moving into the jails.

ext_if="vtnet0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"

block in all
pass out quick keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA keep state
pass in inet proto tcp from any to any port 80 flags S/SA keep state
pass in inet proto tcp from any to any port 443 flags S/SA keep state
pass in inet proto udp from any to any port 51820

I’m still using the host at this point so I have to allow ports: 80, 443, 22 and port 51820 is my WireGuard VPN to proxy to my NextCloud instance in a jail on a server running in my apartment.

With pf configured set it to auto start on boot and fire it up:

doas sysrc pf_enable=YES
doas service pf start

At this point we get a TCP socket error and our connection drops, so we have to ssh back in.

Now it’s time to bootstrap Bastille:

doas bastille bootstrap 13.0-RELEASE update

I will be moving Caddy into a jail and also running NextCloud in a jail on my VPS instead of at home and also I will have a jail for my database server that NextCloud uses.

doas bastille create database 13.0-RELEASE 10.10.10.2
doas bastille create nextcloud 13.0-RELEASE 10.10.10.3
doas bastille create caddy 13.0-RELEASE 10.10.10.4

Here is what I have so far:

bsh ➜  doas bastille list
 JID             IP Address      Hostname                      Path
 caddy           10.10.10.4      caddy                         /usr/local/bastille/jails/caddy/root
 database        10.10.10.2      database                      /usr/local/bastille/jails/database/root
 nextcloud       10.10.10.3      nextcloud                     /usr/local/bastille/jails/nextcloud/root

I’m going to do some cheating here just to show how awesome jails are. Instead of building from scratch I will simply tar my database and nextcloud from my home server to the cloud and then I will simply tar all the websites and the Caddyfile from the host to the jail. Then modify pf and ssh reboot and everything will come up in jails on the VPS. This method which I did not use the first time worked so well I did not even have to turn off my up time monitor and there was no downtime beyond a second for the reboot. This is so freaking awesome so here we go:

First I will tar up my database and nextcloud jails on my home server

doas su
cd /usr/local/bastille/jails/
tar cvf ~/database.tar database
tar cvf ~/nextcloud.tar nextcloud

You should shut down each jail before taring up the jails. I just turned off syncing on my iPhone and paused DAVX on my android to limit any activity. I also logged out of all web instances I had on NextCloud. (I wanted to see if I could do this with no down time, obviously on a business system I would have maintenance window and shut things down). I moved these files to regular user and change permissions. I then just scp the files to the VPS instance.

cd
scp database.tar titania:~/
scp nextcloud.tar titania:~/

ssh into cloud instance and reverse the process.

ssh titania
doas su
bastille stop database
bastille stop nextcloud
cd /usr/local/bastille/jails/
tar xvf /home/user/database.tar
tar xvf /home/user/nextcloud.tar

Create regular user in caddy jail and setup SSH for deploying websites from workstation.

doas bastille console caddy
pkg update
pkg upgrade
pkg install caddy
sysrc caddy_enable="YES"
adduser
exit

Enable ssh for caddy jail

doas bastille sysrc caddy sshd_enable="YES"

It will start when we reboot we do not need to start it yet. We want to copy the web sites and the Caddyfile into the jail also we need our SSH key.

doas rsync /home/user/ /usr/local/bastille/jails/caddy/root/home/user/
doas rsync /usr/local/etc/caddy/ /usr/local/bastille/jails/caddy/root/usr/local/etc/caddy/

The file permissions for the regular user in caddy will need to be fixed

doas bastille console caddy
chown -R user:group /home/user

Also since I am no longer using the WireGuard VPN to hit the Nextcloud instance at home I will need to change the Caddy File in the caddy jail to reflect this.

vi /usr/local/etc/caddy/Caddyfile

Change proxy for nc.rkey.tech From:

nc.rkey.tech {
        reverse_proxy 10.20.10.2:80
        header {
        Strict-Transport-Security "max-age=15768000;includeSubDomains;"
        }

        redir /.well-known/carddav /remote.php/dav 301
        redir /.well-known/caldav /remote.php/dav 301

        log {
                output file /var/log/caddy/nc.rkey.tech.access.log
                format json
        }
}

To:

nc.rkey.tech {
        reverse_proxy 10.10.10.3:80
        header {
        Strict-Transport-Security "max-age=15768000;includeSubDomains;"
        }

        redir /.well-known/carddav /remote.php/dav 301
        redir /.well-known/caldav /remote.php/dav 301

        log {
                output file /var/log/caddy/nc.rkey.tech.access.log
                format json
        }
}

10.20.10.2:80 was the WireGuard to home and 10.10.10.3:80 is now the NextCloud jail locally. For ease of use and habit I’m keeping the jail at the standard SSH port for deployments and so I will need to change the host port. So back on the host I make the following changes. I change the /etc/ssh/sshd_config file. From:

#Port 22

To:

Port 5000

On the host I also need to keep Caddy from stating at boot.

doas sysrc caddy_enable="NO"

Then in the /etc/pf.conf file I need to remove the following by commenting out or deleting:

#pass in inet proto tcp from any to any port ssh flags S/SA keep state
#pass in inet proto tcp from any to any port 80 flags S/SA keep state
#pass in inet proto tcp from any to any port 443 flags S/SA keep state
#pass in inet proto udp from any to any port 51820

Because the Caddy server is now in a jail and because my host SSH port has changed I need to add the following:

nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr pass inet proto tcp from any to any port {80, 443, 22} -> 10.10.10.4
rdr-anchor "rdr/*"

pass in inet proto tcp from any to any port 5000 flags S/SA keep state

At this point I say “Hold onto your butts” and reboot the VPS instance.

Within about a second I browse to rkey.tech, rkey.online and nc.rkey.tech and all 3 sites were up and operational. I then ssh into the host ssh -p 5000 titania and then ssh into the jail ssh caddy and this all worked as well.

1.2 - Creating Vultr Instance

Creating Vultr Instance

Matching Blog Post

Creating instance at Vultr & setting up DNS

Setting up my CAA: I use a Caddy server that auto-magically takes care of my SSL certs for me. I like lazy solutions that work.

0 issue "letsencrypt.org"
0 iodef "mailto:rkey@rkey.tech"

Next is all the mail stuff to work with ProtonMail: In case anyone is wondering those are not the real keys :)

CNAME protonmail._domainkey protonmail.domainkey.dn39a43a188d9439487409be25ea.domains.proton.ch
CNAME protonmail2._domainkey protonmail2.domainkey.dn39a43a188d9439487409bewida.domains.proton.ch
CNAME protonmail3._domainkey protonmail3.domainkey.dn39a43a188d9439487409bewida.domains.proton.ch
MX mail.protonmail.ch 300 10
MX mailsec.protonmail.ch 300 20
TXT "protonmail-verification=ca39a43a188d9439487409be"
TXT "v=spf1 include:_spf.protonmail.ch mx ~all"
TXT _dmarc "v=DMARC1; p=quarantine; rua=mailto:rkey@rkey.tech; ruf=mailto:rkey@rkey.tech; sp=quarantine; aspf=s; fo=1;"

Before changing the DNS pointers I need to migrate my Caddy server at Digital Ocean (DO) over to Vultr.

I do that by replicating my deployment script to mirror sites.

bsh ➜  cat deploy.sh
rm -rf public/
rm public.tar
HUGO_ENV="production" hugo --gc || exit 1
echo OK, now that stuff is built
rsync -azP --delete public/ titania:~/rkey.tech/ # Caddy server at Vultr
rsync -azP --delete public/ spirit:~/rkey.tech/ # Caddy server at DO
echo OK, now that stuff is uploaded
echo ======================================
echo Done
echo ======================================

create user and sync ssh keys.

rsync -a --chown user:group ~/.ssh /home/user/

Make sure the new user can su to root. I use doas.

🕙[ 13:47:15 ] bsh ➜  doas cat /usr/local/etc/doas.conf
permit nopass keepenv :wheel

Turn off password authentication and root user login by adding to the end of /etc/ssh/sshd_config

PermitRootLogin no
PasswordAuthentication no

Configure Caddy

titania.rkey.tech {
        root * /home/user/wtf/
        file_server
        log {
                output file /var/log/caddy/titania.rkey.tech.access.log
                format json
        }
}

2 - Ghost Blog

Setup Ghost Blog In FreeBSD Jail

Matching Blog Post

Create new jail to install Ghost Blog into:

$ doas bastille create ghost 13.0-RELEASE 10.10.10.5

Create database user for the Ghost Blog

bsh ➜  doas bastille console database
[database]:

Log into the database server

mysql -u root -p
Enter password: **********

Create database user

[(none)]> create user 'ghost'@'10.17.63.5' identified by '**********************';

Create the database

[(none)]> create database ghostdb;

Grant all privileges on the database to the user

[(none)]> grant all on `ghostdb`.* to 'ghost'@'10.17.63.5';

Make it so the user can log in over jails network

[(none)]> alter user 'ghost'@'10.17.63.5' identified with mysql_native_password;

After the above step I always have to enter the password for the user again. It’s been decades since I worked as DBA, so maybe I’m just doing things out of step here.

[(none)]> alter user 'ghost'@'10.17.63.5' identified by '**********************';

Flush privileges and exit

[(none)]> flush privileges;
[(none)]> quit
Bye

Log into Ghost jail and install and configure Ghost. Note initially I tried latest npm-node package, but kept backing down the version until I found a version that worked which is npm-node14. Doas is just for my convenience and not required if you prefer to su - instead. And of course I need to the MySQL client to connect to the database in the database jail.

doas bastille console ghost
# pkg update
# pkg upgrade
# pkg install doas npm npm-node14 openssl curl mysql80-client

Create ghost user

# adduser

Create startup script so the Ghost Blog will start on reboot.

# vi /usr/local/etc/rc.d/ghost

Example of script below

#!/bin/sh

# PROVIDE: ghost
# REQUIRE: mysql
# KEYWORD: shutdown

. /etc/rc.subr

name="ghost"
rcvar="ghost_enable"
extra_commands="status"

load_rc_config ghost

start_cmd="ghost_start"
stop_cmd="ghost_stop"
restart_cmd="ghost_restart"
status_cmd="ghost_status"

PATH=/bin:/usr/bin:/usr/local/bin:/home/ghost/.bin

ghost_start()
{
    su ghost -c "/usr/local/bin/ghost start -d /usr/local/www/msrobota.online"
}
ghost_stop()
{
    su ghost -c "/usr/local/bin/ghost stop -d /usr/local/www/msrobota.online"
}

ghost_restart()
{
    ghost_stop;
    ghost_start;
}

ghost_status()
{
    su ghost -c "/usr/local/bin/ghost status -d /usr/local/www/msrobota.online"
}

run_rc_command "$1"

Make the script executable.

# chmod +x /usr/local/etc/rc.d/ghost

Create directory for the blog to be installed into and run from.

# mkdir -p /usr/local/www/msrobota.online

Make user ghost own the directory

# chown -R ghost:ghost /usr/local/www/msrobota.online

Install ghost command line tools

# npm install ghost-cli@latest -g

change to ghost user and install ghost

# su - ghost
$ cd /usr/local/www/msrobota.online
$ ghost install

During the install you will get errors about unsupported system, since everyone who writes software nowadays assumes we have all been assimilated into the Linux collective.

The configuration file built from the answers provided by the install will look like the file below.

$ cat config.production.json
{
  "url": "https://msrobota.online",
  "server": {
    "port": 2368,
    "host": "127.0.0.1"
  },
  "database": {
    "client": "mysql",
    "connection": {
      "host": "10.17.63.2",
      "user": "ghost",
      "password": "***************************",
      "database": "ghostdb"
    }
  },
  "mail": {
    "transport": "Direct"
  },
  "logging": {
    "transports": [
      "file",
      "stdout"
    ]
  },
  "process": "local",
  "paths": {
    "contentPath": "/usr/local/www/msrobota.online/content"
  }
}

You can start ghost from the install directory with $ ghost start or by using the rc script created earlier with $ doas service ghost start I had already setup a DNS zone file as shown earlier for Creating Vultr Instance for the Ghost Blog’s domain of https://msrobota.online So we just need to add that domain to the caddy server jail configuration

bsh ➜  doas bastille console caddy
# vi /usr/local/etc/caddy/Caddyfile

And add

msrobota.online {
        reverse_proxy 10.17.63.5:2368
        log {
                output file /var/log/caddy/msrobota.online.access.log
                format json
        }
}

Then restart caddy with # service caddy restart You will not want to wast time and create your user and secure that account because until you do anyone can do so. I used to create firewall rules and allow only access from my workstation or rather ISP account. But that triples the amount of time. So I just make sure as soon as I restart caddy I’m setting up the owner account at https://msrobota.online/ghost/ I used to then go in and configure the server to host dark only themes, open external links in new windows/tabs and remove the Ghost branding by modifying the /usr/local/www/msrobota.online/current/content/themes/casper/default.hbs file, but with version 3 you can configure the server to default to dark mode at Settings --> design --> site wide --> Color scheme and for simplicity I now just use code injection. To remove the Ghost Branding I inject the following code in the footer

<style>
  /* The footer links to ghost keep coming back */
  .site-footer a[href^="https://ghost.org"] { display: none; }
</style>

And to ensure external links open in new tabs/windows I inject the following code also in the footer

<script>
  const anchors = document.querySelectorAll('a');

  for (x = 0, l = anchors.length; x < l; x++) {
    const regex = new RegExp('/' + window.location.host + '/');
    
    if (!regex.test(anchors[x].href)) {
      anchors[x].setAttribute('target', '_blank');
      anchors[x].setAttribute('rel', 'noopener');
    }
  }
</script>

That’s it now all that remains is to begin blogging, which is the most difficult part me.

3 - Setup GitLab Repo

Setup GitLab Repo

Matching Blog Post

Set global variable first.

git config --global user.name "Robyn Key"
git config --global user.email "rkey@rkey.tech"

Then I create a blank project on GitLab. I do not initialize the repo, as that would cause issue when uploading this project for the first time. With the empty repo created I then initialize and upload my project.

git init --initial-branch=main
git remote add origin git@gitlab.com:rkey/rkey.tech.git
git add .
git commit -m "Initial commit"
git push -u origin main

I have an install script I do not want in the repo and I also do not want the public directory in the repo either. So my .gitignore file looks like this:

bsh ➜  cat .gitignore
public
.hugo_build.lock
deploy.sh
public.tar

4 - Build RKey Tech

Build rkey.tech project

Matching Blog Post

This document is old and probably not accurate anyway as I initially built the site using the Congo theme and the changed to the GeekDocs theme but just modified from memory on what was different. This will be completely redone in the near future to reflect the Docsy build and deployment. So stay tuned for that.

Update 19-July-2022: When I first built my site I built it using the Congo theme with Hugo. I did that because I could not figure out how to build my preferred theme GeekDocs correctly. Well is was a simple Markdown error on my part so now RKey.Tech will be a GeekDocs theme and my personal only site RKey.Online will use ~~Congo~~ GeekBlog theme.

Create new Hugo project

hugo new site rkey.tech

cd into new project and initialize

cd rkey.tech
git init

Verify and initialize ‘go’

go version
  go version go1.17.7 linux/amd64
hugo mod init rkey.tech

I initially tried using webpack, but my luck putsing with npm on Linux is no better than the luck I have on FreeBSD. I have come to realize Java freaking hates me!! If you want to use the theme from a cloned branch instead of a release tarball you’ll need to install webpack locally and run the build script once to create all required assets. (This did not work for me)

# install required packages from package.json
npm install

# run the build script to build required assets
npm run build

So I just I just downloaded the pre-release bundle

mkdir -p themes/hugo-geekdoc/
curl -L https://github.com/thegeeklab/hugo-geekdoc/releases/latest/download/hugo-geekdoc.tar.gz | tar -xz -C themes/hugo-geekdoc/ --strip-components=1

From here I just copied the files and directories from the theme directry to the site directory.
NOTE: You should only need to do this for files you are modifying from the downloaded theme. I guess the saying do as I/they say not as I do is appropiate here.

cd themes/hugo-geekdoc/
cp -a archetypes assets data i18n images layouts static ../../

From there it is just a matter of configuring toml or yaml files.

The first one to tackle is config.toml in the site root. I left this mostly as is.

cat config.toml
baseURL = 'https://rkey.tech/'
languageCode = 'en-us'
title = 'RKey Tech'
#theme = "hugo-geekdoc"

pluralizeListTitles = false

# Geekdoc required configuration
pygmentsUseClasses = true
pygmentsCodeFences = true
disablePathToLower = true
geekdocFileTreeSortBy = "date"
geekdocSearchShowParent = true

# Required if you want to render robots.txt template
enableRobotsTXT = true

# Needed for mermaid shortcodes
[markup]
  [markup.goldmark.renderer]
    # Needed for mermaid shortcode
    unsafe = true
  [markup.tableOfContents]
    startLevel = 1
    endLevel = 9

[taxonomies]
   tag = "tags"

Then I added my deployment script.

cat deploy.sh
rm -rf public/
rm public.tar
HUGO_ENV="production" hugo --gc || exit 1
echo OK, now that stuff is built
rsync -azP --delete public/ caddy:~/rkey.tech/
echo OK, now that stuff is uploaded
echo ======================================
echo Done
echo ======================================

I then made the following modification to i18n/en.yaml for footer brandinig.

footer_build_with: >
  Copyright 2022 - Robyn Key
  for <img alt="RKey.tech" src="/images/rkeytechlogostny.png.webp" width="50" height="25"<a> and <a href="https://rkey.online" target="_blank"><img alt="RKey.Online" src="/images/rkeyonline.png.webp" width="60" height="60"></a>  
footer_legal_notice: Legal Notice
footer_privacy_policy: Privacy Policy
footer_content_license_prefix: >
  Content licensed under

For my links in the header I modified layouts/partials/head/custom.html and added the following to this blank file.

<!-- You can add custom elements to the page header here. -->
<a href="https://rkey.online" target="_blank"><img alt="RKey.Online" src="/images/rkeyonline.png.webp" width="70" height="70"></a>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://gitlab.com/RWK3y" target="_blank"><img alt="GitLab" src="/images/icons8-gitlab-70.png.webp" width="30" height="30"></a>
<a href="https://github.com/RWK3y" target="_blank"><img alt="GitHub" src="/images/icons8-github.svg" width="30" height="30"></a>
<a href="https://bsd.network/@R0bWK3y" target="_blank"><img alt="BSDNetwork@Mastodon" src="/images/mastodon.png.webp" width="30" height="30"></a>
<a rel="me" href="https://twitter.com/RWK3y" target="_blank"><img alt="Twitter" src="/images/icons8-twitter-circled.svg" width="30" height="30"></a>
<a rel="me" href="https://facebook.com/FB.RWK3y" target="_blank"><img alt="Facebook" src="/images/icons8-facebook.svg" width="30" height="30"></a>
<a rel="me" href="https://www.instagram.com/fb.rwk3y/" target="_blank"><img alt="Instagram" src="/images/icons8-instagram.svg" width="30" height="30"></a>
<a rel="me" href="https://www.reddit.com/user/RWK3y" target="_blank"><img alt="Reddit" src="/images/icons8-reddit.svg" width="30" height="30"></a>
<a href="https://infosec.exchange/@rkey" target="_blank"><img alt="InfoSecExchange@Mastodon" src="/images/mastodon.png.webp" width="30" height="30"></a>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<a href="https://www.standwithukraine.how/" target="_blank">StandWithUkraine<img src="/images/icons8-ukraine-70.png.webp" width="30" height="30"></a>
<a href="https://protectdemocracy.org/" target="_blank">ProtectDemocracy<img src="/images/icons8-usa-70.png.webp" width="30" height="30"></a>

From there I just extract my favicons I created at icongen

cd static/favicons
unzip icongen.zip

All my images for the site are kept in static/images for brand images I usually go to Icons8. I generally strive to download .svg files, barring that I will download .png files and then use either convert on the command line from the ImageMagick program or more recently I discovered libwebp and now use cwebp to convert images from the command line.

The Geedocs theme turned out to be the perfect solution for my online documentation. My origional theme used Congo, which is a beautiful theme, but not suited quite as well as the Geekdocs theme for documentation purposes. I really like the layout better for even using as a blog.