Welcome to CodeForLife Docs and TILs

Objectives: Docs with tips or tricks within Python, Ruby, Linux, Data Ops, Data Engineering, some Javascript and Frontends.

This is only scripts with minimal explanation. The full explanation is going on my blog.

TILs: "Today I learned" are simple tricks that you annotated to remind yourself on a later date, or share with fellow devs.

ArchLinux | Manjaro Based Setup

Tips and tricks for a ideal initial setup

Ranking mirrors for faster downloads

Manjaro

 sudo pacman-mirrors -c Brazil,United_States

ArchLinux

curl -s "https://www.archlinux.org/mirrorlist/?country=BR&use_mirror_status=on" | sed -e 's/^#Server/Server/' -e '/^#/d' | rankmirrors -n 5 -

Update system

sudo pacman -Suy sudo pacman -S yay # only works on Manjaro

Interesting terminal companions and programming languages

yay -S micro-manjaro
yay -S --needed thefuck nodenv nodenv-node-build-git rbenv ruby-build elixir fish tmux autoconf automake bison bind-tools fasd htop make patch ed fzf gcc mosh ruby tk yarn php php-fpm python-pip python-pillow python-numpy nfs-utils lsof strace tldr mosh
tldr --update

Interesting graphical interface apps and tools

yay -S --needed adobe-source-sans-pro-fonts chromium deluge firefox-developer-edition otf-fira-code otf-fira-sans p7zip ttf-roboto ttf-ubuntu-font-family
yay -S --needed albert-lite atom-editor-bin visual-studio-code-bin ttf-iosevka pinta simple-scan xsel python-pyusb

Interesting python packages

Interesting utilities

yay -S --needed telegram-desktop foliate flameshot neofetch

No password for sudo

sudo visudo -f /etc/sudoers.d/99-mine

Be careful with %wheel, I prefer to put just myself :D

%wheel ALL=(ALL) NOPASSWD: ALL
rodrigo ALL=(ALL) NOPASSWD: ALL

Using my .dotfiles with TMUX and Fish Shell

```sh
cd
git clone https://github.com/tmux-plugins/tpm ~/.tmux/plugins/tpm
git clone https://gitlab.com/rdlu/dotfiles.git .dotfiles
ln -s ~/.dotfiles/tmux.conf .tmux.conf
tmux source ~/.tmux.conf
ln -s ~/.dotfiles/fish/config.fish ~/.config/fish/config.fish
ln -s ~/.dotfiles/fish/omf ~/.config/omf
curl -L https://get.oh-my.fish | fish
sudo chsh (whoami) -s /usr/bin/fish
```

After that restart your fish prompt and:

omf update

CTRL+A; SHIFT+U for tmux plugins

Fixes and Workarounds for ArchLinux

I can use in emergencies or having bugs.

Using dnsmasq like in Ubuntu for DNS resolutions

It's useful when using Docker containers + Reverse Proxies. You can just add your local address for wildcard resolution later.

sudo pacman -S dhclient dnsmasq

Then configure NetworkManager:

micro /etc/NetworkManager/NetworkManager.conf

To this:

[main]
plugins=keyfile
dhcp=dhclient
dns=dnsmasq

Rereading /etc/hosts and restarting:

echo 'addn-hosts=/etc/hosts' | sudo tee /etc/NetworkManager/dnsmasq.d/hosts.conf > /dev/null
sudo systemctl restart NetworkManager

Adding wildcard subdomains to dnsmasq

Create/edit a NetworkManager configuration for dnsmasq hosts:

sudo micro /etc/NetworkManager/dnsmasq.d/hosts.conf

Add, changing the domain to your preference:

address=/domain.tld/127.0.0.3
address=/sub.domain.tld/127.0.0.4

Restart NetworkManager

sudo systemctl restart NetworkManager

Comparing new configuration files after updates

SUDO_EDITOR=meld sudo -e /etc/file{,.pacnew}

Bluetooth AD2P not working with Gnome

yay -S pulseaudio-bluetooth-a2dp-gdm-fix

Firefox Text fields: Using dark themes with Gnome

Create a new property using about:config

Name: widget.content.gtk-theme-override Value: Adwaita:light

Adwaita must be present, KDE users can rely on breeze.

Firefox versus HiDPI and Wayland

Firefox Developer Edition

cp /usr/share/applications/firefox-developer-edition.desktop ~/.local/share/applications/
sed -i 's/Exec=/Exec=env MOZ_ENABLE_WAYLAND=1 /g' ~/.local/share/applications/firefox-developer-edition.desktop

Regular Firefox

cp /usr/share/applications/firefox.desktop ~/.local/share/applications/
sed -i 's/Exec=/Exec=env MOZ_ENABLE_WAYLAND=1 /g' ~/.local/share/applications/firefox.desktop

Undoing

rm ~/.local/share/applications/firefox*

Slow Playback / Hardware Acceleration

If your video playback is slow, change in about:config this setting: layers.acceleration.force-enabled to true

It works just fine on my three main computers (AMD A8 with open source radeon/mesa; Intel i5 with iGPU and mesa; Intel i7 with nVidia GTX 1050Ti and proprietary nvidia)

GPG, SSH Auth with Keybase

Installing requirements:

sudo pacman -S kbfs keybase keybase-gui micro

If you are having problems with gpg-agent as ssh agent mode, restart gnupg from scratch (export your important keys first)

rm -rf ~/.gnupg

Login to Keybase using the GUI or keybase login. Now import private key and public keys from keybase:

keybase pgp export --secret | gpg --allow-secret-key-import --import
keybase pgp export | gpg --import

Setup GPG Agent:

micro ~/.gnupg/gpg-agent.conf

With the below contents:

enable-ssh-support

default-cache-ttl-ssh 10800
max-cache-ttl-ssh 10800

pinentry-program /usr/bin/pinentry-gnome3

Setup PAM environment as extra:

micro ~/.pam-environment

With the below contents:

SSH_AGENT_PID	DEFAULT=
SSH_AUTH_SOCK	DEFAULT="${XDG_RUNTIME_DIR}/gnupg/S.gpg-agent.ssh"

Add your keys to SSH control file, that keeps the "authorized" keys to use by the emulated SSH agent:

ls ~/.gnupg/private-keys-v1.d/ | sed s/.key// >> ~/.gnupg/sshcontrol

Now logout and login again, or event better, reboot your machine. Check your SSH keys with:

ssh-add -l

If you are getting errors like error fetching identities: agent refused operation then you need to remove offending keys that are not Authorization keys. You can check them with:

gpg --list-keys --with-keygrip

Then comment out the keys with # or remove auth with ! at ~/.gnupg/sshcontrol:

micro ~/.gnupg/sshcontrol````

KVM Setup on Arch

Basic installation

sudo pacman -S --needed qemu virt-manager virt-viewer dnsmasq vde2 bridge-utils openbsd-netcat ebtables iptables
sudo yay -S --needed libguestfs
sudo systemctl enable --now libvirtd.service

Enable normal user account to use KVM

sudo micro /etc/libvirt/libvirtd.conf

Uncomment these lines:

unix_sock_group = "libvirt"
unix_sock_rw_perms = "0770"

Back to shell:

sudo usermod -a -G libvirt $(whoami)
newgrp libvirt
sudo systemctl restart libvirtd.service

In my test we need more configuration to not get a laggy 2D experience, even on Manjaro KDE.

Julia

nteract notebooks plus IJulia Packages managed by pacman

It's a wonderful idea to delegate the package updates to the operating system.

ArchLinux AUR has many Julia packages already, including julia-ijulia for notebooks

yay -S julia-distrohelper julia-loadpath

Reboot or logout/login from your user, because these 2 packages manages the julia's loadpath using system packs as a fallback. Without that Julia envs will not recognize the system installed packages.

yay -S --needed julia-ijulia nteract-bin

Now you can install any packages starting with julia-* and notebook apps like nteract-bin will recognize Julia as a option to notebook environments.

Vivaldi Browser

Installation

yay -S --needed vivaldi

Proprietary codecs

You can install from AUR also, but they are slow to download and build (downloading an entire Google Chrome deb, etc)

yay -S --needed vivaldi-ffmpeg-codecs vivaldi-widevine vivaldi-codecs-ffmpeg-extra-bin

Or, below you can use the prebuilt binaries from Vivaldi to install in the user space.

Vivaldi Browser Proprietary Media on user space (no root)

The update-ffmpeg and update-widevine scripts included in the Vivaldi install directory are provided to fix situations where proprietary media (AVC/H.264 and AAC) and Widevine (DRM/EME) respectively, are not setup correctly.

These scripts are primarily intended to be run as root (or under sudo) as they create and update files and directories that are root owned. However both support a command line option (--user) that adjusts their installation directories and thus allows them to be run without escalation.

The --user options were made for internal usage, with locally ‘unpacked’ copies of Vivaldi (i.e. not installed). However, it is possible to use them with standard installs (albeit with a little tweaking in the case of update-widevine).

update-ffmpeg

To install proprietary media for just your current user issue the following:

/opt/vivaldi/update-ffmpeg --user

update-widevine

To install Widevine for just your current user, a couple more steps are required. This is because the script still attempts to adjust symlinks within the (typically root owned) Vivaldi install directory.

You can adjust the script as it runs, to disable the symlink creation, which would otherwise cause failure:

sed -r 's/^ *(rm|ln) -f.*/:/' /opt/vivaldi/update-widevine | sh -eus -- --user

Now create meta data in your Vivaldi user data directory that points to alternative Widevine install location–triple click to select the entire line:

echo "{\"Path\":\"$HOME/.local/lib/vivaldi/WidevineCdm\"}" > "${XDG_CONFIG_HOME:-$HOME/.config}/vivaldi/WidevineCdm/latest-component-updated-widevine-cdm"

Note: Change all “vivaldi” references in the above commands to “vivaldi-snapshot”, if you use the snapshot version.

(Original)[https://gist.github.com/ruario/995728dd8123540d331052e0ce648439]

Python Tips

Install

ArchLinux or Manjaro system packages

For Data Science / Engineering

yay -S --needed python-ipykernel python-pandas python-numexpr python-beautifulsoup4 python-sqlalchemy python-xlrd python-xlwt python-tabulate python-psycopg2 python-pymysql python-seaborn python-statsmodels gnuplot python-numpy python-pandas-datareader python-scipy python-openpyxl python-matplotlib python-bottleneck mysql-python python-mysql-connector cython python-thrift python-fastparquet

For development

yay -S --needed python-flask python-django python-bcrypt python-coverage python-pytest-pylint python-pylint python-sqlalchemy

Python VirtualEnvs

VirtualEnvs on Fish Shell

First you need to use oh my fish, preferably with my .dotfiles

omf install virtualfish
pip install virtualfish
vf install compat_aliases projects environment auto_activation global_requirements update_python 

Then for existing projects:

vf new myprojectenv
vf connect
cd
vf disconnect
cd ~/Projects/myproject

Check if the environment auto changes.

Pipenv

pacman -S python-pipenv pyenv
omf install pyenv
pyenv install 3.7.7
pipenv

Then on any project folder:

pipenv install --python 3.7

Django

Using Django Shell Plus inside a VS Code Notebook

Preparing Django and the Python Virtual Env

Django Shell Plus is a nice way to explore your django project.

First on your project venv install the requisites:

pip install jupyterlab jupyter ipython django-extensions

Now add django-extensions to your DJANGO_PROJECT/settings.py:

INSTALLED_APPS = [
    'django_extensions',

Setting a fixed authentication

It's boring to copy and paste the token everytime you start the notebook to the VSCode, so we have two ways:

ps: disabling jupyter security does not work currently in vscode

With a Password

A nice trick is to set a password instead copying and pasting the jupyter token everytime you starts it, because jupyter changes it everytime you starts it.

Run this:

jupyter notebook password

To define a local password. This way VSCode will ask your password at least 4 times, but with a known value.

With a fixed token

Open ~/.jupyter/jupyter_notebook_config.json with your favorite editor then paste:

{
  "NotebookApp": {
    "token": "MY_SECURE_TOKEN"
  }
}

The real trick is here: you need to use a "remote" jupyter notebook, because the notebook started by VS Code is in a state that I cannot load Django successfully, even settings paths, messing with os.change_dir and similar workarounds.

Press CTRL+SHIFT+P in VSCode and type remote jupyter. When asked for the server address put this:

http://localhost:8888/?token=MY_SECURE_TOKEN

Now reload VSCode. Run in a separate terminal the external Django Shell Plus notebook, inside your project dir:

cd DJANGO_PROJECT

Then

python manage.py shell_plus --notebook --no-browser

Create a new notebook file anywhere inside your project, like in DJANGO_PROJECT/notebooks/test.ipynb.

Before running your first django code, run any notebook cell, it will connect to your "remote" jupyter server and ask for the password, unless you are using the fixed token.

Now FINALLY change the kernel to Django Shell Plus on the VSCode GUI, upper right. Only then you can import your models and start exploring your django data.

On your first cell put this:

import os
os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"
from myapp.models import Model1, Model2
Model1.objects.first()

Fluent Python

Chapter 9 on pytest

Chapter 9 of the Fluent Python book, by Luciano Ramalho, presents a Vector2d class.

It's a good reminder and example for experienced developers that are learning python. For instance, I can remember lots of concepts at once and compare with my other known programming languages, like Ruby.

Directory Structure suggestion

I also converted to the follow structure, recommended for a isolated component/module in python. I was used to Rails framework conventions with a different autoloader, so this reminds me that I need to create this structure by myself in python way.

And yes, as a seasoned Rubist, it's weird to put __init__ everywhere. Don't forget that.

+ fluentpy
|- __init__.py
|+ chapter9
 |- __init__.py
 |- vector2d.py
|+ tests
 |- __init__.py
 |+ chapter9
  |- __init__.py
  |- test_vector2d.py

Tests, converted to pytest

The original tests Ramalho presents on the book are the also widely used doctest. Buy the book and see there the original. The book is worthy every penny. I just converted (and added some bits) to pytest to make most of my learning and to remember in the future.

The coverage is 100% of the original code.

In the file fluentpy/tests/chapter9/test_vector2d.py:

import math
import pytest

import fluentpy.chapter9.vector2d as v2d

class TestVector2d:
    v1 = v2d.Vector2d(3, 4)

    def test_init_int(self):
        x, y = self.v1
        assert (x, y) == (3.0, 4.0), 'Must return a tuple'

    def test_clone_eval(self):
        v1_clone = eval('v2d.' + repr(self.v1))
        assert self.v1 == v1_clone, \
            'repr must be evaluated'

    def test_str(self):
        assert str(self.v1) == '(3.0, 4.0)', \
            'str conversion need to print a tuple'

    def test_bytes(self):
        assert bytes(self.v1) == b'd\x00\x00\x00\x00\x00\x00\x08@\x00\x00\x00\x00\x00\x00\x10@', \
            'byte representation must match'

    def test_abs(self):
        assert abs(self.v1) == 5.0, \
            'abs must be working'

    def test_bool(self):
        assert (bool(self.v1), bool(v2d.Vector2d(0, 0))) == (True, False), \
            'bool conversion must evaluate False on null vectors'

    def test_from_bytes(self):
        v1_clone = v2d.Vector2d.frombytes(bytes(self.v1))
        assert self.v1 == v1_clone

    def test_format(self):
        assert format(self.v1) == '(3.0, 4.0)', \
            'format without params'
        assert format(self.v1, '.2f') == '(3.00, 4.00)', \
            'format 2 decimal places'
        assert format(self.v1, '.3e') == '(3.000e+00, 4.000e+00)', \
            'format cientific exp mode'

    def test_angle(self):
        assert v2d.Vector2d(0, 0).angle() == 0.0, \
            'angle on null vector is zero'
        assert v2d.Vector2d(1, 0).angle() == 0.0, \
            'angle with y component zero is zero'
        epsilon = 10 ** -8
        assert abs(v2d.Vector2d(0, 1).angle() - math.pi / 2) < epsilon, \
            'angle must be valid on circle'
        # interesting reading: https://www.euclideanspace.com/maths/geometry/trig/inverse/index.htm
        assert abs(v2d.Vector2d(1, 1).angle() - math.pi / 4) < epsilon

    def test_format_polar(self):
        v_1_1 = v2d.Vector2d(1, 1)
        assert format(v_1_1, 'p') == '<1.4142135623730951, 0.7853981633974483>', \
            'vector in polar format must match'
        assert format(v_1_1, '.3ep') == '<1.414e+00, 7.854e-01>', \
            'vector in polar format, with 3 decimals and exponential rest must match'
        assert format(v_1_1, '0.5fp') == '<1.41421, 0.78540>', \
            'vector in polar format, with 5 decimals, truncated rest must match'

    def test_x_y_handling(self):
        # x and y must be read only
        with pytest.raises(AttributeError):  # as e_info:
            self.v1.x = 1
        with pytest.raises(AttributeError):
            self.v1.y = 2

    def test_hashing(self):
        v2 = v2d.Vector2d(3.1, 4.2)
        assert hash(self.v1) == 7, \
            'hash of vector (3, 4) is 7'
        assert hash(v2) == 384307168202284039, \
            'hash of a non integer components must work'
        assert hash(self.v1) != hash(v2), \
            'hash of (3, 4) must not match to (3.n, 4.n)'
        assert hash(self.v1) == hash(v2d.Vector2d(3, 4)), \
            'two different instance of vectors with same components must match'

    def test_set(self):
        assert len({self.v1, v2d.Vector2d(0, 0)}) == 2, \
            'converting to a set must work, with two different vectors'
        assert len({self.v1, self.v1}) == 1, \
            'converting to set must work, with two equivalent vectors'

The original code to be tested

File: fluentpy/chapter9/vector2d.py

from array import array
import math


class Vector2d:
    type_code = 'd'

    def __init__(self, x, y):
        self.__x = float(x)
        self.__y = float(y)

    @property  # getter
    def x(self):
        return self.__x

    @property  # getter
    def y(self):
        return self.__y

    def __iter__(self):
        return (i for i in (self.x, self.y))

    def __repr__(self):
        class_name = type(self).__name__
        return "{}({!r}, {!r})".format(class_name, *self)

    def __str__(self):
        return str(tuple(self))

    def __bytes__(self):
        return (bytes([ord(self.type_code)])) + \
               bytes(array(self.type_code, self))

    def __eq__(self, other):
        return tuple(self) == tuple(other)

    def __hash__(self):
        return hash(self.x) ^ hash(self.y)

    def __abs__(self):
        return math.hypot(self.x, self.y)

    def __bool__(self):
        return bool(abs(self))

    def angle(self):
        return math.atan2(self.y, self.x)

    def __format__(self, format_spec=''):
        if format_spec.endswith('p'):
            format_spec = format_spec[:-1]
            coords = (abs(self), self.angle())
            outer_format = '<{}, {}>'
        else:
            coords = self
            outer_format = '({}, {})'
        components = (format(c, format_spec) for c in coords)
        return outer_format.format(*components)

    @classmethod
    def frombytes(cls, octets):
        type_code = chr(octets[0])
        memv = memoryview(octets[1:]).cast(type_code)
        return cls(*memv)

In this code we have lots of methods that are used by common python functions. For instance str(something) converts the something to string. The something needs to respond to __str__, that will be called by str(). It's the equivalent in ruby to implement to_s in a class/module and use the something.to_s, that is a convetion to string conversions in Ruby.

Another example is __iter__ that must be implemented to make the object compatible with splat/unpack syntax *something or "matching" with x, y = something.

Every detail here is important for a reason. You will not use everything always.

Other Data Tools

More languages to play on Manjaro and a notebook

yay -S nteract-bin ihaskell-git r sagemath-jupyter julia

NVIDIA Accelerated Docker containers on ArchLinux

yay -S nvidia-container-runtime
sudo tee /etc/docker/daemon.json <<EOF
{
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker
docker run nvidia/cuda:10.1-base nvidia-smi

Compiling Psycopg2 for Lambda on ArchLinux

Please pay attention to the version numbers, change to newer versions as appropriate.

cd ~/Downloads
pacman -S postgresql postgresql-libs
wget http://initd.org/psycopg/tarballs/PSYCOPG-2-7/psycopg2-2.7.5.tar.gz
tar xf psycopg2-2.7.5.tar.gz
cd psycopg2-2.7.5/
sed -i 's/libpq.a/libpq.so/g' setup.py
gedit setup.cfg

Change setup.cfg build_ext section to that:

[build_ext]
define = 
pg_config = /usr/bin/pg_config
use_pydatetime = 1
mx_include_dir = 
have_ssl = 0
static_libpq = 1
libraries = ssl crypto

After that, we will finally build:

python setup.py build
cd build/lib.linux-x86_64-3.7
cp -R psycopg2/ ~/Projects/{MY_LAMBDA_PROJECT}/````

Redshift

Grant read only access to a group on a schema

-- Create Read-Only Group     
CREATE GROUP ro_group;

-- Create User
CREATE USER ro_user WITH password PASSWORD;

-- Add User to Read-Only Group
ALTER GROUP ro_group ADD USER ro_user;

-- Grant Usage permission to Read-Only Group to specific Schema
GRANT USAGE ON SCHEMA "ro_schema" TO GROUP ro_group;

-- Grant Select permission to Read-Only Group to specific Schema
GRANT SELECT ON ALL TABLES IN SCHEMA "ro_schema" TO GROUP ro_group;

-- Alter Default Privileges to maintain the permissions on new tables
ALTER DEFAULT PRIVILEGES IN SCHEMA "ro_schema" GRANT SELECT ON TABLES TO GROUP ro_group;

-- Revoke CREATE privileges from group
REVOKE CREATE ON SCHEMA "ro_schema" FROM GROUP ro_group;

Testing sorting keys

create table datasci.test_table_compound
compound sortkey (created_at, id)
as select * from public.test_table;

create table datasci.test_table_interleaved
interleaved sortkey (created_at, id)
as select * from public.test_table;

This copy for me over a million rows tables works pretty fast (< 1 minute), so it's worthy to copy some optimized tables.

MySQL

Number generator

The purpose is to join or count in special ocasions you can rely on the index, or use a custom order, on old versions without powerful window functions.

You can use maths + join on numbers to create discrete time windows based on created_at, for instance.

CREATE OR REPLACE VIEW generator_16
AS SELECT 0 n UNION ALL SELECT 1  UNION ALL SELECT 2  UNION ALL 
   SELECT 3   UNION ALL SELECT 4  UNION ALL SELECT 5  UNION ALL
   SELECT 6   UNION ALL SELECT 7  UNION ALL SELECT 8  UNION ALL
   SELECT 9   UNION ALL SELECT 10 UNION ALL SELECT 11 UNION ALL
   SELECT 12  UNION ALL SELECT 13 UNION ALL SELECT 14 UNION ALL 
   SELECT 15;

CREATE OR REPLACE VIEW generator_256
AS SELECT ( ( hi.n << 4 ) | lo.n ) AS n
     FROM generator_16 lo, generator_16 hi;

CREATE OR REPLACE VIEW generator_4k
AS SELECT ( ( hi.n << 8 ) | lo.n ) AS n
     FROM generator_256 lo, generator_16 hi;

CREATE OR REPLACE VIEW generator_64k
AS SELECT ( ( hi.n << 8 ) | lo.n ) AS n
     FROM generator_256 lo, generator_256 hi;

CREATE OR REPLACE VIEW generator_1m
AS SELECT ( ( hi.n << 16 ) | lo.n ) AS n
     FROM generator_64k lo, generator_16 hi;
-- create table for results

drop table if exists numbers ;

create table `numbers` (
  `i` int(11) signed 
  , primary key(`i`)
) ENGINE=myisam DEFAULT CHARSET=latin1;

INSERT INTO numbers(i)
SELECT n FROM generator_64k WHERE n < 64000

Creating database and privileges

mysql -h 127.0.0.1 -uroot -p
CREATE USER 'ghost'@'%';
GRANT ALL PRIVILEGES ON ghost.* To 'ghost'@'%' IDENTIFIED BY 'ghost123';
FLUSH PRIVILEGES;

CTRL-D to disconnect

OpenWRT tricks

Blocking URLs at dnsmasq

uci add_list dhcp.@dnsmasq[0].address='/address.com/127.0.0.1'
uci commit dhcp
/etc/init.d/dnsmasq restart

Some useful packages

ca-bundle curl ddns-scripts ddns-scripts_cloudflare.com-v4 etherwake flent-tools ip-tiny iptables-mod-conntrack-extra iptables-mod-ipopt kmod-ifb kmod-ipt-conntrack-extra kmod-ipt-ipopt kmod-ipt-raw kmod-sched-cake kmod-sched-core kmod-udptunnel4 kmod-udptunnel6 kmod-wireguard libcap libcurl4 libelf1 libmbedtls12 libmnl0 libncurses6 libqrencode librt luci-app-ddns luci-app-sqm luci-app-wireguard luci-app-wol luci-compat luci-lib-ipkg luci-proto-wireguard nano netperf qrencode speedtest-netperf sqm-scripts tc terminfo wireguard wireguard-tools zlib

Ubuntu/Debian Based

Keybase

cd ~/Downloads
curl -O https://prerelease.keybase.io/keybase_amd64.deb
sudo dpkg -i keybase_amd64.deb
sudo apt-get install -f
run_keybase

Node / Yarn

curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list

sudo apt update
sudo apt install git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev software-properties-common libffi-dev nodejs yarn

Linux Brew

sudo apt install build-essential curl file git
sh -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)"
nano ~/.profile

test -d ~/.linuxbrew && eval $(~/.linuxbrew/bin/brew shellenv)
test -d /home/linuxbrew/.linuxbrew && eval $(/home/linuxbrew/.linuxbrew/bin/brew shellenv)
test -r ~/.bash_profile && echo "eval \$($(brew --prefix)/bin/brew shellenv)" >>~/.bash_profile
echo "eval \$($(brew --prefix)/bin/brew shellenv)" >>~/.profile

Configure a Ghost Blog at GCE

Node + Yarn + Requisites

DBUS and SETCAP are not installed by default on GCE Debian

curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -

curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -

echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list

sudo apt-get update && sudo apt-get install -y yarn nodejs dbus libcap2-bin wget

Update Practices

sudo apt update && sudo apt upgrade

weekly

Ghost Engine

echo "PATH=$PATH:$HOME/.yarn/bin" >> ~/.bashrc
source ~/.bashrc
yarn global add ghost-cli@latest
sudo rm -rf /var/www/ghost
sudo mkdir -p /var/www/ghost
sudo chown $(whoami): /var/www/ghost

ghost install --url=https://MYSITE.COM --admin-url=https://ADMIN.MYSITE.COM --db=sqlite3 --mail=SMTP --mailservice=Mailgun --mailuser=postmaster@MYSITE.COM --mailpass=MAILGUN_SMTP_PW --no-stack --no-setup-ssl --no-prompt -d /var/www/ghost

Update Practices

ghost update

whenever a new version comes

Caddy Webserver

Auto HTTPS using Let's Encrypt

Caddyfile

First we need to create a Caddyfile, that is a configuration file for Caddy HTTPS Server. nano Caddyfile or vim Caddyfile and paste this:

MYSITE.COM {
  status 404 /ghost
  proxy / http://127.0.0.1:2368 {
    transparent
    fail_timeout 300s
    header_upstream X-Forwarded-Ssl on
  }

  tls ADMINEMAIL@MYSITE.COM
  gzip
}

ADMIN.MYSITE.COM {
  status 404 //
  proxy / http://127.0.0.1:2368/ {
    transparent
    fail_timeout 300s
    header_upstream X-Forwarded-Ssl on
  }
  tls ADMINEMAIL@MYSITE.COM
  gzip
}

www.MYSITE.COM {
  redir https://MYSITE.COM
  tls ADMINEMAIL@MYSITE.COM
}

Obviously modify to your site settings.

I separated the admin site for hardening access. It has stricter rules at Cloudflare Edge. The ADMIN is some long string (kind of phrase password). It can make life harder for bots. Plus with stricter rules at my Cloudflare Edge for these bots (for instance, mandatory Captcha thoughter rate limits), it will not eat Network Fees from Google Cloud (or whatever you are using).

Installation

Now we will install and start the service:

curl https://getcaddy.com | bash -s personal
sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/caddy
sudo mkdir /etc/caddy
sudo chown -R root:www-data /etc/caddy
sudo mkdir /etc/ssl/caddy
sudo chown -R root:www-data /etc/ssl/caddy
sudo chmod 0770 /etc/ssl/caddy
sudo cp ~/Caddyfile /etc/caddy/
sudo chown www-data:www-data /etc/caddy/Caddyfile
sudo chmod 444 /etc/caddy/Caddyfile
sudo chown www-data:www-data /var/www
sudo chmod 555 /var/www
wget https://raw.githubusercontent.com/mholt/caddy/master/dist/init/linux-systemd/caddy.service
sudo cp caddy.service /etc/systemd/system/
sudo chown root:root /etc/systemd/system/caddy.service
sudo chmod 644 /etc/systemd/system/caddy.service
sudo systemctl daemon-reload
sudo systemctl start caddy.service
sudo systemctl enable caddy.service

Update practices

Whenever a new caddy version comes, or weekly:

curl https://getcaddy.com | bash -s personal
sudo pkill -USR2 caddy

Automating this is out of this doc scope (tip: CRON)

Logs

To follow connection logs on Caddy:

journalctl -f -u caddy.service

Or requests at node server:

journalctl -f -u ghost_MYSITE.service

Git

Redundancy

Add Multiple repositories

Refer to my dotfiles/fish/functions.fish file

Push to all remotes at same time

Creating an alias is a good idea

git config --global alias.pushall '!git remote | xargs -L1 git push --all'````