jcreek/jcreek.github.io
1
0
Fork 0
mirror of https://github.com/jcreek/jcreek.github.io.git synced 2026-07-13 02:53:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore(#16): Remove jekyll files

Browse Source
This commit is contained in:
Josh Creek
2022-10-20 22:15:01 +01:00
parent 38b094f99e
commit a931890dcd
28 changed files with 0 additions and 2931 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deploying/docker-compose-for-asp-net-core-with-postgres-and-s3-backups.md
-321
View File
@@ -1,321 +0,0 @@
---
layout: post
parent: Containers & Deploying
nav_order: 1
title: "Docker Compose for ASP.Net Core with Postgres + S3 backups"
date: 2021-07-12 21:01:45 +0100
categories: deploying docker
---
# {{page.title}}
_{{page.date}}_
In this post, I will cover how I set up the following application structure, run from a single `docker-compose` file:
- ASP.Net Core MVC & Razor Web Application
- ASP.Net Core Entity Framework Migrations
- PostgreSQL
- SMTP Server
- S3 Backups for PostgreSQL
- S3 Restores for PostgreSQL
The directory structure, including pertinent files, looks like this.
```
Repository Root Folder
│ .dockerignore
│ database.env
│ docker-compose.yml
│
└───ASP.Net Core Project Folder
│ │ Dockerfile
│ │ Migrations.Dockerfile
│ │ Setup.sh
│
└───postgres-backup-s3
│ │ Dockerfile
│
└───postgres-restore-s3
│ │ Dockerfile
```
Starting with an almost blank `docker-compose.yml` file, over the course of this post we'll add each service so that the entire infrastructure can be brought up with one single `docker-compose up` command.
```docker
version: '3.4'
services:
```
N.B. One optional thing not covered here is to include `restart: always` for each service in the `docker-compose.yml` file to ensure they come back online after the host machine reboots. I wouldn't recommend using this for anything but the core web app, database and mail server.
## ASP.Net Core Web Application
For this we want to specify a few key things:
- the name to give the container, to make it easier to work with than the docker auto-generated names
- the internal port to expose on the host machine's external port
- the Dockerfile to use to build the application
- the folder to map to make log files accessible from the host machine without having to use the terminal
- the environment variable to set the app into production mode instead of the default
- the other containers this one will depend on
```docker
version: '3.4'
services:
aspprojectname:
container_name: myaspprojectname
ports:
- "80:80"
build:
context: .
dockerfile: MyAspProjectName/Dockerfile
volumes:
- ./MyAspProjectName/logs:/app/logs
environment:
- ASPNETCORE_ENVIRONMENT=Production
depends_on:
- db
# - migrations
```
You will notice here that I am commenting out the migrations dependency. This is because with the S3 backup and restore images it's not really needed, and it requires more powerful hardware to run than any of the other images and is a large docker image, so worth avoiding if possible, or only using to set up the database initially, then deleting.
The Dockerfile itself is pretty standard for ASP.Net Core apps. In this instance, the app is running on dotnet 5.
```docker
FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443
FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build
WORKDIR /src
COPY ["MyAspProjectName/MyAspProjectName.csproj", "MyAspProjectName/"]
RUN dotnet restore "MyAspProjectName/MyAspProjectName.csproj" --disable-parallel
COPY . .
WORKDIR "/src/MyAspProjectName"
RUN dotnet build "MyAspProjectName.csproj" -c Release -o /app/build
FROM build AS publish
RUN dotnet publish "MyAspProjectName.csproj" -c Release -o /app/publish
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "MyAspProjectName.dll"]
```
## ASP.Net Core Entity Framework Migrations
This one is pretty similar to the main web app in terms of what needs adding to add the service to the `docker-compose.yml` file.
```docker
migrations:
container_name: dbmigrations
build:
context: .
dockerfile: MyAspProjectName/Migrations.Dockerfile
environment:
- ASPNETCORE_ENVIRONMENT=Production
depends_on:
- db
```
The Dockerfile itself is where the magic happens, building the web app, installing the global dotnet-ef tools needed and then running the migrations.
```docker
FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build
WORKDIR /src
COPY ["MyAspProjectName/MyAspProjectName.csproj", "MyAspProjectName/"]
COPY ["MyAspProjectName/Setup.sh", "MyAspProjectName/"]
ENV PATH="${PATH}:/root/.dotnet/tools"
RUN dotnet tool install --global dotnet-ef
RUN dotnet restore "MyAspProjectName/MyAspProjectName.csproj" --disable-parallel
COPY . .
WORKDIR "/src/MyAspProjectName/."
RUN /root/.dotnet/tools/dotnet-ef migrations add InitialMigrations
RUN chmod +x ./Setup.sh
CMD /bin/bash ./Setup.**sh**
```
Personally, I don't use this unless I have to, for the reasons previously stated, but it's worth sharing in case it's useful to anyone else.
## PostgreSQL
There's two key parts to add to the `docker-compose.yml` file for this one. The service itself, with the port to expose and an `env` file to store sensitive information, and the volume mapped to a directory within the container.
```docker
services:
db:
container_name: myappdb
image: "postgres"
ports:
- "5432:5432"
env_file:
- database.env # configure postgres
volumes:
- database-data:/var/lib/postgresql/data/ # persist data even if container shuts down
volumes:
database-data: # named volumes can be managed easier using docker-compose
```
That `env` file is remarkably simple.
```text
POSTGRES_USER=postgres
POSTGRES_PASSWORD=passwordgoeshere
POSTGRES_DB=yourdbname
```
## SMTP Server
The service for the SMTP server is super simple.
```docker
mail:
image: bytemark/smtp
```
## S3 Backups for PostgreSQL
The `docker-compose.yml` for this is as below, which sets a daily schedule and connection details for both the Postgres database and S3. Note that the Postgres host uses the internal Docker hostname for the database container.
```docker
pgbackups3:
build:
context: .
dockerfile: postgres-backup-s3/Dockerfile
links:
- db
environment:
SCHEDULE: '@daily'
S3_REGION: eu-west-2
S3_ACCESS_KEY_ID: keygoeshere
S3_SECRET_ACCESS_KEY: secretkeygoeshere
S3_BUCKET: yourapp-backups
S3_PREFIX: backup
POSTGRES_HOST: db
POSTGRES_DATABASE: yourdbname
POSTGRES_USER: postgres
POSTGRES_PASSWORD: passwordgoeshere
POSTGRES_EXTRA_OPTS: '--schema=public --blobs'
```
I won't go into how this application works here, that'll be in another post.
## S3 Restores for PostgreSQL
The `docker-compose.yml` for this is as below, which sets connection details for both the Postgres database and S3. Note that the Postgres host uses the internal Docker hostname for the database container.
This container should be run at setup time then stopped and commented out of the `docker-compose.yml` file or removed entirely, to prevent it from being accidentally run and restoring unintentionally, overwriting the database (notice the drop public option - this will wipe everything in the database before restoring).
```docker
pgrestores3:
build:
context: .
dockerfile: postgres-restore-s3/Dockerfile
links:
- db
environment:
S3_ACCESS_KEY_ID: keygoeshere
S3_SECRET_ACCESS_KEY: secretkeygoeshere
S3_BUCKET: yourapp-backups
S3_PREFIX: backup
POSTGRES_HOST: db
POSTGRES_DATABASE: yourdbname
POSTGRES_USER: postgres
POSTGRES_PASSWORD: passwordgoeshere
DROP_PUBLIC: 'yes'
```
I won't go into how this application works here, that'll be in another post.
## Putting it all together
The complete `docker-compose.yml` file, after the initial `docker-compose up` command has been run, looks like this. Note that migrations is commented out (we don't need it as we have the S3 backup and restore) and that the S3 restore is commented out to prevent accidental restores.
```docker
version: '3.4'
services:
aspprojectname:
container_name: myaspprojectname
ports:
- "80:80"
build:
context: .
dockerfile: MyAspProjectName/Dockerfile
volumes:
- ./MyAspProjectName/logs:/app/logs
environment:
- ASPNETCORE_ENVIRONMENT=Production
depends_on:
- db
# - migrations
# migrations:
# container_name: dbmigrations
# build:
# context: .
# dockerfile: MyAspProjectName/Migrations.Dockerfile
# environment:
# - ASPNETCORE_ENVIRONMENT=Production
# depends_on:
# - db
db:
container_name: myappdb
image: "postgres"
ports:
- "5432:5432"
env_file:
- database.env # configure postgres
volumes:
- database-data:/var/lib/postgresql/data/ # persist data even if container shuts down
mail:
image: bytemark/smtp
pgbackups3:
build:
context: .
dockerfile: postgres-backup-s3/Dockerfile
links:
- db
environment:
SCHEDULE: '@daily'
S3_REGION: eu-west-2
S3_ACCESS_KEY_ID: keygoeshere
S3_SECRET_ACCESS_KEY: secretkeygoeshere
S3_BUCKET: yourapp-backups
S3_PREFIX: backup
POSTGRES_HOST: db
POSTGRES_DATABASE: yourdbname
POSTGRES_USER: postgres
POSTGRES_PASSWORD: passwordgoeshere
POSTGRES_EXTRA_OPTS: '--schema=public --blobs'
# pgrestores3:
# build:
# context: .
# dockerfile: postgres-restore-s3/Dockerfile
# links:
# - db
# environment:
# S3_ACCESS_KEY_ID: keygoeshere
# S3_SECRET_ACCESS_KEY: secretkeygoeshere
# S3_BUCKET: yourapp-backups
# S3_PREFIX: backup
# POSTGRES_HOST: db
# POSTGRES_DATABASE: yourdbname
# POSTGRES_USER: postgres
# POSTGRES_PASSWORD: passwordgoeshere
# DROP_PUBLIC: 'yes'
volumes:
database-data: # named volumes can be managed easier using docker-compose
```
docs/deploying/docker-compose-s3-postgres-backup-and-restore.md
-316
View File
@@ -1,316 +0,0 @@
---
layout: post
parent: Containers & Deploying
nav_order: 2
title: "Docker Compose for S3 Backup and Restore of PostgreSQL"
date: 2021-07-12 21:27:19 +0100
categories: deploying docker
---
# {{page.title}}
_{{page.date}}_
In this post, I will cover how you can set up two Docker containers for backing up and restoring Postgres databases using S3 in AWS.
The complete `docker-compose.yml` file looks like this. Note that it's set to link these containers to a database container also configured within the same compose file - this could be modified to connect to an external database if desired by removing the `links` and changing the `POSTGRES_HOST` environment variables.
```docker
version: '3.4'
services:
pgbackups3:
build:
context: .
dockerfile: postgres-backup-s3/Dockerfile
links:
- db
environment:
SCHEDULE: '@daily'
S3_REGION: eu-west-2
S3_ACCESS_KEY_ID: keygoeshere
S3_SECRET_ACCESS_KEY: secretkeygoeshere
S3_BUCKET: yourapp-backups
S3_PREFIX: backup
POSTGRES_HOST: db
POSTGRES_DATABASE: yourdbname
POSTGRES_USER: postgres
POSTGRES_PASSWORD: passwordgoeshere
POSTGRES_EXTRA_OPTS: '--schema=public --blobs'
pgrestores3:
build:
context: .
dockerfile: postgres-restore-s3/Dockerfile
links:
- db
environment:
S3_ACCESS_KEY_ID: keygoeshere
S3_SECRET_ACCESS_KEY: secretkeygoeshere
S3_BUCKET: yourapp-backups
S3_PREFIX: backup
POSTGRES_HOST: db
POSTGRES_DATABASE: yourdbname
POSTGRES_USER: postgres
POSTGRES_PASSWORD: passwordgoeshere
DROP_PUBLIC: 'yes'
```
## Backing up PostgreSQL to S3
The Dockerfile used looks like this:
```docker
FROM alpine:3.13
RUN apk update \
&& apk add coreutils \
&& apk add postgresql-client \
&& apk add python3 py3-pip && pip3 install --upgrade pip && pip3 install awscli \
&& apk add openssl \
&& apk add curl \
&& curl -L --insecure https://github.com/odise/go-cron/releases/download/v0.0.6/go-cron-linux.gz | zcat > /usr/local/bin/go-cron && chmod u+x /usr/local/bin/go-cron \
&& apk del curl \
&& rm -rf /var/cache/apk/*
ENV POSTGRES_DATABASE **None**
ENV POSTGRES_HOST **None**
ENV POSTGRES_PORT 5432
ENV POSTGRES_USER **None**
ENV POSTGRES_PASSWORD **None**
ENV POSTGRES_EXTRA_OPTS ''
ENV S3_ACCESS_KEY_ID **None**
ENV S3_SECRET_ACCESS_KEY **None**
ENV S3_BUCKET **None**
ENV S3_REGION us-west-1
ENV S3_PATH 'backup'
ENV S3_ENDPOINT **None**
ENV S3_S3V4 no
ENV SCHEDULE **None**
COPY ["postgres-backup-s3/run.sh", "run.sh"]
COPY ["postgres-backup-s3/backup.sh", "backup.sh"]
CMD ["sh", "run.sh"]
```
This is a combination of parts from [here](https://github.com/itbm/postgresql-backup-s3/blob/master/Dockerfile) and the original [here](https://github.com/schickling/dockerfiles/tree/master/postgres-backup-s3) with some further customisation by me.
It essentially sets up a Linux environment to be able to connect to both S3 and Postgres, and to be able to run two script files.
### `run.sh`
This one handles a bit of aws configuration for S3 connections, as well as setting up the behaviour for scheduling or just immediately running the container. I like to set it to run daily so I can set it and forget it.
```bash
#! /bin/sh
set -e
if [ "${S3_S3V4}" = "yes" ]; then
aws configure set default.s3.signature_version s3v4
fi
if [ "${SCHEDULE}" = "**None**" ]; then
sh backup.sh
else
exec go-cron "$SCHEDULE" /bin/sh backup.sh
fi
```
### `backup.sh`
This one handles checking all the requisite connection details are present, then makes a compressed dump file of the PostgreSQL database, and uploads it to S3.
```bash
#! /bin/sh
set -e
set -o pipefail
if [ "${S3_ACCESS_KEY_ID}" = "**None**" ]; then
echo "You need to set the S3_ACCESS_KEY_ID environment variable."
exit 1
fi
if [ "${S3_SECRET_ACCESS_KEY}" = "**None**" ]; then
echo "You need to set the S3_SECRET_ACCESS_KEY environment variable."
exit 1
fi
if [ "${S3_BUCKET}" = "**None**" ]; then
echo "You need to set the S3_BUCKET environment variable."
exit 1
fi
if [ "${POSTGRES_DATABASE}" = "**None**" ]; then
echo "You need to set the POSTGRES_DATABASE environment variable."
exit 1
fi
if [ "${POSTGRES_HOST}" = "**None**" ]; then
if [ -n "${POSTGRES_PORT_5432_TCP_ADDR}" ]; then
POSTGRES_HOST=$POSTGRES_PORT_5432_TCP_ADDR
POSTGRES_PORT=$POSTGRES_PORT_5432_TCP_PORT
else
echo "You need to set the POSTGRES_HOST environment variable."
exit 1
fi
fi
if [ "${POSTGRES_USER}" = "**None**" ]; then
echo "You need to set the POSTGRES_USER environment variable."
exit 1
fi
if [ "${POSTGRES_PASSWORD}" = "**None**" ]; then
echo "You need to set the POSTGRES_PASSWORD environment variable or link to a container named POSTGRES."
exit 1
fi
if [ "${S3_ENDPOINT}" == "**None**" ]; then
AWS_ARGS=""
else
AWS_ARGS="--endpoint-url ${S3_ENDPOINT}"
fi
# env vars needed for aws tools
export AWS_ACCESS_KEY_ID=$S3_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY=$S3_SECRET_ACCESS_KEY
export AWS_DEFAULT_REGION=$S3_REGION
export PGPASSWORD=$POSTGRES_PASSWORD
POSTGRES_HOST_OPTS="-h $POSTGRES_HOST -p $POSTGRES_PORT -U $POSTGRES_USER $POSTGRES_EXTRA_OPTS"
echo "Creating dump of ${POSTGRES_DATABASE} database from ${POSTGRES_HOST}..."
pg_dump $POSTGRES_HOST_OPTS $POSTGRES_DATABASE | gzip > dump.sql.gz
echo "Uploading dump to $S3_BUCKET"
cat dump.sql.gz | aws $AWS_ARGS s3 cp - s3://$S3_BUCKET/$S3_PREFIX/${POSTGRES_DATABASE}_$(date +"%Y-%m-%dT%H:%M:%SZ").sql.gz || exit 2
echo "SQL backup uploaded successfully"
```
## Restoring PostgreSQL from S3
The Dockerfile used looks like this:
```docker
FROM alpine:3.13
RUN apk update \
&& apk add coreutils \
&& apk add postgresql-client \
&& apk add python3 py3-pip && pip3 install --upgrade pip && pip3 install awscli \
&& apk add openssl \
&& apk add curl \
&& curl -L --insecure https://github.com/odise/go-cron/releases/download/v0.0.6/go-cron-linux.gz | zcat > /usr/local/bin/go-cron && chmod u+x /usr/local/bin/go-cron \
&& apk del curl \
&& rm -rf /var/cache/apk/*
ENV POSTGRES_DATABASE **None**
ENV POSTGRES_HOST **None**
ENV POSTGRES_PORT 5432
ENV POSTGRES_USER **None**
ENV POSTGRES_PASSWORD **None**
ENV S3_ACCESS_KEY_ID **None**
ENV S3_SECRET_ACCESS_KEY **None**
ENV S3_BUCKET **None**
ENV S3_REGION us-west-1
ENV S3_PATH 'backup'
ENV DROP_PUBLIC 'no'
COPY ["postgres-restore-s3/restore.sh", "restore.sh"]
CMD ["sh", "restore.sh"]
```
This is once again a combination of parts from [here](https://github.com/itbm/postgresql-backup-s3/blob/master/Dockerfile) and the original [here](https://github.com/schickling/dockerfiles/tree/master/postgres-backup-s3) with some further customisation by me.
It essentially sets up a Linux environment to be able to connect to both S3 and Postgres, and to be able to run one script file.
### `restore.sh`
This one handles checking all the requisite connection details are present, then retrieves the most recent compressed dump file of the PostgreSQL database from S3, drops everything in the public space in PostgreSQL, and restores the backup. Crucially it then deletes the downloaded backup dump file to enable running the same container again for future restores without issue.
```bash
#! /bin/sh
set -e
set -o pipefail
if [ "${S3_ACCESS_KEY_ID}" = "**None**" ]; then
echo "You need to set the S3_ACCESS_KEY_ID environment variable."
exit 1
fi
if [ "${S3_SECRET_ACCESS_KEY}" = "**None**" ]; then
echo "You need to set the S3_SECRET_ACCESS_KEY environment variable."
exit 1
fi
if [ "${S3_BUCKET}" = "**None**" ]; then
echo "You need to set the S3_BUCKET environment variable."
exit 1
fi
if [ "${POSTGRES_DATABASE}" = "**None**" ]; then
echo "You need to set the POSTGRES_DATABASE environment variable."
exit 1
fi
if [ "${POSTGRES_HOST}" = "**None**" ]; then
if [ -n "${POSTGRES_PORT_5432_TCP_ADDR}" ]; then
POSTGRES_HOST=$POSTGRES_PORT_5432_TCP_ADDR
POSTGRES_PORT=$POSTGRES_PORT_5432_TCP_PORT
else
echo "You need to set the POSTGRES_HOST environment variable."
exit 1
fi
fi
if [ "${POSTGRES_USER}" = "**None**" ]; then
echo "You need to set the POSTGRES_USER environment variable."
exit 1
fi
if [ "${POSTGRES_PASSWORD}" = "**None**" ]; then
echo "You need to set the POSTGRES_PASSWORD environment variable or link to a container named POSTGRES."
exit 1
fi
# env vars needed for aws tools
export AWS_ACCESS_KEY_ID=$S3_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY=$S3_SECRET_ACCESS_KEY
export AWS_DEFAULT_REGION=$S3_REGION
export PGPASSWORD=$POSTGRES_PASSWORD
POSTGRES_HOST_OPTS="-h $POSTGRES_HOST -p $POSTGRES_PORT -U $POSTGRES_USER"
echo "Finding latest backup"
LATEST_BACKUP=$(aws s3 ls s3://$S3_BUCKET/$S3_PREFIX/ | sort | tail -n 1 | awk '{ print $4 }')
echo "Fetching ${LATEST_BACKUP} from S3"
aws s3 cp s3://$S3_BUCKET/$S3_PREFIX/${LATEST_BACKUP} dump.sql.gz
gzip -d dump.sql.gz
if [ "${DROP_PUBLIC}" == "yes" ]; then
echo "Recreating the public schema"
psql $POSTGRES_HOST_OPTS -d $POSTGRES_DATABASE -c "drop schema public cascade; create schema public;"
fi
echo "Restoring ${LATEST_BACKUP}"
psql $POSTGRES_HOST_OPTS -d $POSTGRES_DATABASE < dump.sql
echo "Restore complete"
rm -f ./dump.sql
echo "Deleted dump files"
```
docs/deploying/elasticsearch-and-kibana.md
-67
View File
@@ -1,67 +0,0 @@
---
layout: post
parent: Containers & Deploying
nav_order: 3
title: "Docker Compose for Elasticsearch and Kibana 7.9"
date: 2021-07-13 21:38:05 +0100
categories: deploying docker
---
# {{page.title}}
_{{page.date}}_
In this post, I will cover how you can set up Elasticsearch and Kibana with a single `docker-compose.yml` file.
## Set up and install ELK stack on Ubuntu server
You must run `sudo sysctl -w vm.max_map_count=262144` to get Elasticsearch to work. To make this permanent, run `sudo nano /etc/sysctl.conf` and add `vm.max_map_count=262144` to the end of the file on a new line, then save and exit.
Go to the directory with the files in. Make sure you've used SFTP to put the `docker-compose.yml` file in there first. For example you may need to use this command `cd /home/administrator/docker/elk-stack`
Install it all with `docker-compose up -d` and you're finished! It really is that simple.
The compose file is as below.
```docker
version: '2.2'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.3
container_name: elasticsearch
environment:
- node.name=elasticsearch
- discovery.seed_hosts=elasticsearch
- cluster.initial_master_nodes=elasticsearch
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms2g -Xmx2g"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- esdata1:/usr/share/elasticsearch/data
ports:
- 9200:9200
kibana:
image: docker.elastic.co/kibana/kibana:7.9.3
container_name: kibana
environment:
ELASTICSEARCH_URL: "http://elasticsearch:9200"
ELASTICSEARCH_SHARDTIMEOUT: 300000
ports:
- 5601:5601
depends_on:
- elasticsearch
volumes:
esdata1:
driver: local
```
docs/deploying/index.md
-6
View File
@@ -1,6 +0,0 @@
---
layout: default
title: Containers & Deploying
nav_order: 2
has_children: true
---