feat(*): Add post - Docker Compose for ASP.Net Core with Postgres + S3 backups

2026-07-12 18:43:50 +00:00 · 2021-07-12 22:35:21 +01:00
parent 98bd29d26c
commit a5b3be56dc
1 changed files with 321 additions and 0 deletions
@@ -0,0 +1,321 @@
+---
+layout: post
+parent: Containers & Deploying
+nav_order: 1
+title:  "Docker Compose for ASP.Net Core with Postgres + S3 backups"
+date:   2021-07-11 23:01:45 +0100
+categories: deploying docker
+---
+
+# {{page.title}}
+
+_{{page.date}}_
+
+In this post, I will cover how I set up the following application structure, run from a single `docker-compose` file:
+
+- ASP.Net Core MVC & Razor Web Application
+- ASP.Net Core Entity Framework Migrations
+- PostgreSQL
+- SMTP Server
+- S3 Backups for PostgreSQL
+- S3 Restores for PostgreSQL
+
+The directory structure, including pertinent files, looks like this.
+
+```
+Repository Root Folder
+│   .dockerignore
+│   database.env  
+│   docker-compose.yml
+│
+└───ASP.Net Core Project Folder
+│   │   Dockerfile
+│   │   Migrations.Dockerfile
+│   │   Setup.sh
+│   
+└───postgres-backup-s3
+│    │   Dockerfile
+│   
+└───postgres-restore-s3
+│    │   Dockerfile
+```
+
+Starting with an almost blank `docker-compose.yml` file, over the course of this post we'll add each service so that the entire infrastructure can be brought up with one single `docker-compose up` command.
+
+```docker
+version: '3.4'
+
+services:
+```
+
+N.B. One optional thing not covered here is to include `restart: always` for each service in the `docker-compose.yml` file to ensure they come back online after the host machine reboots. I wouldn't recommend using this for anything but the core web app, database and mail server.
+
+## ASP.Net Core Web Application
+
+For this we want to specify a few key things:
+
+- the name to give the container, to make it easier to work with than the docker auto-generated names
+- the internal port to expose on the host machine's external port
+- the Dockerfile to use to build the application
+- the folder to map to make log files accessible from the host machine without having to use the terminal
+- the environment variable to set the app into production mode instead of the default
+- the other containers this one will depend on
+
+```docker
+version: '3.4'
+
+services:
+  aspprojectname:
+    container_name: myaspprojectname
+    ports:
+      - "80:80"
+    build:
+      context: .
+      dockerfile: MyAspProjectName/Dockerfile
+    volumes:
+      - ./MyAspProjectName/logs:/app/logs
+    environment:
+      - ASPNETCORE_ENVIRONMENT=Production
+    depends_on:
+      - db
+      # - migrations
+```
+
+You will notice here that I am commenting out the migrations dependency. This is because with the S3 backup and restore images it's not really needed, and it requires more powerful hardware to run than any of the other images and is a large docker image, so worth avoiding if possible, or only using to set up the database initially, then deleting.
+
+The Dockerfile itself is pretty standard for ASP.Net Core apps. In this instance, the app is running on dotnet 5. 
+
+```docker
+FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS base
+WORKDIR /app
+EXPOSE 80
+EXPOSE 443
+
+FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build
+WORKDIR /src
+COPY ["MyAspProjectName/MyAspProjectName.csproj", "MyAspProjectName/"]
+RUN dotnet restore "MyAspProjectName/MyAspProjectName.csproj" --disable-parallel
+COPY . .
+WORKDIR "/src/MyAspProjectName"
+RUN dotnet build "MyAspProjectName.csproj" -c Release -o /app/build
+
+FROM build AS publish
+RUN dotnet publish "MyAspProjectName.csproj" -c Release -o /app/publish
+
+FROM base AS final
+WORKDIR /app
+COPY --from=publish /app/publish .
+ENTRYPOINT ["dotnet", "MyAspProjectName.dll"]
+```
+
+## ASP.Net Core Entity Framework Migrations
+
+This one is pretty similar to the main web app in terms of what needs adding to add the service to the `docker-compose.yml` file.
+
+```docker
+  migrations:
+    container_name: dbmigrations
+    build: 
+      context: .
+      dockerfile: MyAspProjectName/Migrations.Dockerfile
+    environment:
+      - ASPNETCORE_ENVIRONMENT=Production
+    depends_on: 
+      - db
+```
+
+The Dockerfile itself is where the magic happens, building the web app, installing the global dotnet-ef tools needed and then running the migrations.
+
+```docker
+FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build
+
+WORKDIR /src
+COPY ["MyAspProjectName/MyAspProjectName.csproj", "MyAspProjectName/"]
+COPY ["MyAspProjectName/Setup.sh", "MyAspProjectName/"]
+
+ENV PATH="${PATH}:/root/.dotnet/tools"
+RUN dotnet tool install --global dotnet-ef
+
+RUN dotnet restore "MyAspProjectName/MyAspProjectName.csproj" --disable-parallel
+COPY . .
+WORKDIR "/src/MyAspProjectName/."
+
+RUN /root/.dotnet/tools/dotnet-ef migrations add InitialMigrations
+
+RUN chmod +x ./Setup.sh
+CMD /bin/bash ./Setup.**sh**
+```
+
+Personally, I don't use this unless I have to, for the reasons previously stated, but it's worth sharing in case it's useful to anyone else.
+
+## PostgreSQL
+
+There's two key parts to add to the `docker-compose.yml` file for this one. The service itself, with the port to expose and an `env` file to store sensitive information, and the volume mapped to a directory within the container.
+
+```docker
+services:
+  db:
+    container_name: myappdb
+    image: "postgres"
+    ports:
+      - "5432:5432"
+    env_file:
+    - database.env # configure postgres
+    volumes:
+    - database-data:/var/lib/postgresql/data/ # persist data even if container shuts down
+
+volumes:
+    database-data: # named volumes can be managed easier using docker-compose
+```
+
+That `env` file is remarkably simple.
+
+```text
+POSTGRES_USER=postgres
+POSTGRES_PASSWORD=passwordgoeshere
+POSTGRES_DB=yourdbname
+```
+
+## SMTP Server
+
+The service for the SMTP server is super simple.
+
+```docker
+mail:
+    image: bytemark/smtp
+```
+
+## S3 Backups for PostgreSQL
+
+The `docker-compose.yml` for this is as below, which sets a daily schedule and connection details for both the Postgres database and S3. Note that the Postgres host uses the internal Docker hostname for the database container.
+
+```docker
+pgbackups3:
+    build:
+      context: .
+      dockerfile: postgres-backup-s3/Dockerfile
+    links:
+      - db
+    environment:
+      SCHEDULE: '@daily'
+      S3_REGION: eu-west-2
+      S3_ACCESS_KEY_ID: keygoeshere
+      S3_SECRET_ACCESS_KEY: secretkeygoeshere
+      S3_BUCKET: yourapp-backups
+      S3_PREFIX: backup
+      POSTGRES_HOST: db
+      POSTGRES_DATABASE: yourdbname
+      POSTGRES_USER: postgres
+      POSTGRES_PASSWORD: passwordgoeshere
+      POSTGRES_EXTRA_OPTS: '--schema=public --blobs'
+```
+
+I won't go into how this application works here, that'll be in another post.
+
+## S3 Restores for PostgreSQL
+
+The `docker-compose.yml` for this is as below, which sets  connection details for both the Postgres database and S3. Note that the Postgres host uses the internal Docker hostname for the database container.
+
+This container should be run at setup time then stopped and commented out of the `docker-compose.yml` file or removed entirely, to prevent it from being accidentally run and restoring unintentionally, overwriting the database (notice the drop public option - this will wipe everything in the database before restoring).
+
+```docker
+pgrestores3:
+    build:
+      context: .
+      dockerfile: postgres-restore-s3/Dockerfile
+    links:
+      - db
+    environment:
+      S3_ACCESS_KEY_ID: keygoeshere
+      S3_SECRET_ACCESS_KEY: secretkeygoeshere
+      S3_BUCKET: yourapp-backups
+      S3_PREFIX: backup
+      POSTGRES_HOST: db
+      POSTGRES_DATABASE: yourdbname
+      POSTGRES_USER: postgres
+      POSTGRES_PASSWORD: passwordgoeshere
+      DROP_PUBLIC: 'yes'
+```
+
+I won't go into how this application works here, that'll be in another post.
+
+## Putting it all together
+
+The complete `docker-compose.yml` file, after the initial `docker-compose up` command has been run, looks like this. Note that migrations is commented out (we don't need it as we have the S3 backup and restore) and that the S3 restore is commented out to prevent accidental restores.
+
+```docker
+version: '3.4'
+
+services:
+  aspprojectname:
+    container_name: myaspprojectname
+    ports:
+      - "80:80"
+    build:
+      context: .
+      dockerfile: MyAspProjectName/Dockerfile
+    volumes:
+      - ./MyAspProjectName/logs:/app/logs
+    environment:
+      - ASPNETCORE_ENVIRONMENT=Production
+    depends_on:
+      - db
+      # - migrations
+    # migrations:
+    # container_name: dbmigrations
+    # build: 
+    #     context: .
+    #     dockerfile: MyAspProjectName/Migrations.Dockerfile
+    # environment:
+    #     - ASPNETCORE_ENVIRONMENT=Production
+    # depends_on: 
+    #     - db
+    db:
+        container_name: myappdb
+        image: "postgres"
+        ports:
+            - "5432:5432"
+        env_file:
+        - database.env # configure postgres
+        volumes:
+        - database-data:/var/lib/postgresql/data/ # persist data even if container shuts down
+    mail:
+        image: bytemark/smtp
+    pgbackups3:
+        build:
+            context: .
+            dockerfile: postgres-backup-s3/Dockerfile
+        links:
+            - db
+        environment:
+            SCHEDULE: '@daily'
+            S3_REGION: eu-west-2
+            S3_ACCESS_KEY_ID: keygoeshere
+            S3_SECRET_ACCESS_KEY: secretkeygoeshere
+            S3_BUCKET: yourapp-backups
+            S3_PREFIX: backup
+            POSTGRES_HOST: db
+            POSTGRES_DATABASE: yourdbname
+            POSTGRES_USER: postgres
+            POSTGRES_PASSWORD: passwordgoeshere
+            POSTGRES_EXTRA_OPTS: '--schema=public --blobs'    
+    # pgrestores3:
+    #     build:
+    #         context: .
+    #         dockerfile: postgres-restore-s3/Dockerfile
+    #     links:
+    #         - db
+    #     environment:
+    #         S3_ACCESS_KEY_ID: keygoeshere
+    #         S3_SECRET_ACCESS_KEY: secretkeygoeshere
+    #         S3_BUCKET: yourapp-backups
+    #         S3_PREFIX: backup
+    #         POSTGRES_HOST: db
+    #         POSTGRES_DATABASE: yourdbname
+    #         POSTGRES_USER: postgres
+    #         POSTGRES_PASSWORD: passwordgoeshere
+    #         DROP_PUBLIC: 'yes'
+
+volumes:
+    database-data: # named volumes can be managed easier using docker-compose
+```