jcreek.github.io/search/search_index.json

{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":""},{"location":"#welcome-to-the-site","title":"Welcome to the site","text":"<p>This is the place where I ramble about technology and coding. Sometimes Computer Science education too.</p> <p>Some tools you might be interested in:</p> <ul> <li>Estimation Poker - a web app designed to assist scrum teams in performing estimations quickly and with minimal effort.</li> <li>Selection Wheel - a random selection spinner, use as a web app or install using Chrome on Windows and Android, or Safari on Mac and iOS.</li> <li>Hathorsoft Plan - a lesson planning tool designed for trainee teachers and NQTs but suitable for all.</li> </ul>"},{"location":"#about-me","title":"About Me","text":""},{"location":"#highlights","title":"Highlights","text":"<ul> <li>Full Stack Web Developer, with a bit of devops and old-school (literally) network management</li> <li>Computer Science Educator</li> <li>Teach First 2015 Ambassador</li> <li>Writer</li> <li>Photographer</li> <li>Content Creator</li> <li>Google Trusted Tester, plus once worked at Google for 2 weeks</li> </ul>"},{"location":"#blurb","title":"Blurb","text":"<p>I like to spend my time working on agile projects to solve problems in the education and ecommerce spheres, creating educational content, writing poetry, doing portraiture and event photography, playing competitive video games and playing piano and ukulele or singing in a choir.</p> <p>I also enjoy recording audiobooks for LibriVox, making YouTube videos on a variety of themes and streaming on Twitch. You can find me using the links below.</p>"},{"location":"#technologies","title":"Technologies","text":"<p>I code in a variety of languages, and use a number of different tools. Here they are for your perusal.</p> <ul> <li>HTML5</li> <li>CSS3</li> <li>JavaScipt ES6</li> <li>SCSS</li> <li>Node.js</li> <li>Vue.js</li> <li>jQuery</li> <li>C#</li> <li>ASP.Net Core</li> <li>SQL/MSSQL/PostgreSQL/MySQL</li> <li>Wordpress (if I really have to)</li> <li>MongoDB (if I really have to)</li> <li>Python (a bit rusty these days but fine for making simple scripts)</li> <li>Docker &amp; Docker Compose</li> <li>AWS &amp; Azure</li> <li>Proxmox &amp; TrueNAS Scale</li> <li>Windows/MacOS/Linux</li> <li>Photoshop/Lightroom</li> <li>Davinci Resolve/Premiere Pro</li> </ul>"},{"location":"#education","title":"Education","text":""},{"location":"#ucl-institute-of-education","title":"UCL Institute of Education","text":"<p>Postgraduate Certificate in Education</p>"},{"location":"#university-of-exeter","title":"University of Exeter","text":"<p>English with Proficiency in French</p>"},{"location":"#awards","title":"Awards","text":"<ul> <li>2011 Headgate Theatre Young Playwrights Award</li> <li>2013 Microsoft Imagine Cup World Citizenship Award (Ticklo)</li> </ul>"},{"location":"deploying/docker-compose-for-asp-net-core-with-postgres-and-s3-backups/","title":"Docker Compose for ASP.Net Core with Postgres + S3 backups","text":"<p>2021-07-12</p> <p>In this post, I will cover how I set up the following application structure, run from a single <code>docker-compose</code> file:</p> <ul> <li>ASP.Net Core MVC &amp; Razor Web Application</li> <li>ASP.Net Core Entity Framework Migrations</li> <li>PostgreSQL</li> <li>SMTP Server</li> <li>S3 Backups for PostgreSQL</li> <li>S3 Restores for PostgreSQL</li> </ul> <p>The directory structure, including pertinent files, looks like this.</p> <pre><code>Repository Root Folder\n\u2502   .dockerignore\n\u2502   database.env  \n\u2502   docker-compose.yml\n\u2502\n\u2514\u2500\u2500\u2500ASP.Net Core Project Folder\n\u2502   \u2502   Dockerfile\n\u2502   \u2502   Migrations.Dockerfile\n\u2502   \u2502   Setup.sh\n\u2502   \n\u2514\u2500\u2500\u2500postgres-backup-s3\n\u2502    \u2502   Dockerfile\n\u2502   \n\u2514\u2500\u2500\u2500postgres-restore-s3\n\u2502    \u2502   Dockerfile\n</code></pre> <p>Starting with an almost blank <code>docker-compose.yml</code> file, over the course of this post we'll add each service so that the entire infrastructure can be brought up with one single <code>docker-compose up</code> command.</p> <pre><code>version: '3.4'\n\nservices:\n</code></pre> <p>N.B. One optional thing not covered here is to include <code>restart: always</code> for each service in the <code>docker-compose.yml</code> file to ensure they come back online after the host machine reboots. I wouldn't recommend using this for anything but the core web app, database and mail server.</p>","tags":["deploying","docker"]},{"location":"deploying/docker-compose-for-asp-net-core-with-postgres-and-s3-backups/#aspnet-core-web-application","title":"ASP.Net Core Web Application","text":"<p>For this we want to specify a few key things:</p> <ul> <li>the name to give the container, to make it easier to work with than the docker auto-generated names</li> <li>the internal port to expose on the host machine's external port</li> <li>the Dockerfile to use to build the application</li> <li>the folder to map to make log files accessible from the host machine without having to use the terminal</li> <li>the environment variable to set the app into production mode instead of the default</li> <li>the other containers this one will depend on</li> </ul> <pre><code>version: '3.4'\n\nservices:\n  aspprojectname:\n    container_name: myaspprojectname\n    ports:\n      - \"80:80\"\n    build:\n      context: .\n      dockerfile: MyAspProjectName/Dockerfile\n    volumes:\n      - ./MyAspProjectName/logs:/app/logs\n    environment:\n      - ASPNETCORE_ENVIRONMENT=Production\n    depends_on:\n      - db\n      # - migrations\n</code></pre> <p>You will notice here that I am commenting out the migrations dependency. This is because with the S3 backup and restore images it's not really needed, and it requires more powerful hardware to run than any of the other images and is a large docker image, so worth avoiding if possible, or only using to set up the database initially, then deleting.</p> <p>The Dockerfile itself is pretty standard for ASP.Net Core apps. In this instance, the app is running on dotnet 5. </p> <pre><code>FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS base\nWORKDIR /app\nEXPOSE 80\nEXPOSE 443\n\nFROM mcr.microsoft.com/dotnet/sdk:5.0 AS build\nWORKDIR /src\nCOPY [\"MyAspProjectName/MyAspProjectName.csproj\", \"MyAspProjectName/\"]\nRUN dotnet restore \"MyAspProjectName/MyAspProjectName.csproj\" --disable-parallel\nCOPY . .\nWORKDIR \"/src/MyAspProjectName\"\nRUN dotnet build \"MyAspProjectName.csproj\" -c Release -o /app/build\n\nFROM build AS publish\nRUN dotnet publish \"MyAspProjectName.csproj\" -c Release -o /app/publish\n\nFROM base AS final\nWORKDIR /app\nCOPY --from=publish /app/publish .\nENTRYPOINT [\"dotnet\", \"MyAspProjectName.dll\"]\n</code></pre>","tags":["deploying","docker"]},{"location":"deploying/docker-compose-for-asp-net-core-with-postgres-and-s3-backups/#aspnet-core-entity-framework-migrations","title":"ASP.Net Core Entity Framework Migrations","text":"<p>This one is pretty similar to the main web app in terms of what needs adding to add the service to the <code>docker-compose.yml</code> file.</p> <pre><code>  migrations:\n    container_name: dbmigrations\n    build: \n      context: .\n      dockerfile: MyAspProjectName/Migrations.Dockerfile\n    environment:\n      - ASPNETCORE_ENVIRONMENT=Production\n    depends_on: \n      - db\n</code></pre> <p>The Dockerfile itself is where the magic happens, building the web app, installing the global dotnet-ef tools needed and then running the migrations.</p> <pre><code>FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build\n\nWORKDIR /src\nCOPY [\"MyAspProjectName/MyAspProjectName.csproj\", \"MyAspProjectName/\"]\nCOPY [\"MyAspProjectName/Setup.sh\", \"MyAspProjectName/\"]\n\nENV PATH=\"${PATH}:/root/.dotnet/tools\"\nRUN dotnet tool install --global dotnet-ef\n\nRUN dotnet restore \"MyAspProjectName/MyAspProjectName.csproj\" --disable-parallel\nCOPY . .\nWORKDIR \"/src/MyAspProjectName/.\"\n\nRUN /root/.dotnet/tools/dotnet-ef migrations add InitialMigrations\n\nRUN chmod +x ./Setup.sh\nCMD /bin/bash ./Setup.**sh**\n</code></pre> <p>Personally, I don't use this unless I have to, for the reasons previously stated, but it's worth sharing in case it's useful to anyone else.</p>","tags":["deploying","docker"]},{"location":"deploying/docker-compose-for-asp-net-core-with-postgres-and-s3-backups/#postgresql","title":"PostgreSQL","text":"<p>There's two key parts to add to the <code>docker-compose.yml</code> file for this one. The service itself, with the port to expose and an <code>env</code> file to store sensitive information, and the volume mapped to a directory within the container.</p> <pre><code>services:\n  db:\n    container_name: myappdb\n    image: \"postgres\"\n    ports:\n      - \"5432:5432\"\n    env_file:\n    - database.env # configure postgres\n    volumes:\n    - database-data:/var/lib/postgresql/data/ # persist data even if container shuts down\n\nvolumes:\n    database-data: # named volumes can be managed easier using docker-compose\n</code></pre> <p>That <code>env</code> file is remarkably simple.</p> <pre><code>POSTGRES_USER=postgres\nPOSTGRES_PASSWORD=passwordgoeshere\nPOSTGRES_DB=yourdbname\n</code></pre>","tags":["deploying","docker"]},{"location":"deploying/docker-compose-for-asp-net-core-with-postgres-and-s3-backups/#smtp-server","title":"SMTP Server","text":"<p>The service for the SMTP server is super simple.</p> <pre><code>mail:\n    image: bytemark/smtp\n</code></pre>","tags":["deploying","docker"]},{"location":"deploying/docker-compose-for-asp-net-core-with-postgres-and-s3-backups/#s3-backups-for-postgresql","title":"S3 Backups for PostgreSQL","text":"<p>The <code>docker-compose.yml</code> for this is as below, which sets a daily schedule and connection details for both the Postgres database and S3. Note that the Postgres host uses the internal Docker hostname for the database container.</p> <pre><code>pgbackups3:\n    build:\n      context: .\n      dockerfile: postgres-backup-s3/Dockerfile\n    links:\n      - db\n    environment:\n      SCHEDULE: '@daily'\n      S3_REGION: eu-west-2\n      S3_ACCESS_KEY_ID: keygoeshere\n      S3_SECRET_ACCESS_KEY: secretkeygoeshere\n      S3_BUCKET: yourapp-backups\n      S3_PREFIX: backup\n      POSTGRES_HOST: db\n      POSTGRES_DATABASE: yourdbname\n      POSTGRES_USER: postgres\n      POSTGRES_PASSWORD: passwordgoeshere\n      POSTGRES_EXTRA_OPTS: '--schema=public --blobs'\n</code></pre> <p>I won't go into how this application works here, that'll be in another post.</p>","tags":["deploying","docker"]},{"location":"deploying/docker-compose-for-asp-net-core-with-postgres-and-s3-backups/#s3-restores-for-postgresql","title":"S3 Restores for PostgreSQL","text":"<p>The <code>docker-compose.yml</code> for this is as below, which sets  connection details for both the Postgres database and S3. Note that the Postgres host uses the internal Docker hostname for the database container.</p> <p>This container should be run at setup time then stopped and commented out of the <code>docker-compose.yml</code> file or removed entirely, to prevent it from being accidentally run and restoring unintentionally, overwriting the database (notice the drop public option - this will wipe everything in the database before restoring).</p> <pre><code>pgrestores3:\n    build:\n      context: .\n      dockerfile: postgres-restore-s3/Dockerfile\n    links:\n      - db\n    environment:\n      S3_ACCESS_KEY_ID: keygoeshere\n      S3_SECRET_ACCESS_KEY: secretkeygoeshere\n      S3_BUCKET: yourapp-backups\n      S3_PREFIX: backup\n      POSTGRES_HOST: db\n      POSTGRES_DATABASE: yourdbname\n      POSTGRES_USER: postgres\n      POSTGRES_PASSWORD: passwordgoeshere\n      DROP_PUBLIC: 'yes'\n</code></pre> <p>I won't go into how this application works here, that'll be in another post.</p>","tags":["deploying","docker"]},{"location":"deploying/docker-compose-for-asp-net-core-with-postgres-and-s3-backups/#putting-it-all-together","title":"Putting it all together","text":"<p>The complete <code>docker-compose.yml</code> file, after the initial <code>docker-compose up</code> command has been run, looks like this. Note that migrations is commented out (we don't need it as we have the S3 backup and restore) and that the S3 restore is commented out to prevent accidental restores.</p> <pre><code>version: '3.4'\n\nservices:\n  aspprojectname:\n    container_name: myaspprojectname\n    ports:\n      - \"80:80\"\n    build:\n      context: .\n      dockerfile: MyAspProjectName/Dockerfile\n    volumes:\n      - ./MyAspProjectName/logs:/app/logs\n    environment:\n      - ASPNETCORE_ENVIRONMENT=Production\n    depends_on:\n      - db\n      # - migrations\n  # migrations:\n  # container_name: dbmigrations\n  # build: \n  #     context: .\n  #     dockerfile: MyAspProjectName/Migrations.Dockerfile\n  # environment:\n  #     - ASPNETCORE_ENVIRONMENT=Production\n  # depends_on: \n  #     - db\n  db:\n      container_name: myappdb\n      image: \"postgres\"\n      ports:\n          - \"5432:5432\"\n      env_file:\n      - database.env # configure postgres\n      volumes:\n      - database-data:/var/lib/postgresql/data/ # persist data even if container shuts down\n  mail:\n      image: bytemark/smtp\n  pgbackups3:\n      build:\n          context: .\n          dockerfile: postgres-backup-s3/Dockerfile\n      links:\n          - db\n      environment:\n          SCHEDULE: '@daily'\n          S3_REGION: eu-west-2\n          S3_ACCESS_KEY_ID: keygoeshere\n          S3_SECRET_ACCESS_KEY: secretkeygoeshere\n          S3_BUCKET: yourapp-backups\n          S3_PREFIX: backup\n          POSTGRES_HOST: db\n          POSTGRES_DATABASE: yourdbname\n          POSTGRES_USER: postgres\n          POSTGRES_PASSWORD: passwordgoeshere\n          POSTGRES_EXTRA_OPTS: '--schema=public --blobs'    \n  # pgrestores3:\n  #     build:\n  #         context: .\n  #         dockerfile: postgres-restore-s3/Dockerfile\n  #     links:\n  #         - db\n  #     environment:\n  #         S3_ACCESS_KEY_ID: keygoeshere\n  #         S3_SECRET_ACCESS_KEY: secretkeygoeshere\n  #         S3_BUCKET: yourapp-backups\n  #         S3_PREFIX: backup\n  #         POSTGRES_HOST: db\n  #         POSTGRES_DATABASE: yourdbname\n  #         POSTGRES_USER: postgres\n  #         POSTGRES_PASSWORD: passwordgoeshere\n  #         DROP_PUBLIC: 'yes'\n\nvolumes:\n    database-data: # named volumes can be managed easier using docker-compose\n</code></pre>","tags":["deploying","docker"]},{"location":"deploying/docker-compose-s3-postgres-backup-and-restore/","title":"Docker Compose for S3 Backup and Restore of PostgreSQL","text":"<p>2021-07-12</p> <p>In this post, I will cover how you can set up two Docker containers for backing up and restoring Postgres databases using S3 in AWS.</p> <p>The complete <code>docker-compose.yml</code> file looks like this. Note that it's set to link these containers to a database container also configured within the same compose file - this could be modified to connect to an external database if desired by removing the <code>links</code> and changing the <code>POSTGRES_HOST</code> environment variables.</p> <pre><code>version: '3.4'\n\nservices:\n    pgbackups3:\n        build:\n            context: .\n            dockerfile: postgres-backup-s3/Dockerfile\n        links:\n            - db\n        environment:\n            SCHEDULE: '@daily'\n            S3_REGION: eu-west-2\n            S3_ACCESS_KEY_ID: keygoeshere\n            S3_SECRET_ACCESS_KEY: secretkeygoeshere\n            S3_BUCKET: yourapp-backups\n            S3_PREFIX: backup\n            POSTGRES_HOST: db\n            POSTGRES_DATABASE: yourdbname\n            POSTGRES_USER: postgres\n            POSTGRES_PASSWORD: passwordgoeshere\n            POSTGRES_EXTRA_OPTS: '--schema=public --blobs'    \n    pgrestores3:\n        build:\n            context: .\n            dockerfile: postgres-restore-s3/Dockerfile\n        links:\n            - db\n        environment:\n            S3_ACCESS_KEY_ID: keygoeshere\n            S3_SECRET_ACCESS_KEY: secretkeygoeshere\n            S3_BUCKET: yourapp-backups\n            S3_PREFIX: backup\n            POSTGRES_HOST: db\n            POSTGRES_DATABASE: yourdbname\n            POSTGRES_USER: postgres\n            POSTGRES_PASSWORD: passwordgoeshere\n            DROP_PUBLIC: 'yes'\n</code></pre>","tags":["deploying","docker"]},{"location":"deploying/docker-compose-s3-postgres-backup-and-restore/#backing-up-postgresql-to-s3","title":"Backing up PostgreSQL to S3","text":"<p>The Dockerfile used looks like this:</p> <pre><code>FROM alpine:3.13\n\nRUN apk update \\\n    &amp;&amp; apk add coreutils \\\n    &amp;&amp; apk add postgresql-client \\\n    &amp;&amp; apk add python3 py3-pip &amp;&amp; pip3 install --upgrade pip &amp;&amp; pip3 install awscli \\\n    &amp;&amp; apk add openssl \\\n    &amp;&amp; apk add curl \\\n    &amp;&amp; curl -L --insecure https://github.com/odise/go-cron/releases/download/v0.0.6/go-cron-linux.gz | zcat &gt; /usr/local/bin/go-cron &amp;&amp; chmod u+x /usr/local/bin/go-cron \\\n    &amp;&amp; apk del curl \\\n    &amp;&amp; rm -rf /var/cache/apk/*\n\nENV POSTGRES_DATABASE **None**\nENV POSTGRES_HOST **None**\nENV POSTGRES_PORT 5432\nENV POSTGRES_USER **None**\nENV POSTGRES_PASSWORD **None**\nENV POSTGRES_EXTRA_OPTS ''\nENV S3_ACCESS_KEY_ID **None**\nENV S3_SECRET_ACCESS_KEY **None**\nENV S3_BUCKET **None**\nENV S3_REGION us-west-1\nENV S3_PATH 'backup'\nENV S3_ENDPOINT **None**\nENV S3_S3V4 no\nENV SCHEDULE **None**\n\nCOPY [\"postgres-backup-s3/run.sh\", \"run.sh\"]\nCOPY [\"postgres-backup-s3/backup.sh\", \"backup.sh\"]\n\nCMD [\"sh\", \"run.sh\"]\n</code></pre> <p>This is a combination of parts from here and the original here with some further customisation by me.</p> <p>It essentially sets up a Linux environment to be able to connect to both S3 and Postgres, and to be able to run two script files.</p>","tags":["deploying","docker"]},{"location":"deploying/docker-compose-s3-postgres-backup-and-restore/#runsh","title":"<code>run.sh</code>","text":"<p>This one handles a bit of aws configuration for S3 connections, as well as setting up the behaviour for scheduling or just immediately running the container. I like to set it to run daily so I can set it and forget it.</p> <pre><code>#! /bin/sh\n\nset -e\n\nif [ \"${S3_S3V4}\" = \"yes\" ]; then\n    aws configure set default.s3.signature_version s3v4\nfi\n\nif [ \"${SCHEDULE}\" = \"**None**\" ]; then\n  sh backup.sh\nelse\n  exec go-cron \"$SCHEDULE\" /bin/sh backup.sh\nfi\n</code></pre>","tags":["deploying","docker"]},{"location":"deploying/docker-compose-s3-postgres-backup-and-restore/#backupsh","title":"<code>backup.sh</code>","text":"<p>This one handles checking all the requisite connection details are present, then makes a compressed dump file of the PostgreSQL database, and uploads it to S3.</p> <pre><code>#! /bin/sh\n\nset -e\nset -o pipefail\n\nif [ \"${S3_ACCESS_KEY_ID}\" = \"**None**\" ]; then\n  echo \"You need to set the S3_ACCESS_KEY_ID environment variable.\"\n  exit 1\nfi\n\nif [ \"${S3_SECRET_ACCESS_KEY}\" = \"**None**\" ]; then\n  echo \"You need to set the S3_SECRET_ACCESS_KEY environment variable.\"\n  exit 1\nfi\n\nif [ \"${S3_BUCKET}\" = \"**None**\" ]; then\n  echo \"You need to set the S3_BUCKET environment variable.\"\n  exit 1\nfi\n\nif [ \"${POSTGRES_DATABASE}\" = \"**None**\" ]; then\n  echo \"You need to set the POSTGRES_DATABASE environment variable.\"\n  exit 1\nfi\n\nif [ \"${POSTGRES_HOST}\" = \"**None**\" ]; then\n  if [ -n \"${POSTGRES_PORT_5432_TCP_ADDR}\" ]; then\n    POSTGRES_HOST=$POSTGRES_PORT_5432_TCP_ADDR\n    POSTGRES_PORT=$POSTGRES_PORT_5432_TCP_PORT\n  else\n    echo \"You need to set the POSTGRES_HOST environment variable.\"\n    exit 1\n  fi\nfi\n\nif [ \"${POSTGRES_USER}\" = \"**None**\" ]; then\n  echo \"You need to set the POSTGRES_USER environment variable.\"\n  exit 1\nfi\n\nif [ \"${POSTGRES_PASSWORD}\" = \"**None**\" ]; then\n  echo \"You need to set the POSTGRES_PASSWORD environment variable or link to a container named POSTGRES.\"\n  exit 1\nfi\n\nif [ \"${S3_ENDPOINT}\" == \"**None**\" ]; then\n  AWS_ARGS=\"\"\nelse\n  AWS_ARGS=\"--endpoint-url ${S3_ENDPOINT}\"\nfi\n\n# env vars needed for aws tools\nexport AWS_ACCESS_KEY_ID=$S3_ACCESS_KEY_ID\nexport AWS_SECRET_ACCESS_KEY=$S3_SECRET_ACCESS_KEY\nexport AWS_DEFAULT_REGION=$S3_REGION\n\nexport PGPASSWORD=$POSTGRES_PASSWORD\nPOSTGRES_HOST_OPTS=\"-h $POSTGRES_HOST -p $POSTGRES_PORT -U $POSTGRES_USER $POSTGRES_EXTRA_OPTS\"\n\necho \"Creating dump of ${POSTGRES_DATABASE} database from ${POSTGRES_HOST}...\"\n\npg_dump $POSTGRES_HOST_OPTS $POSTGRES_DATABASE | gzip &gt; dump.sql.gz\n\necho \"Uploading dump to $S3_BUCKET\"\n\ncat dump.sql.gz | aws $AWS_ARGS s3 cp - s3://$S3_BUCKET/$S3_PREFIX/${POSTGRES_DATABASE}_$(date +\"%Y-%m-%dT%H:%M:%SZ\").sql.gz || exit 2\n\necho \"SQL backup uploaded successfully\"\n</code></pre>","tags":["deploying","docker"]},{"location":"deploying/docker-compose-s3-postgres-backup-and-restore/#restoring-postgresql-from-s3","title":"Restoring PostgreSQL from S3","text":"<p>The Dockerfile used looks like this:</p> <pre><code>FROM alpine:3.13\n\nRUN apk update \\\n    &amp;&amp; apk add coreutils \\\n    &amp;&amp; apk add postgresql-client \\\n    &amp;&amp; apk add python3 py3-pip &amp;&amp; pip3 install --upgrade pip &amp;&amp; pip3 install awscli \\\n    &amp;&amp; apk add openssl \\\n    &amp;&amp; apk add curl \\\n    &amp;&amp; curl -L --insecure https://github.com/odise/go-cron/releases/download/v0.0.6/go-cron-linux.gz | zcat &gt; /usr/local/bin/go-cron &amp;&amp; chmod u+x /usr/local/bin/go-cron \\\n    &amp;&amp; apk del curl \\\n    &amp;&amp; rm -rf /var/cache/apk/*\n\nENV POSTGRES_DATABASE **None**\nENV POSTGRES_HOST **None**\nENV POSTGRES_PORT 5432\nENV POSTGRES_USER **None**\nENV POSTGRES_PASSWORD **None**\nENV S3_ACCESS_KEY_ID **None**\nENV S3_SECRET_ACCESS_KEY **None**\nENV S3_BUCKET **None**\nENV S3_REGION us-west-1\nENV S3_PATH 'backup'\nENV DROP_PUBLIC 'no'\n\nCOPY [\"postgres-restore-s3/restore.sh\", \"restore.sh\"]\n\nCMD [\"sh\", \"restore.sh\"]\n</code></pre> <p>This is once again a combination of parts from here and the original here with some further customisation by me.</p> <p>It essentially sets up a Linux environment to be able to connect to both S3 and Postgres, and to be able to run one script file.</p>","tags":["deploying","docker"]},{"location":"deploying/docker-compose-s3-postgres-backup-and-restore/#restoresh","title":"<code>restore.sh</code>","text":"<p>This one handles checking all the requisite connection details are present, then retrieves the most recent compressed dump file of the PostgreSQL database from S3, drops everything in the public space in PostgreSQL, and restores the backup. Crucially it then deletes the downloaded backup dump file to enable running the same container again for future restores without issue.</p> <pre><code>#! /bin/sh\n\nset -e\nset -o pipefail\n\nif [ \"${S3_ACCESS_KEY_ID}\" = \"**None**\" ]; then\n  echo \"You need to set the S3_ACCESS_KEY_ID environment variable.\"\n  exit 1\nfi\n\nif [ \"${S3_SECRET_ACCESS_KEY}\" = \"**None**\" ]; then\n  echo \"You need to set the S3_SECRET_ACCESS_KEY environment variable.\"\n  exit 1\nfi\n\nif [ \"${S3_BUCKET}\" = \"**None**\" ]; then\n  echo \"You need to set the S3_BUCKET environment variable.\"\n  exit 1\nfi\n\nif [ \"${POSTGRES_DATABASE}\" = \"**None**\" ]; then\n  echo \"You need to set the POSTGRES_DATABASE environment variable.\"\n  exit 1\nfi\n\nif [ \"${POSTGRES_HOST}\" = \"**None**\" ]; then\n  if [ -n \"${POSTGRES_PORT_5432_TCP_ADDR}\" ]; then\n    POSTGRES_HOST=$POSTGRES_PORT_5432_TCP_ADDR\n    POSTGRES_PORT=$POSTGRES_PORT_5432_TCP_PORT\n  else\n    echo \"You need to set the POSTGRES_HOST environment variable.\"\n    exit 1\n  fi\nfi\n\nif [ \"${POSTGRES_USER}\" = \"**None**\" ]; then\n  echo \"You need to set the POSTGRES_USER environment variable.\"\n  exit 1\nfi\n\nif [ \"${POSTGRES_PASSWORD}\" = \"**None**\" ]; then\n  echo \"You need to set the POSTGRES_PASSWORD environment variable or link to a container named POSTGRES.\"\n  exit 1\nfi\n\n# env vars needed for aws tools\nexport AWS_ACCESS_KEY_ID=$S3_ACCESS_KEY_ID\nexport AWS_SECRET_ACCESS_KEY=$S3_SECRET_ACCESS_KEY\nexport AWS_DEFAULT_REGION=$S3_REGION\n\nexport PGPASSWORD=$POSTGRES_PASSWORD\nPOSTGRES_HOST_OPTS=\"-h $POSTGRES_HOST -p $POSTGRES_PORT -U $POSTGRES_USER\"\n\necho \"Finding latest backup\"\n\nLATEST_BACKUP=$(aws s3 ls s3://$S3_BUCKET/$S3_PREFIX/ | sort | tail -n 1 | awk '{ print $4 }')\n\necho \"Fetching ${LATEST_BACKUP} from S3\"\n\naws s3 cp s3://$S3_BUCKET/$S3_PREFIX/${LATEST_BACKUP} dump.sql.gz\ngzip -d dump.sql.gz\n\nif [ \"${DROP_PUBLIC}\" == \"yes\" ]; then\n    echo \"Recreating the public schema\"\n    psql $POSTGRES_HOST_OPTS -d $POSTGRES_DATABASE -c \"drop schema public cascade; create schema public;\"\nfi\n\necho \"Restoring ${LATEST_BACKUP}\"\n\npsql $POSTGRES_HOST_OPTS -d $POSTGRES_DATABASE &lt; dump.sql\n\necho \"Restore complete\"\n\nrm -f ./dump.sql\n\necho \"Deleted dump files\"\n</code></pre>","tags":["deploying","docker"]},{"location":"deploying/elasticsearch-and-kibana/","title":"Docker Compose for Elasticsearch and Kibana 7.9","text":"<p>2021-07-13</p> <p>In this post, I will cover how you can set up Elasticsearch and Kibana with a single <code>docker-compose.yml</code> file.</p>","tags":["deploying","docker"]},{"location":"deploying/elasticsearch-and-kibana/#set-up-and-install-elk-stack-on-ubuntu-server","title":"Set up and install ELK stack on Ubuntu server","text":"<p>You must run <code>sudo sysctl -w vm.max_map_count=262144</code> to get Elasticsearch to work. To make this permanent, run <code>sudo nano /etc/sysctl.conf</code> and add <code>vm.max_map_count=262144</code> to the end of the file on a new line, then save and exit.</p> <p>Go to the directory with the files in. Make sure you've used SFTP to put the <code>docker-compose.yml</code> file in there first. For example you may need to use this command <code>cd /home/administrator/docker/elk-stack</code></p> <p>Install it all with <code>docker-compose up -d</code> and you're finished! It really is that simple.</p> <p>The compose file is as below.</p> <pre><code>version: '2.2'\n\nservices:\n\n  elasticsearch:\n    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.3\n    container_name: elasticsearch\n    environment:\n      - node.name=elasticsearch\n      - discovery.seed_hosts=elasticsearch\n      - cluster.initial_master_nodes=elasticsearch\n      - cluster.name=docker-cluster\n      - bootstrap.memory_lock=true\n      - \"ES_JAVA_OPTS=-Xms2g -Xmx2g\"\n    ulimits:\n      memlock:\n        soft: -1\n        hard: -1\n      nofile:\n        soft: 65536\n        hard: 65536\n    volumes:\n      - esdata1:/usr/share/elasticsearch/data\n    ports:\n      - 9200:9200\n\n  kibana:\n    image: docker.elastic.co/kibana/kibana:7.9.3\n    container_name: kibana\n    environment:\n      ELASTICSEARCH_URL: \"http://elasticsearch:9200\"\n      ELASTICSEARCH_SHARDTIMEOUT: 300000\n    ports:\n      - 5601:5601\n    depends_on:\n      - elasticsearch\n\nvolumes:\n  esdata1:\n    driver: local\n</code></pre>","tags":["deploying","docker"]},{"location":"dev-team/git/","title":"The way I recommend using git in a dev team","text":"<p>2022-03-11</p> <p>Looking back at my old git commits reveals an entertaining, if slightly embarassing mess.</p> <pre><code>$ git log --oneline -10 --author jcreek --before \"Wed Nov 5 2014\"\n8b6d283 Updated instructions\n6b271ba Undid that last change\n8c4f48a MySQL -&gt; MySQLi changes begun\nc07ae83 Fixed that bug\ne5ab087 Some small cosmetic changes and failed fix of bug\n4467af9 Updated to-do list\n3578065 Cleanup and to-do list\nddefa3a Added admin tools, provided link to admin page\n8a2f8b6 Further admin tools\n63cc797 Unimportant changes\n</code></pre> <p>The contrast with more recent commits is staggering.</p> <pre><code>$ git log --oneline -13 --author jcreek --before \"Wed Sep 1 2021\"\nc05c2c5 build(CAN-186): Add missing package information to csproj\n06aca19 docs(CAN-190): Add documentation to StringHelper\n4600ebf (tag: 1.0.1) feat(CAN-182): Add filename validity checks to SftpRepository\nd80cc82 (tag: 1.0.0) chore(CAN-173): Update readme and licence for NuGet\n615994a refactor(CAN-215): Rename to Creek.FileRepository\n0fe9924 feat(CAN-171): Add all repositories to factory\nb22d2f7 feat(CAN-170): Add base implementations of all repositories\n952a9a9 refactor(CAN-211): Rename FileName to Filename in interface\n62665fe test(CAN-169): Add SftpRepositoryShould tests\nebccd4a feat(CAN-168): Add GenerateStreamFromString helper method for testing\n82fd258 fix(CAN-135): Initialise content with a new stream so it can be written to\n4232dd9 fix(CAN-162): Add missing throw to ensure exceptions get passed up the stack\n26a953d feat(CAN-167): Add initialiser for SftpRepository to use an IConfiguration\n</code></pre> <p>My recent commits follow a clear convention, are concise and clear at a glance. Without having to do any code comparison you can quickly assess what was done in any given commit. A commit message shows whether a developer is a good collaborator, with good ones reducing the need to re-establish the context of a commit as much as possible.</p> <p>I never used to care about my commit messages because it was only me reading them, and because I rarely, if ever, used tools like <code>git log</code>, <code>git blame</code>, <code>revert</code> or <code>rebase</code>. Nowadays I use git blame all day, every day, thanks to VSCode git extensions putting the blame on every line of code, and it's incredibly useful as part of a team. If I'm looking at code I always know who worked on it, when they worked on it, and the context of why they did what they did, which makes the codebases infinitely more maintainable.</p> <p>There is a single git commit that I think should be required reading for all developers. It is from a developer by the name of Dan Carley working on GOV.UK, and it has the rather unassuming name of \"Convert template to US-ASCII to fix error\".</p> <p></p> <p>I found it through this blog and I'd highly recommend reading through it, but essentially the thing I like most about this commit is it explains why the change was made, not just what the change was.</p> <p>As with most developers, I have my own strong opinions founded in habit for what dev teams should do in terms of Git usage. In the rest of this doc I will outline that - let me know if you have any suggestions for improvements, or if it helped you.</p>","tags":["dev teams","git"]},{"location":"dev-team/git/#branching-pull-requests-merging","title":"Branching, Pull Requests &amp; Merging","text":"<p>For every ticket, there should be a separate git branch. This code will not be merged into the main development branch until it has been through a pull request and been reviewed.</p> <p>If it's a big feature ticket, a feature branch should be created for that ticket, and it should be split into sub-tasks, each of which have their own branch and get merged into the feature branch after PRs are approved. Once all the sub-tasks have been PRed and their branches have been merged into the feature branch, the feature branch can be PRed and merge into the development branch. If you find you're needing to do all your changes in one branch, do so, then split it up by making new branches and cherry-picking or re-committing once you have your finished code. This will make it significantly easier to review, and to review well.</p> <p>Unless you're working on a sub-task branch, the merge strategy for PRs should involve squashing. Merge commits are fine, and they leave all individual commits available for comparison, reverting and cherry-picking, but they also include all of the ongoing dev work and changes based on PR feedback, which can clutter up a project. </p> <p>Not using squashed commits makes having good quality commit messages even more important. Committing often and messily is a bad habit (unless you're squashing). Instead break down tasks into suitably small sub-tasks, so that they can be worked on in isolation without taking long enough to warrant committing unfinished code.</p> <p>A key part of the PR process is reviewing code, not only for bugs, but for quality. The git commit messages are a critical part of this. No developer is perfect, so helping one another in PRs is key to having an excellent git history. If a PR comes in with bad commit messages, it should be rejected and redone with good commit messages.</p>","tags":["dev teams","git"]},{"location":"dev-team/git/#git-messages","title":"Git Messages","text":"<p>Git commit messges subjects should always follow this pattern:</p> <p><code>&lt;type&gt;(scope): &lt;description&gt;</code></p> <p>Types are:</p> <ul> <li><code>build</code>: build-related changes</li> <li><code>ci</code>: continuous integration-related changes</li> <li><code>chore</code>: updating grunt tasks etc; no production code change</li> <li><code>docs</code>: changes to the documentation</li> <li><code>feat</code>: new feature for the user, not a new feature for build script</li> <li><code>fix</code>: bug fix for the user, not a fix to a build script</li> <li><code>perf</code>: a code change that improves performance</li> <li><code>refactor</code>: refactoring production code, eg. renaming a variable</li> <li><code>revert</code>: reverting things</li> <li><code>style</code>: formatting, missing semi colons, etc; no production code change</li> <li><code>test</code>: adding missing tests, refactoring tests; no production code change</li> </ul> <p>The scope, generally considered optional, really should be mandatory. Typically this would be a ticket number, enabling devs both now and in the future to easily find commits associated with tickets and tickets associated with commits. Full details of what was being worked on can often be found in a ticket if they are not in the commit message, and in the event a feature's code needs to be examined or reverted. If you use a ticket management system like Jira having the ticket number in your commit message enables the management system to automatically detect it and it can be embedded directly in a card that you are currently working on, and even associated with Pull Requests and Epics.</p> <p>The description is also incredibly important. A properly formed description should always be able to complete the following sentence:</p> <p><code>If applied, this commit will &lt;description&gt;</code></p> <p>It contains a succinct description of the change:</p> <ul> <li>Use the imperative, present tense: \"change\" not \"changed\" nor \"changes\"</li> <li>No full stop/period at the end</li> </ul> <p>The body of a commit message should also use the imperative, present tense: \"change\" not \"changed\" nor \"changes\". The body should include the motivation for the change and contrast this with previous behavior.</p> <p>The 7 rules of a great commit message:</p> <ul> <li>Separate subject from the body with a blank line</li> <li>Limit the subject line to 50 characters</li> <li>Summary in the present tense. Not capitalized.</li> <li>Do not end the subject line with a period</li> <li>Use the imperative mood in the subject line</li> <li>Wrap the body at 72 characters</li> <li>Use the body to explain what and why vs. how</li> </ul> <p>By incorporating the above convention, you can quickly determine what changes have been made and what the commit message for the changes would look like. With that format, you as a developer can immediately know (or at least guess), what\u2019s changed in the specific commit. If there\u2019s an issue with a new merge to the master branch, you can also quickly scan the git history to figure out what changes possibly can cause the problem without the need to look at the differences.</p> <p>Some very good reasons to use this convention are:</p> <ul> <li>Automatically generating CHANGELOGs.</li> <li>Automatically determining a semantic version bump (based on the types of commits landed).</li> <li>Communicating the nature of changes to teammates, the public, and other stakeholders.</li> <li>Triggering build and publish processes.</li> <li>Making it easier for people to contribute to your projects by allowing them to explore a more structured commit history.</li> </ul>","tags":["dev teams","git"]},{"location":"dev-team/git/#code-reviews-in-prs","title":"Code Reviews (in PRs)","text":"<p>When reviewing code, here are the key things to be paying attention to:</p> <ul> <li>Refactoring to remove old/inefficient/not working code: Ensure that the code changes include removal of redundant or outdated code snippets. Refactoring should improve the code's efficiency, readability, and maintainability without altering its functionality.</li> <li>Code style: Review for consistency with the project's coding standards. This includes adherence to naming conventions, file organisation, indentation, and comment quality. Consistent code style aids in understanding and maintaining the code base.</li> <li>Code smells: Look for indicators of deeper problems in the code, such as duplicate code, long methods, large classes, and improper use of object-oriented principles. These \"smells\" can indicate areas that may need refactoring.</li> <li>Best practices and design patterns: Check if the code follows established best practices for the language and framework being used. Verify the appropriate use of design patterns which can help in solving commonly occurring problems in a more efficient way.</li> <li>Performance considerations: Analyze the changes for potential performance issues, such as memory leaks, inefficient algorithms, and unnecessary database queries. The goal is to ensure that the new code does not introduce performance regressions.</li> <li>Security aspects: Evaluate the code for security vulnerabilities. This includes checking for SQL injection, proper handling of user data, adherence to authentication and authorization mechanisms, and ensuring data privacy.</li> <li>Testing: Ensure that the new code includes relevant unit tests or integration tests that cover the new functionality and any changes to existing functionality. Good test coverage is essential for maintaining code quality over time.</li> <li>Documentation: Verify that any new methods, classes, or significant logic changes are accompanied by appropriate comments or documentation. This is crucial for future maintainability, making it easier for others to understand the purpose and functionality of the code.</li> <li>Overall code quality and readability: Assess the overall quality of the code. The code should be readable, well-organized, and logically structured. It should be easy for someone else on the team to understand and maintain.</li> </ul> <p>Remember, as a code reviewer, you are responsible for ensuring the quality and integrity of the code base. Your review can help prevent issues and improve the overall quality of the software. Therefore, it's important to be thorough and provide constructive feedback during the review process. A reviewer is equally responsible for any bugs or issues found in after a PR has been approved and merged. If a PR is too big or complex to reasonably review well, reject it and flag this to the developer who submitted it. Where possible you should test the branch, but at the very least any UI changes should be demonstrated via a screenshot or gif in the description of the PR. </p>","tags":["dev teams","git"]},{"location":"dev-team/git/#automating-git-commit-message-formatting","title":"Automating Git Commit Message Formatting","text":"<p>You can tell git to set up your commit messages with a set structure. This is done by running the command <code>git config --global commit.template ~/.gitmessage</code> or by manually setting <code>commit.template</code> in <code>~/.gitconfig</code> using your text editor of choice:</p> <pre><code>[commit]\n  template = ~/.gitmessage\n</code></pre> <p>Next, create <code>~/.gitmessage</code> with your new default template.</p> <p>For your convenience, the following git commit template can be used:</p> <pre><code># A properly formed Git commit subject line should always be able to complete\n# the following sentence:\n#     * If applied, this commit will &lt;description&gt;\n#\n# ** Example:\n# &lt;type&gt;(scope): &lt;description&gt;\n#\n# [optional body]\n#\n# [optional footer]\n\n# ** Type\n# Must be one of the following:\n# * build: build-related changes\n# * ci: continuous integration-related changes\n# * chore: updating grunt tasks etc; no production code change\n# * docs: changes to the documentation\n# * feat: new feature for the user, not a new feature for build script\n# * fix: bug fix for the user, not a fix to a build script\n# * perf: a code change that improves performance\n# * refactor: refactoring production code, eg. renaming a variable\n# * revert: reverting things\n# * style: formatting, missing semi colons, etc; no production code change\n# * test: adding missing tests, refactoring tests; no production code change\n\n# ** Subject\n# The subject contains a succint description of the change:\n# * Use the imperative, present tense: \"change\" not \"changed\" nor \"changes\"\n# * No full stop/period at the end\n\n# ** Scope\n# A scope should be provided to a commit\u2019s type if possible, to provide additional contextual information\n# and is contained within parenthesis, e.g. feat(JIRA-1234): Add NewGitDocumentation endpoint\n\n# ** Body\n# Just as in the subject, use the imperative, present tense: \"change\" not \"changed\" nor \"changes\".\n# The body should include the motivation for the change and contrast this with previous behavior.\n\n# ** Rules\n# The 7 rules of a great commit message\n#    1. Separate subject from body with a blank line\n#    2. Limit the subject line to 50 characters\n#    3. Summary in present tense. Not capitalized\n#    4. Do not end the subject line with a period\n#    5. Use the imperative mood in the subject line\n#    6. Wrap the body at 72 characters\n#    7. Use the body to explain what and why vs. how\n</code></pre>","tags":["dev teams","git"]},{"location":"dev-team/readme-files/","title":"How to write an excellent readme file","text":"<p>2022-03-16</p> <p>All git projects need a readme file, to explain what it is, why it exists, how it works and how to use it. A good starting point with guidance can be found below.</p> <pre><code># Project title\n\nA little info about the application and an overview that explains **what** the application is for.\n\n## Motivation\n\nA short description of the motivation behind the creation and maintenance of the application. This should explain **why** the application exists.\n\n## How the dev/staging/prod versions are built\n\nThis should be a brief explanation of how the application gets to where it's meant to be used. Typically this will involve linking to build pipelines in Jenkins/Azure/etc.\n\n## Code style\n\nUsually this would just link to the documentation for your established code style. In the event that the application uses additional or different styles or linting tooling this can be detailed here.\n\n## Screenshots\n\nInclude some logos and demos screenshots to briefly give a rough idea of the application.\n\n## Tech/framework used\n\nHopefully this will be identical across most of your organisation's applications, but the value of this section really becomes clear when an application does differ, and developers can see this at a glance here, without having to dig into the code or run the project locally to work it out.\n\n## Features\n\nWhat are the key features of this application? A few bullet points are ideal here.\n\n## Getting it running locally for development\n\nProvide a step by step series of examples and explanations about how to get a development environment running. If it is dependent on other local environmental factors, for example a specific database, this should also be covered here. This should be a single source of truth for getting this application running, without being dependent on any external sources of knowledge, even if that means there is some duplication across projects within this readme file.\n\n## API Reference\n\nIf it's appropriate, a link should be included here to the API documentation, for example OpenAPI docs or a Swagger file.\n\n## Tests\n\nDescribe and show how to run the tests with basic examples.\n\n## How to use as a user\n\nA brief guide on how to use the application AS A USER should be included here. A developer picking up the application should be able to find their way around it quickly as a user based on the guidance included here, to be able to sufficiently understand how most parts of the application are used.\n\n## Anything else that seems useful\n\nIf there is anything else that seems useful to include in the readme file, this should be included here. Documentation of specific parts of the code SHOULD NOT live in this readme file, instead they should be in external documentation such as Confluence or in the code itself, whichever is more appropriate. References to such documentation could be included in this readme file for significant examples, but largely is not necessary.\n</code></pre>","tags":["dev teams","git","readme"]},{"location":"home-lab/opnsense-nintendo-switch/","title":"Opnsense Nintendo Switch Settings","text":"<p>2025-04-19</p> <p>When trying to play some (mostly older) Nintendo Switch games with online multiplayer there are specific settings needed in Opnsense to ensure that the games can connect to other players and the Nintendo servers. This guide documents how I set up Opnsense to work with the Nintendo Switch.</p> <p>To see if this applies to you, you can do an Internet Test on the Nintendo Switch. If you get a NAT Type of A or B, then you are good to go. If you get a NAT Type of C or D, then you will need to follow the steps below.</p> <p>Alternatively, when trying to play the game Mario Kart 8 Deluxe, you may get the below error message:</p> <pre><code>Error Code: 2618-0516\n\nUnable to connect to the other console(s).\nNAT traversal process has failed.\nPlease try again later.\n\nIf the problem persists, your network conditions may not be suited to stable online play.\n</code></pre> <p>By following these steps I changed from a NAT Type of D to a NAT Type of B.</p> <p>This guide is based on:</p> <ul> <li>a post on the Opnsense forums and enabled me to play Mario Kart 8 Deluxe online with friends.</li> <li>a thread on Reddit</li> <li>another thread on Reddit</li> <li>a post on Nintendo's support site</li> </ul>","tags":["opnsense","nintendo switch","nat"]},{"location":"home-lab/opnsense-nintendo-switch/#1-assign-a-static-ip-address","title":"1. Assign a Static IP Address","text":"<p>Go to https://10.0.0.1/services_dhcp.php?if=lan where <code>10.0.0.1</code> is the IP address of your Opnsense router. Click on the <code>+</code> button to add a new DHCP static mapping.</p> <p>It's important to note that your Nintendo Switch has two network interfaces: one for the Wi-Fi and one for the Ethernet. You will need to set up a static IP address for whichever interface you're intending to use for online multiplayer. You can find the mac address of the interface by going to <code>System Settings &gt; Internet Settings &gt; Internet Settings &gt; Connection Test</code> on the Nintendo Switch.</p>","tags":["opnsense","nintendo switch","nat"]},{"location":"home-lab/opnsense-nintendo-switch/#2set-up-a-nat-rule","title":"2.Set up a NAT rule","text":"<p>Go to https://10.0.0.1/firewall_nat_out.php where <code>10.0.0.1</code> is the IP address of your Opnsense router. I selected the<code>Hybrid outbound NAT rule generation</code> mode. Add a new manual NAT rule by clicking the <code>+</code> button.</p> <ul> <li>Disabled: Unchecked</li> <li>Do Not NAT: Unchecked</li> <li>Interface: WAN</li> <li>TP/IP Version: IPv4</li> <li>Protocol: Any</li> <li>Source Invert: Unchecked</li> <li>Source Address: Nintendo switch (an alias to the switch's IP)</li> <li>Source Port: Any</li> <li>Destination Invert: Unchecked</li> <li>Destination Address:Any</li> <li>Destination Port: Any</li> <li>Translation/Target: WAN address</li> <li>Log: Unchecked</li> <li>Translation/port: Blank</li> <li>Static port: Checked</li> <li>Pool Options: Default</li> </ul> <p>The remaining fields are all blank (Set Local Tag, Match Local Tag, No XMLRPC Sync and Desription).</p>","tags":["opnsense","nintendo switch","nat"]},{"location":"home-lab/opnsense-nintendo-switch/#3-port-forward","title":"3. Port Forward","text":"<p>Go to https://10.0.0.1/firewall_nat.php where <code>10.0.0.1</code> is the IP address of your Opnsense router. Click on the <code>Port Forward</code> tab and then click the <code>+</code> button to add a new port forward rule.</p> <ul> <li>Disabled: Unchecked</li> <li>No RDR(NOT): Unchecked</li> <li>Interface: WAN</li> <li>TCP/IP Version: IPv4</li> <li>Protocol: UDP</li> <li>Source:</li> <li>Destination/Invert: Unchecked</li> <li>Destination: Single host or network</li> <li>Destination Address: 10.0.0.104 (the IP address of the Nintendo Switch)</li> <li>Destination Port Range: 45000-65535</li> <li>Redirect target IP: 10.0.0.104 (the IP address of the Nintendo Switch)</li> <li>Redirect target port: 45000</li> <li>Pool Options: Default</li> <li>NAT Reflection: Use system default</li> <li>Filter rule association: Add associated filter rule</li> <li>Description: Nintendo Switch Online Multiplayer</li> <li>Log: Unchecked</li> <li>No XMLRPC Sync: Unchecked</li> </ul>","tags":["opnsense","nintendo switch","nat"]},{"location":"home-lab/setup-guide/","title":"How I set up my home lab with Cloudflare","text":"<p>2025-03-02</p>","tags":["home lab","cloudflare","self-hosting"]},{"location":"home-lab/setup-guide/#overview","title":"Overview","text":"<p>This guide walks through setting up my home lab using the domain <code>creeknet.uk</code>, managing DNS, security, remote access, reverse proxy, and VPN to ensure secure and private access to my services. This exists both as a record for myself and a guide for others looking to set up a similar home lab.</p>","tags":["home lab","cloudflare","self-hosting"]},{"location":"home-lab/setup-guide/#1-cloudflare-setup","title":"1. Cloudflare Setup","text":"","tags":["home lab","cloudflare","self-hosting"]},{"location":"home-lab/setup-guide/#11-register-domain-set-up-dns","title":"1.1 Register Domain &amp; Set Up DNS","text":"<ol> <li>Log into Cloudflare and add <code>creeknet.uk</code>. The <code>.uk</code> domains are amongst the cheapest, at around $5 per year.</li> <li>Go to DNS &gt; Add A Records for all publicly accessible services:    These records allow external access to specific services.     \u26a0\ufe0f Cloudflare Proxy (Orange Cloud \u2601\ufe0f) should be OFF (DNS Only) for services that require non-HTTP ports (e.g., Plex, VPN).</li> </ol> Subdomain Record Type Points To Proxy Status Purpose <code>ai.creeknet.uk</code> A My Public IP DNS Only \u2601\ufe0f (Gray) AI Server <code>home.creeknet.uk</code> A My Public IP DNS Only \u2601\ufe0f (Gray) Home Assistant <code>nas.creeknet.uk</code> A My Public IP DNS Only \u2601\ufe0f (Gray) NAS Access <code>plex.creeknet.uk</code> A My Public IP DNS Only \u2601\ufe0f (Gray) Remote Plex Access <code>rustdesk.creeknet.uk</code> A My Public IP DNS Only \u2601\ufe0f (Gray) Remote Desktop (RustDesk) <code>vpn.creeknet.uk</code> A My Public IP DNS Only \u2601\ufe0f (Gray) WireGuard VPN","tags":["home lab","cloudflare","self-hosting"]},{"location":"home-lab/setup-guide/#12-enable-cloudflare-zero-trust-for-web-services","title":"1.2 Enable Cloudflare Zero Trust for Web Services","text":"<p>I did consider that for any web-facing applications that do not have their own authentication I could use Cloudflare Zero Trust to enforce authentication. To do so I would:</p> <ol> <li>Go to Zero Trust \u2192 Access \u2192 Applications.</li> <li>Add an application.</li> <li>Configure Access Policy (Require login via OTP).</li> </ol>","tags":["home lab","cloudflare","self-hosting"]},{"location":"home-lab/setup-guide/#2-setting-up-local-dns-split-dns","title":"2. Setting Up Local DNS (Split DNS)","text":"<p>To ensure local services work even if the internet is down, configure Pi-hole (<code>10.0.0.6</code>):</p> <ul> <li> <p>Add Local DNS Records:</p> </li> <li> <p>Web-services should direct to Nginx:</p> <ul> <li>ai.creeknet.uk \u2192 <code>10.0.0.23</code></li> <li>home.creeknet.uk \u2192 <code>10.0.0.23</code></li> <li>nas.creeknet.uk \u2192 <code>10.0.0.23</code></li> <li>plex.creeknet.uk \u2192 <code>10.0.0.23</code></li> </ul> </li> <li> <p>Non-web services should direct to the actual servers:</p> <ul> <li>rustdesk.creeknet.uk \u2192 <code>10.0.0.23</code> (dockerised on the same server as nginx)</li> <li>vpn.creeknet.uk \u2192 <code>10.0.0.5</code> (redundant if the internet is down but included for completeness)</li> </ul> </li> <li> <p>Router DNS Settings:</p> </li> <li> <p>Primary DNS: <code>10.0.0.6</code> (Pi-hole)</p> </li> <li>Secondary DNS: <code>1.1.1.1</code> (Cloudflare)</li> </ul>","tags":["home lab","cloudflare","self-hosting"]},{"location":"home-lab/setup-guide/#3-reverse-proxy-with-nginx-proxy-manager","title":"3. Reverse Proxy with NGINX Proxy Manager","text":"","tags":["home lab","cloudflare","self-hosting"]},{"location":"home-lab/setup-guide/#31-deploy-nginx-proxy-manager-docker-on-proxrouter","title":"3.1 Deploy NGINX Proxy Manager (Docker) on ProxRouter","text":"<pre><code>version: \"3\"\nservices:\n  npm:\n    image: \"jc21/nginx-proxy-manager:latest\"\n    container_name: npm\n    restart: unless-stopped\n    ports:\n      - \"80:80\"\n      - \"443:443\"\n      - \"81:81\" # Admin UI\n    volumes:\n      - ./data:/data\n      - ./letsencrypt:/etc/letsencrypt\n</code></pre>","tags":["home lab","cloudflare","self-hosting"]},{"location":"home-lab/setup-guide/#32-configure-proxy-hosts","title":"3.2 Configure Proxy Hosts","text":"<ul> <li> <p>ai.creeknet.uk \u2192 <code>http://10.0.0.20:8080</code> (Open Web UI)</p> </li> <li> <p>Provide SSL certificate through Let\u2019s Encrypt</p> </li> <li>Enable Websockets Support</li> <li> <p>Force SSL</p> </li> <li> <p>home.creeknet.uk \u2192 <code>http://10.0.0.13:8123</code> (Home Assistant)</p> </li> <li> <p>Provide SSL certificate through Let\u2019s Encrypt</p> </li> <li>Enable Websockets Support</li> <li>Force SSL</li> <li> <p>Advanced custom nginx config:</p> <pre><code>proxy_set_header Host $host;\nproxy_set_header X-Real-IP $remote_addr;\nproxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\nproxy_set_header X-Forwarded-Proto $scheme;\nproxy_set_header Connection \"upgrade\";\nproxy_redirect off;\n</code></pre> </li> <li> <p>nas.creeknet.uk \u2192 <code>https://10.0.0.9:1001</code> (Asustor NAS)</p> </li> <li> <p>Provide SSL certificate through Let\u2019s Encrypt</p> </li> <li> <p>plex.creeknet.uk \u2192 <code>http://10.0.0.11:32400</code> (Plex Media Server)</p> </li> <li> <p>Provide SSL certificate through Let\u2019s Encrypt</p> </li> </ul>","tags":["home lab","cloudflare","self-hosting"]},{"location":"home-server/generative-ai/ollama/","title":"How to install Ollama with a ChatGPT-like web UI","text":"<p>2024-01-10</p> <p>I am assuming that this is being installed on Ubuntu Server. Instructions compiled from:</p> <ul> <li>https://www.zdnet.com/article/docker-101-how-to-install-docker-on-ubuntu-server-22-04/</li> <li>https://ollama.ai/download/linux</li> <li>https://github.com/jmorganca/ollama/blob/main/docs/faq.md</li> <li>https://github.com/ollama-webui/ollama-webui</li> </ul> <p>First, install Ubuntu Server, ensuring that you have a modern CPU with AVX2 support as we're going to be using the CPU to run the model, rather than a GPU. You'll also need at least 8GB RAM for the model to run because we're using Mistral 7B 0.2, which requires roughly 7GB RAM to run.</p>","tags":["generative ai","ollama","mistral"]},{"location":"home-server/generative-ai/ollama/#install-ollama","title":"Install Ollama","text":"<p>Next we can install Ollama itself. Run the following command to download and install Ollama:</p> <pre><code>curl https://ollama.ai/install.sh | sh\n</code></pre> <p>Next, really insecurely open up Ollama to your entire network, or more securely to just specific IP addresses or domains. For this example, I'm going to expose it to the entire network by running these commands:</p> <pre><code>mkdir -p /etc/systemd/system/ollama.service.d\necho '[Service]' &gt;&gt;/etc/systemd/system/ollama.service.d/environment.conf\n</code></pre> <p>N.B. Change this environment variable if you want to make it more secure than <code>0.0.0.0</code>.</p> <pre><code>echo 'Environment=\"OLLAMA_HOST=0.0.0.0:11434\"' &gt;&gt;/etc/systemd/system/ollama.service.d/environment.conf\n</code></pre> <pre><code>systemctl daemon-reload\nsystemctl restart ollama\n</code></pre>","tags":["generative ai","ollama","mistral"]},{"location":"home-server/generative-ai/ollama/#install-the-mistral-7b-02-model","title":"Install the Mistral 7B 0.2 model","text":"<p>Now we can install the Mistral 7B 0.2 model. Run the following command to download and install the model:</p> <pre><code>ollama run mistral\n</code></pre> <p>Once it is downloaded you can press ctrl+d to exit.</p>","tags":["generative ai","ollama","mistral"]},{"location":"home-server/generative-ai/ollama/#install-docker","title":"Install Docker","text":"<p>This is a prerequisite for the web ui. Run the following commands to install Docker:</p> <pre><code>curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg\n</code></pre> <pre><code>echo \"deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable\" | sudo tee /etc/apt/sources.list.d/docker.list &gt; /dev/null\n</code></pre> <pre><code>sudo apt-get install apt-transport-https ca-certificates curl gnupg lsb-release -y\n</code></pre> <pre><code>sudo apt-get update\nsudo apt-get install docker-ce docker-ce-cli containerd.io -y\n</code></pre> <p>Optionally you can add your user to the docker group so you can run <code>docker</code> commands without <code>sudo</code>. If you do this, you'll need to log out and log back in (or just reboot) for this change to take effect.</p> <pre><code>sudo usermod -aG docker $USER\n</code></pre>","tags":["generative ai","ollama","mistral"]},{"location":"home-server/generative-ai/ollama/#install-the-ollama-web-ui","title":"Install the Ollama web UI","text":"<p>Run this command to create and start a new docker container running the web ui on port 3000:</p> <pre><code>docker build -t ollama-webui .\ndocker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway \\\n-v ollama-webui:/app/backend/data --name ollama-webui --restart always \\\nollama-webui\n</code></pre> <p>N.B. The documentation for this project on GitHub includes examples for if you have Ollama running on a different machine.</p>","tags":["generative ai","ollama","mistral"]},{"location":"home-server/generative-ai/ollama/#access-the-web-ui","title":"Access the web UI","text":"<p>Finally you can visit your Ubuntu machine's IP address with port 3000 and create a new admin account. You can then log in and start using the web UI, selecting your model(s) and entering a prompt to generate text from.</p>","tags":["generative ai","ollama","mistral"]},{"location":"home-server/generative-ai/ollama/#bonus-model","title":"Bonus model","text":"<p>If you want to use a model specifically as a coding assistant then the best currently available is Mixtral...which needs at least 48GB RAM to run. Personally, I'm using the 6.7 billion parameter version of the deepseek-coder model, which only needs 8GB to run. If you want to install it you can run:</p> <pre><code>ollama run deepseek-coder:6.7b\n</code></pre> <p>It is then selectable from the dropdown at the top of the web UI when you start a conversation.</p>","tags":["generative ai","ollama","mistral"]},{"location":"home-server/truenas-scale/installing-homeassistant-on-truenas-scale/","title":"Installing Home Assistant OS on Truenas Scale","text":"<p>2022-10-29 Update</p> <p>There's a newer version of HAOS available, so the below guide has been updated to use that. The original used this:</p> <p>Use wget to get the ova file:</p> <p><code>wget https://github.com/home-assistant/operating-system/releases/download/6.6/haos_ova-6.6.ova</code></p> <p>Extract the ova file using tar:</p> <p><code>tar -xvf haos_ova-6.6.ova</code></p> <p>2022-03-08</p> <p>Mostly taken from https://www.truenas.com/community/threads/home-assistant-vm-on-scale.91058/post-666766 and cleaned up.</p> <p>Make sure to use a location on your data pool as a working directory, don't use any system directory. I made a <code>hass</code> folder on my data pool <code>plex-media</code>:</p> <p><code>cd /mnt/plex-nas/plex-media/hass</code></p> <p>Use wget to get the ova file:</p> <p><code>wget https://github.com/home-assistant/operating-system/releases/download/9.3/haos_ova-9.3.ova</code></p> <p>Extract the ova file using tar:</p> <p><code>tar -xvf haos_ova-9.3.ova</code></p> <p>Convert the vmdk to a raw image file, I had to use the full working directory for the source:</p> <p><code>qemu-img convert -f vmdk -O raw /mnt/plex-nas/plex-media/hass/home-assistant.vmdk hassos.img</code></p> <p>Create a Zvol using the TrueNas Scale GUI - Be sure to make it large enough for dd to complete, I used 35 Gib.</p> <p>Use dd to write the image file to your zvol</p> <p><code>dd if=hassos.img of=/dev/plex-nas/home-assistant</code></p> <p>Create a virtual machine using the gui and attach the zvol you just created as the hdd.</p> <p>Minimum recommended assignments:</p> <ul> <li>2GB RAM</li> <li>32GB Storage</li> <li>2vCPU</li> </ul> <p>All these can be extended if your usage calls for more resources.</p>","tags":["home server","truenas scale","home assistant"]},{"location":"home-server/truenas-scale/mounting-smb-shares-at-boot-on-truenas-scale/","title":"Mounting SMB shares at boot on Truenas Scale","text":"<p>2022-11-23</p> <p>Largely taken from here and saved for posterity.</p> <p>To be able to mount the Samba share at boot, as a first thing we need to create a mountpoint on our local filesystem. For the sake of this article we will create and use the <code>/mnt/asustor-plex</code> directory for this purpose. To create the directory we can run:</p> <p><code>sudo mkdir /mnt/asustor-plex</code></p> <p>Our mountpoint is now ready. Next we need to create an entry in the <code>/etc/fstab</code> file for the Samba share. On any Linux system, the <code>/etc/fstab</code> file contains the instructions needed to mount filesystems at boot. </p> <p>For simplicity we will store the SMB credentials directly in the /etc/fstab file. For example, run <code>nano /etc/fstab</code> and add this line:</p> <p><code>//192.168.0.39/shared_data /mnt/asustor-plex cifs username=myusername,password=mypassword,noperm 0 0</code></p> <ol> <li>In the first entry field we reference the filesystem we want to mount. Normally, when dealing with standard filesystems, we reference them by using their UUID, LABEL or path. In this case, however, we need to provide the IP of the samba server together with the name of the Samba share.</li> <li>In the second field of the entry we specify the mountpoint for the filesystem. </li> <li>The third field, instead, is used to specify the filesystem type: we need to use \u201ccifs\u201d as value here.</li> <li>The fourth field is where we specify mount options: here, as we said above, we used the username and password options to pass our Samba share credentials. This way of specifying credentials has its obvious flaws, since everyone in the system is be able to read the file. Even if the file had more strict permissions, the mount options would be visible in the output of the mount command, which, when invoked without options returns a list of the mounted filesystems and the associated mount options.</li> <li><code>noperm</code> is used to allow all users on the local system to write to the share once it is mounted. This is not particularly secure, but does make things easier in a home lab setup.</li> <li>The last two fields of the fstab entry are used to specify whether the filesystem should be dumped (boolean value) and in what order filesystem should be checked (a value of 0 disables the check altogether).</li> </ol> <p>After we save the entry in the fstab file, to check that the Samba share is mounted without problems, we can simply run:</p> <p><code>sudo mount -a</code></p> <p>Now you can access the files from the share at <code>/mnt/asustor-plex</code> even after reboots.</p>","tags":["home server","truenas scale","smb","samba","nas"]},{"location":"privacy-policy/","title":"{{page.title}}","text":""},{"location":"privacy-policy/#paladins-bounty-marketplace-chrome-extension","title":"Paladins Bounty Marketplace Chrome Extension","text":"<p>This extension uses your bounty_id cookie (from the bounty marketplace) and makes a POST request to the official endpoint for retrieving the list of current offers on the market. This data is not stored anywhere, is just used within your current browser session, and is only sent to the official endpoint that the bounty marketplace itself uses.</p> <p>The below privacy policy applies other than this cookie.</p>"},{"location":"privacy-policy/#what-data-do-we-collect","title":"What data do we collect?","text":"<p>None.</p>"},{"location":"privacy-policy/#how-do-we-collect-your-data","title":"How do we collect your data?","text":"<p>I don't.</p>"},{"location":"privacy-policy/#how-will-we-use-your-data","title":"How will we use your data?","text":"<p>I won't.</p>"},{"location":"privacy-policy/#how-do-we-store-your-data","title":"How do we store your data?","text":"<p>I don't.</p>"},{"location":"privacy-policy/#marketing","title":"Marketing","text":"<p>I don't market anything to you.</p>"},{"location":"privacy-policy/#what-are-your-data-protection-rights","title":"What are your data protection rights?","text":"<p>I would like to make sure you are fully aware of all of your data protection rights. Every user is entitled to the following:</p> <ul> <li> <p>The right to access \u2013 You have the right to request Our Company for copies of your personal data. We may charge you a small fee for this service.</p> </li> <li> <p>The right to rectification \u2013 You have the right to request that Our Company correct any information you believe is inaccurate. You also have the right to request Our Company to complete the information you believe is incomplete.</p> </li> <li> <p>The right to erasure \u2013 You have the right to request that Our Company erase your personal data, under certain conditions.</p> </li> <li> <p>The right to restrict processing \u2013 You have the right to request that Our Company restrict the processing of your personal data, under certain conditions.</p> </li> <li> <p>The right to object to processing \u2013 You have the right to object to Our Company\u2019s processing of your personal data, under certain conditions.</p> </li> <li> <p>The right to data portability \u2013 You have the right to request that Our Company transfer the data that we have collected to another organization, or directly to you, under certain conditions.</p> </li> </ul> <p>If you make a request, I have one month to respond to you. If you would like to exercise any of these rights, please contact me via Twitter. </p>"},{"location":"privacy-policy/#what-are-cookies","title":"What are cookies?","text":"<p>Cookies are text files placed on your computer to collect standard Internet log information and visitor behavior information. When you visit our websites, we may collect information from you automatically through cookies or similar technology</p> <p>For further information, visit allaboutcookies.org.</p>"},{"location":"privacy-policy/#how-do-we-use-cookies","title":"How do we use cookies?","text":"<p>I don't.</p>"},{"location":"privacy-policy/#what-types-of-cookies-do-we-use","title":"What types of cookies do we use?","text":"<p>None.</p>"},{"location":"privacy-policy/#how-to-manage-your-cookies","title":"How to manage your cookies","text":"<p>You can set your browser not to accept cookies, and the above website tells you how to remove cookies from your browser. However, in a few cases, some of our website features may not function as a result.</p>"},{"location":"privacy-policy/#privacy-policies-of-other-websites","title":"Privacy policies of other websites","text":"<p>This website contains links to other websites. My privacy policy applies only to this website, so if you click on a link to another website, you should read their privacy policy.</p>"},{"location":"privacy-policy/#changes-to-our-privacy-policy","title":"Changes to our privacy policy","text":"<p>I keep my privacy policy under regular review and places any updates on this web page. This privacy policy was last updated on 4th January 2022.</p>"},{"location":"privacy-policy/#how-to-contact-us","title":"How to contact us","text":"<p>If you have any questions about this privacy policy, the data I hold on you, or you would like to exercise one of your data protection rights, please do not hesitate to contact me via Twitter.</p>"},{"location":"privacy-policy/#how-to-contact-the-appropriate-authorities","title":"How to contact the appropriate authorities","text":"<p>Should you wish to report a complaint or if you feel that I have not addressed your concern in a satisfactory manner, you may contact the Information Commissioner\u2019s Office.</p>"},{"location":"web-dev/dev-environment-container-vscode/","title":"Setting up remote containers for IAC dev environments in dotnet 6 with postgres and SMTP","text":"<p>2022-04-20</p> <p></p> <p>In this post I will go through how to set up remote containers for a dotnet 6 API project using PostgreSQL and an SMTP server, so that any new devs picking up the project only have to clone down the git repo and open it in Visual Studio code to get started debugging it.</p> <p>No dependencies need to be installed onto developer machines for each project they work on, other than:</p> <ul> <li>Docker</li> <li>Visual Studio Code</li> </ul> <p>Yes, that's all they need. They don't even need to install any extensions - the only one they need will be prompted the first time they open the project in vscode. Any extensions they need for the project will be loaded in a vscode server instance within the docker container, and then their local vscode will allow access to the vscode server as if they're using it natively, right down to providing terminal access within the container.</p>","tags":["web development","dotnet","asp","c#","vscode","devcontainer","remote-container","iac","postgres","smtp","dev environment"]},{"location":"web-dev/dev-environment-container-vscode/#how-to-set-up-the-project","title":"How to set up the project","text":"<p>You will need to ensure that you've created a <code>.gitattributes</code> file in the root folder if you don't have one already, and specified <code>* text=auto eol=lf</code> to default all the line endings to LF, if you're on Windows and find that swapping between the host and the container changes the line endings in all your files. Alternatively, if you're already using Windows line endings, use <code>crlf</code> instead of <code>lf</code>.</p> <p>In the root of the repo create a folder called <code>.devcontainer</code>. Within this you need to create three files:</p> <ol> <li><code>devcontainer.json</code></li> <li><code>docker-compose.yml</code></li> <li><code>Dockerfile</code></li> </ol>","tags":["web development","dotnet","asp","c#","vscode","devcontainer","remote-container","iac","postgres","smtp","dev environment"]},{"location":"web-dev/dev-environment-container-vscode/#devcontainerjson","title":"<code>devcontainer.json</code>","text":"<p>This is customised from the default Microsoft file. You can change the name, the vscode settings (although if you're working with newer dotnet projects I'd recommend leaving <code>omnisharp.useModernNet</code> set to <code>true</code>), add any extensions you want to use in vscode for this project (find the ID for an extension by installing it in your local vscode, then right clicking it and clicking <code>Copy Extension ID</code> - I've included some of my favourites for aspdotnet projects but the only one you need is <code>ms-dotnettools.csharp</code>), the post-create commands and the remote user.</p> <p>There is an issue where Omnisharp, the underlying technology in the C# vscode plugin doesn't work properly unless <code>dotnet restore</code> is run at the point of setting up the dev container, so do not remove that command.</p> <pre><code>// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:\n// https://github.com/microsoft/vscode-dev-containers/tree/v0.231.6/containers/dotnet-postgres\n{\n    \"name\": \"Dotnet 6 API, Postgres &amp; SMTP\",\n    \"dockerComposeFile\": \"docker-compose.yml\",\n    \"service\": \"app\",\n    \"workspaceFolder\": \"/workspace\",\n\n    // Set *default* container specific settings.json values on container create.\n    \"settings\": {\n        \"omnisharp.useModernNet\": true //Use the build of OmniSharp that runs on the .NET 6 SDK. This build requires that the .NET 6 SDK be installed and does not use Visual Studio MSBuild tools or Mono. It only supports newer SDK-style projects that are buildable with dotnet build. Unity projects and other Full Framework projects are not supported. \n    },\n\n    // Add the IDs of extensions you want installed when the container is created.\n    \"extensions\": [\n        \"formulahendry.auto-rename-tag\",\n        \"ms-dotnettools.csharp\",\n        \"EditorConfig.EditorConfig\",\n        \"dbaeumer.vscode-eslint\",\n        \"xabikos.JavaScriptSnippets\",\n        \"PKief.material-icon-theme\",\n        \"eg2.vscode-npm-script\",\n        \"christian-kohler.path-intellisense\",\n        \"esbenp.prettier-vscode\",\n        \"gencer.html-slim-scss-css-class-completion\"\n    ],\n\n    // Use 'forwardPorts' to make a list of ports inside the container available locally.\n    // \"forwardPorts\": [5000, 5001],\n\n    // [Optional] To reuse of your local HTTPS dev cert:\n    //\n    // 1. Export it locally using this command:\n    //    * Windows PowerShell:\n    //        dotnet dev-certs https --trust; dotnet dev-certs https -ep \"$env:USERPROFILE/.aspnet/https/aspnetapp.pfx\" -p \"SecurePwdGoesHere\"\n    //    * macOS/Linux terminal:\n    //        dotnet dev-certs https --trust; dotnet dev-certs https -ep \"${HOME}/.aspnet/https/aspnetapp.pfx\" -p \"SecurePwdGoesHere\"\n    // \n    // 2. Uncomment these 'remoteEnv' lines:\n    //    \"remoteEnv\": {\n    //        \"ASPNETCORE_Kestrel__Certificates__Default__Password\": \"SecurePwdGoesHere\",\n    //        \"ASPNETCORE_Kestrel__Certificates__Default__Path\": \"/home/vscode/.aspnet/https/aspnetapp.pfx\",\n    //    },\n    //\n    // 3. Next, copy your certificate into the container:\n    //      1. Start the container\n    //      2. Drag ~/.aspnet/https/aspnetapp.pfx into the root of the file explorer\n    //      3. Open a terminal in VS Code and run \"mkdir -p /home/vscode/.aspnet/https &amp;&amp; mv aspnetapp.pfx /home/vscode/.aspnet/https\"\n\n    // Use 'postCreateCommand' to run commands after the container is created.\n    \"postCreateCommand\": \"dotnet restore\", // had to use otherwise omnisharp wouldn't work properly, so no intellisense in vscode\n\n    // Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.\n    \"remoteUser\": \"vscode\"\n}\n</code></pre>","tags":["web development","dotnet","asp","c#","vscode","devcontainer","remote-container","iac","postgres","smtp","dev environment"]},{"location":"web-dev/dev-environment-container-vscode/#docker-composeyml","title":"<code>docker-compose.yml</code>","text":"<p>The docker compose file sets up the three components of this example project:</p> <ol> <li>The dotnet 6 API</li> <li>The PostgreSQL database</li> <li>The SMTP server</li> </ol> <p>Take note of the name of the SMTP server container as this will be needed later, as will the environment variables for the database. These should be changed from <code>postgres</code>.</p> <pre><code>version: '3.8'\n\nservices:\n  app:\n    build:\n      context: .\n      dockerfile: Dockerfile\n      args:\n        # Update 'VARIANT' to pick a version of .NET: 3.1, 5.0, 6.0\n        VARIANT: \"6.0\"\n        # Optional version of Node.js\n        NODE_VERSION: \"lts/*\"\n\n    volumes:\n      - ..:/workspace:cached\n\n    # Overrides default command so things don't shut down after the process ends.\n    command: sleep infinity\n\n    # Runs app on the same network as the database container, allows \"forwardPorts\" in devcontainer.json function.\n    network_mode: service:db\n\n    # Uncomment the next line to use a non-root user for all processes.\n    # user: vscode\n\n    # Use \"forwardPorts\" in **devcontainer.json** to forward an app port locally. \n    # (Adding the \"ports\" property to this file will not forward from a Codespace.)\n\n  db:\n    image: postgres:14.1\n    restart: unless-stopped\n    volumes:\n      - postgres-data:/var/lib/postgresql/data\n    environment:\n      POSTGRES_PASSWORD: postgres\n      POSTGRES_USER: postgres\n      POSTGRES_DB: postgres\n\n    # Add \"forwardPorts\": [\"5432\"] to **devcontainer.json** to forward PostgreSQL locally.\n    # (Adding the \"ports\" property to this file will not forward from a Codespace.)\n\n  mail:\n    image: bytemark/smtp\n    restart: unless-stopped\n\nvolumes:\n  postgres-data:\n</code></pre>","tags":["web development","dotnet","asp","c#","vscode","devcontainer","remote-container","iac","postgres","smtp","dev environment"]},{"location":"web-dev/dev-environment-container-vscode/#dockerfile","title":"<code>Dockerfile</code>","text":"<p>This comes directly from Microsoft, I've made no modifications to it.</p> <pre><code># [Choice] .NET version: 6.0, 5.0, 3.1, 6.0-bullseye, 5.0-bullseye, 3.1-bullseye, 6.0-focal, 5.0-focal, 3.1-focal\nARG VARIANT=\"6.0\"\nFROM mcr.microsoft.com/vscode/devcontainers/dotnet:0-${VARIANT}\n\n# [Choice] Node.js version: none, lts/*, 16, 14, 12, 10\nARG NODE_VERSION=\"none\"\nRUN if [ \"${NODE_VERSION}\" != \"none\" ]; then su vscode -c \"umask 0002 &amp;&amp; . /usr/local/share/nvm/nvm.sh &amp;&amp; nvm install ${NODE_VERSION} 2&gt;&amp;1\"; fi\n\n# [Optional] Uncomment this section to install additional OS packages.\n# RUN apt-get update &amp;&amp; export DEBIAN_FRONTEND=noninteractive \\\n#     &amp;&amp; apt-get -y install --no-install-recommends &lt;your-package-list-here&gt;\n\n# [Optional] Uncomment this line to install global node packages.\n# RUN su vscode -c \"source /usr/local/share/nvm/nvm.sh &amp;&amp; npm install -g &lt;your-package-here&gt;\" 2&gt;&amp;1\n</code></pre>","tags":["web development","dotnet","asp","c#","vscode","devcontainer","remote-container","iac","postgres","smtp","dev environment"]},{"location":"web-dev/dev-environment-container-vscode/#appsettingsjson-and-appsettingsdevelopmentjson","title":"<code>appsettings.json</code> and <code>appsettings.Development.json</code>","text":"<p>To get the SMTP working, use these keys. If you're doing anything more fancy with <code>bytemark/smtp</code> than just the default functionality you may need to change the SMTP port.</p> <pre><code>\"SmtpHost\": \"mail\",\n\"SmtpPort\": 25,\n</code></pre> <p>To connect to the Postgres database set your connection string to <code>\"User ID=postgres;Password=postgres;Host=db;Port=5432;Database=postgres;Pooling=true;\"</code>, substituting in your chosen username, password and database name.</p>","tags":["web development","dotnet","asp","c#","vscode","devcontainer","remote-container","iac","postgres","smtp","dev environment"]},{"location":"web-dev/simple-vue-spa-in-asp-dotnet-core/","title":"Creating a reactive SPA simply within an ASP.Net Core web app with Vue.js","text":"<p>2021-07-12</p> <p></p> <p>In this post, I will cover how you can quickly and easily use Vue.js to make a reactive 'SPA' within ASP.Net Core. This is a slightly dirty way to do things, but for rapid prototyping or a simple project it does the job just fine without any unnecessary complications. By 'SPA' I mean a Single Page Application, except without any navigation as that's handled by the underlying ASP.Net Core application, as is the API called by Vue.js methods.</p> <p>In the <code>&lt;head&gt;</code> tags of your Razor page/view, you'll need to include a reference to Vue.js, for example via a CDN.</p> <p><code>&lt;script src=\"https://cdn.jsdelivr.net/npm/vue@2.6.14/dist/vue.js\"&gt;&lt;/script&gt;</code></p> <p>At the bottom of your Razor page/view, you'll need to include a <code>&lt;script&gt;</code> tag to hold your Vue instance definition. In Vue.js this includes:</p> <ul> <li>the element to have the Vue instance applied to</li> <li>the data object including any default properties</li> <li>computed properties (these act as if part of the data object but reactively update like functions)</li> <li>methods (functions that can interact with the data object)</li> </ul> <p>In this example, it's a Vue instance for creating invoices and updating stock levels. The above gif shows some of the functionality achieved with this, including dynamically adding classes, and looping through elements in an array with markup for each.</p> <pre><code>&lt;script&gt;\n    const createInvoice = new Vue({\n        el: '#create-invoice',\n        data() {\n            return {\n                errorMessage: '',\n                customer: {\n                    Name: '',\n                    Address1: '',\n                    Address2: '',\n                    Address3: '',\n                    PostCode: ''\n                },\n                invoiceNumber: new Date().valueOf(),\n                products: [],\n                successMessage: ''\n            };\n        },\n        computed: {\n            calculatedVat() {\n                // take total and get 20% of it\n                return this.total * 0.2;\n            },\n            subTotal() {\n                // Take total and get 80% of it\n                return this.total * 0.8;\n            },\n            total() {\n                // For each product, add total price\n                let total = 0;\n                this.products.forEach(product =&gt; {\n                    total += product.totalPrice;\n                });\n\n                return total - this.promoDiscount;\n            },\n        },\n        methods: {\n            calculateItemTotal(product) {\n                const totalPrice = product.unitPrice * product.quantity;\n                Vue.set(product, 'totalPrice', totalPrice)\n            },\n            getDuplicateBarcodeClass(product) {\n                const matchingBarcodes = this.products.filter(obj =&gt; {\n                    return obj.barcode === product.barcode;\n                })\n\n                if (matchingBarcodes.length &gt; 1) {\n                    return 'alert-danger';\n                }\n            },\n            removeProduct(product) {\n                const index = this.products.findIndex(\n                    (x) =&gt; x.barcode === product.barcode\n                );\n\n                if (index &gt; -1) {\n                    this.products.splice(index, 1);\n                }\n            },\n            updateStock() {\n                // Check the user really wants to update the stock and start a new invoice\n                const userResponse = confirm(\"Are you sure you wish to finish this invoice and update stock levels? You will not be able to print it again.\");\n                if (userResponse) {\n                    const data = {\n                        invoiceNumber: String(this.invoiceNumber),\n                        products: this.products,\n                    };\n\n                    // Update the stock &amp; log the invoice\n                    fetch(`Invoices/UpdateStockAndLogInvoice/`, {\n                        method: \"POST\",\n                        headers: {\n                            \"Content-Type\": \"application/json\",\n                        },\n                        body: JSON.stringify(data),\n                    })\n                        .then((response) =&gt; {\n                            return response.json();\n                        })\n                        .then((data) =&gt; {\n                            if (data.statusCode === 404) {\n                                Vue.set(this, \"errorMessage\", data.message);\n                            } else {\n                                Vue.set(this, 'successMessage', data.message);\n\n                                // If no errors, clear the products list to be ready for the next invoice\n                                Vue.set(this, \"products\", []);\n                            }\n                        })\n                        .catch((err) =&gt; {\n                            console.error(err);\n                            Vue.set(this, \"errorMessage\", err);\n                        });\n                }\n            },\n            validateNumber: (event) =&gt; {\n                // Only accept integers, so prevent anything but digits 0-9\n                let keyCode = event.keyCode;\n                if (keyCode &lt; 48 || keyCode &gt; 57) {\n                    event.preventDefault();\n                }\n            },\n            viewAndPrintInvoice() {\n                if (this.isDuplicateBarcode) {\n                    alert(\"There is a duplicate barcode - look for red barcodes and remove duplicates\");\n                }\n                else {\n                    window.print();\n                }\n            }\n        },\n    });\n&lt;/script&gt;\n</code></pre> <p>To hook this into the DOM, just add the element identifier to an element, for example <code>&lt;div id=\"create-invoice\"&gt;</code> will bind a div to this Vue instance.</p> <p>Within that, you are free to use a variety of tools, like <code>v-if</code>, interpolation and <code>v-on</code> event handlers. In this example, if there's any text in the <code>successMessage</code> property of the Vue data object, this alert div will be displayed, interpolating the message, and clearing it (thus hiding the alert) when the close button is clicked.</p> <pre><code>&lt;div v-if=\"successMessage\" class=\"alert alert-success alert-dismissible fade show\" role=\"alert\"&gt;\n    &lt;i class=\"bi bi-check-circle-fill\"&gt;&lt;/i&gt;\n    &lt;strong&gt;Success:&lt;/strong&gt; {{ successMessage }}\n    &lt;button type=\"button\" class=\"close\" aria-label=\"Close\" v-on:click=\"successMessage = ''\"&gt;\n        &lt;span aria-hidden=\"true\"&gt;&amp;times;&lt;/span&gt;\n    &lt;/button&gt;\n&lt;/div&gt;\n</code></pre> <p>In this example, a <code>v-for</code> is used to loop through all the products in the array, producing a <code>&lt;tr&gt;</code> for each of them, with an appropriate class bound  from a method taking that product as a parameter. <code>v-if</code> and <code>v-else</code> statements enable conditionally showing one (or none) of three different pooltips. The product's <code>name</code> property is interpolated and shown, as is the <code>unitPrice</code> property, which is nicely formatted for readability.</p> <p>The <code>barcode</code> field is mapped to a <code>v-model</code>, which means as it changes it directly alters the value in the Vue data object, and as that changes this updates reactively. It also makes use of <code>v-on:blur</code> to run a method that makes an API call to retrieve product data, conditionally sets the disabled property once a barcode has been entered, and binds a class based on the output of a method taking the product as a parameter.</p> <p>The <code>quantity</code> field also uses a <code>v-model</code>, but specifically forces it to be a number (no need for Typescript here) runs a method on keypress (the <code>@@</code> syntax is purely due to Razor escaping, normally you'd only need one) and on change.</p> <pre><code>&lt;tr v-for=\"product in products\" v-bind:class=\"getProductRowClass(product)\"&gt;\n    &lt;td&gt;\n        &lt;div class=\"tooltip\" v-if=\"product.barcode.length &gt; 0 &amp;&amp; product.stockLevel - product.quantity &lt; 0\"&gt;\n            &lt;i class=\"bi bi-x-circle-fill text-danger\"&gt;&lt;/i&gt;\n            &lt;span class=\"tooltiptext\"&gt;You will need to order more stock to fulfill this order&lt;/span&gt;\n        &lt;/div&gt;\n        &lt;div class=\"tooltip\" v-else-if=\"product.barcode.length &gt; 0 &amp;&amp; product.stockLevel - product.quantity == 0\"&gt;\n            &lt;i class=\"bi bi-exclamation-triangle-fill text-warning\"&gt;&lt;/i&gt;\n            &lt;span class=\"tooltiptext\"&gt;You will run out of stock&lt;/span&gt;\n        &lt;/div&gt;\n        &lt;div class=\"tooltip\" v-else-if=\"product.barcode.length &gt; 0 &amp;&amp; product.stockLevel - product.quantity &lt;= 10\"&gt;\n            &lt;i class=\"bi bi-info-circle-fill text-info\"&gt;&lt;/i&gt;\n            &lt;span class=\"tooltiptext\"&gt;Stock will be low after this order&lt;/span&gt;\n        &lt;/div&gt;\n    &lt;/td&gt;\n    &lt;td&gt;\n        &lt;label&gt;{{product.name}}&lt;/label&gt;\n    &lt;/td&gt;\n    &lt;td&gt;\n        &lt;input type=\"text\" name=\"barcode\" v-model=\"product.barcode\" v-on:blur=\"getProductInformation(product)\" :disabled=\"product.barcode.length &gt; 0\" v-bind:class=\"getDuplicateBarcodeClass(product)\"&gt;\n    &lt;/td&gt;\n    &lt;td&gt;\n        &lt;input type=\"number\" step=\"1\" name=\"quantity\" v-model.number=\"product.quantity\" @@keypress=\"validateNumber\" v-on:change=\"calculateItemTotal(product)\"&gt;\n    &lt;/td&gt;\n    &lt;td&gt;\n        &lt;label&gt;&amp;#163;{{parseFloat(product.unitPrice).toFixed(2)}}&lt;/label&gt;\n    &lt;/td&gt;\n&lt;/tr&gt;\n</code></pre> <p>This is a very simple, quick and dirty example, but hopefully it gives you an idea of the kind of things you can quickly and easily achieve using Vue.js in this way. While this example uses ASP.Net Core, you could easily plug this into any other website, so long as it's using HTML, CSS and JavaScript, and there is an API to make requests to.</p>","tags":["web development","dotnet","c#","vue","asp"]},{"location":"web-dev/dotnet-csharp/jsnlog-asp-net-6/","title":"Adding JSNLog to ASP .Net 6 with Serilog","text":"<p>2022-01-18</p> <p>In this post I will go through the steps required to get an optimal install of JSNLog in ASP dotnet 6 projects using Serilog. JSNLog is a fantastic tool for logging uncaught javascript exceptions to the backend's logging system, and can be used for a variety of other things. This config includes batching of messages to reduce calls made to the server, as well as buffering, where helpful debugging logs are only sent if there's a fatal log that's also being sent, to make it easier to diagnose bugs without unnecessary diagnostic logs appearing at all times.</p> <p>First you need to install <code>JSNLog</code> and <code>Destructurama.JsonNet</code> via NuGet. After that, make the following changes to these files.</p>","tags":["web development","dotnet","asp","c#","serilog","jsnlog"]},{"location":"web-dev/dotnet-csharp/jsnlog-asp-net-6/#_viewimportscshtml","title":"_ViewImports.cshtml","text":"<p>Add <code>@addTagHelper \"*, jsnlog\"</code> to enable the tag helper.</p>","tags":["web development","dotnet","asp","c#","serilog","jsnlog"]},{"location":"web-dev/dotnet-csharp/jsnlog-asp-net-6/#startupcs","title":"Startup.cs","text":"<pre><code>using Destructurama;\nusing JSNLog;\n\n...\npublic Startup(IConfiguration configuration, IWebHostEnvironment environment)\n{\n    ...\n    Log.Logger = SerilogLogger.CreateSerilogLogger(serilogConfiguration).Destructure.JsonNetTypes().CreateLogger();\n    ...\n}\n\n...\n\npublic void Configure(\n    ...,\n    ILoggerFactory loggerFactory)\n{\n    ...\n    loggerFactory.AddSerilog();\n\n    JsnlogConfiguration jsnlogConfiguration = new JsnlogConfiguration\n    {\n        ajaxAppenders = new List&lt;AjaxAppender&gt; {\n            new AjaxAppender {\n                name = \"appender1\",\n                storeInBufferLevel = \"TRACE\", // Log messages with severity smaller than TRACE are ignored\n                level = \"WARN\", // Log messages with severity equal or greater than TRACE and lower than WARN are stored in the internal buffer, but not sent to the server\n                                // Log messages with severity equal or greater than WARN and lower than FATAL are sent to the server on their own\n                sendWithBufferLevel = \"FATAL\", // Log messages with severity equal or greater than FATAL are sent to the server, along with all messages stored in the internal buffer\n                bufferSize = 20, // Stores the last up to 20 debug messages in browser memory,\n                batchSize = 20,\n                batchTimeout = 2000, // Logs are guaranteed to be sent within this period (in ms), even if the batch size has not been reached yet\n                maxBatchSize = 50 // When the server is unreachable and log messages are being stored until it is reachable again, this is the maximum number of messages that will be stored. Cannot be smaller than batchSize\n            }\n        },\n        loggers = new List&lt;Logger&gt; { // Get the loggers to use the new appender\n            new Logger {\n                appenders = \"appender1\"\n            }\n        },\n        insertJsnlogInHtmlResponses = false, // There's an outstanding bug setting this to true so the workaround is using the jl-javascript-logger-definitions tag helper in _Layout.cshtml, via the reference in _ViewImports.cshtml\n        productionLibraryPath = null // We're using a fallback from the CDN with hashing, so do not use this\n    };\n\n    // Must be before UseStaticFiles and UseAuthorization\n    app.UseJSNLog(new CustomLoggingAdapter(loggerFactory), jsnlogConfiguration);\n    ...\n}\n</code></pre>","tags":["web development","dotnet","asp","c#","serilog","jsnlog"]},{"location":"web-dev/dotnet-csharp/jsnlog-asp-net-6/#_layoutcshtml","title":"_Layout.cshtml","text":"<p>This one has some interesting configuration.</p> <p>Firstly, using local fallbacks if the CDN versions of the scripts are unavailable, or do not match their sha384 hashes (i.e. have been hacked or changed).</p> <p>Secondly, it injects some additional information into the logs: client IP address, user ID (your model may differ from my example), and the user agent.</p> <p>Thirdly, it sets up logging any uncaught JS exceptions, and any exceptions inside promises where no rejection method is provided.</p> <p>Fourthly, using stacktrace.js it handles getting the relevant stacktrace using sourcemaps.</p> <pre><code>&lt;script src=\"https://cdnjs.cloudflare.com/ajax/libs/jsnlog/2.30.0/jsnlog.min.js\"\n    asp-fallback-src=\"~/jsnlog.min.js\"\n    asp-fallback-test=\"window.JL\"\n    crossorigin=\"anonymous\"\n    integrity=\"sha384-ANmgu3V8Mc5/Usd/GeIS0xu0spgLKTIqkSMQAgVvV5C2SRp/rICkLLw5XG/u6BQ9\"&gt;\n&lt;/script&gt;\n&lt;script src=\"https://cdnjs.cloudflare.com/ajax/libs/stacktrace.js/2.0.2/stacktrace.min.js\"\n    asp-fallback-src=\"~/stacktrace.min.js\"\n    asp-fallback-test=\"window.StackTrace\"\n    crossorigin=\"anonymous\"\n    integrity=\"sha384-4PjQM+vlPbdcaPnFOyBIOfqz90Hvhp+QHb3rBMOy78OaxDHw9mnmzjUSNqJkn+W5\"&gt;\n&lt;/script&gt;\n\n&lt;jl-javascript-logger-definitions /&gt;\n\n&lt;script&gt;\n    async function fetchClientIp() {\n        try {\n            let response = await fetch('https://api.ipify.org?format=json');\n            return await response.json();\n        } catch (error) {\n            JL().fatalException(\"Failed to get client IP for logging\", error);\n        }\n    }\n    fetchClientIp().then(data =&gt; {\n        const userDetailsForLogging = {\n            'clientIp': data.ip,\n            'userId': '@Model.UserId',\n            'userAgent': navigator.userAgent\n        }\n        @* Log uncaught JavaScript errors to the serverside log *@\n        if (window) {\n            window.onerror = function (errorMsg, url, lineNumber, column, errorObj) {\n                var callback = function(stackframes) {\n                    var stringifiedStack = stackframes.map(function(sf) {\n                        return sf.toString();\n                    }).join('\\n');\n                    const msgObj = {\n                        'msg': 'Uncaught exception',\n                        'sourceMapStack': stringifiedStack,\n                        'user': userDetailsForLogging\n                    };\n                    JL('serverLog').fatalException({\n                        msg: JSON.stringify(msgObj),\n                        errorMsg: errorMsg, \n                        url: url, \n                        lineNumber: lineNumber, \n                        column: column\n                    }, errorObj);\n                };\n                var errback = function(err) { console.log(err.message); };\n                StackTrace.fromError(errorObj).then(callback).catch(errback);\n                return false;\n            };\n        }\n\n        @* Log uncaught JavaScript exceptions inside promises where no rejection method is provided *@\n        if (typeof window !== 'undefined') {\n            window.onunhandledrejection = function (event) {\n                var callback = function(stackframes) {\n                    var stringifiedStack = stackframes.map(function(sf) {\n                        return sf.toString();\n                    }).join('\\n');\n\n                    const msgObj = {\n                        'msg': 'Unhandled promise rejection',\n                        'sourceMapStack': stringifiedStack,\n                        'user': userDetailsForLogging\n                    };\n                    JL(\"onerrorLogger\").fatalException({\n                        msg: JSON.stringify(msgObj),\n                        errorMsg: event.reason ? event.reason.message : null\n                    }, event.reason);\n                };\n                var errback = function(err) { console.log(err.message); };\n                StackTrace.fromError(errorObj).then(callback).catch(errback);\n                return false;\n            };\n        }\n    });\n&lt;/script&gt;\n</code></pre>","tags":["web development","dotnet","asp","c#","serilog","jsnlog"]},{"location":"web-dev/dotnet-csharp/jsnlog-asp-net-6/#customloggingadaptercs","title":"CustomLoggingAdapter.cs","text":"<pre><code>using JSNLog;\nusing Newtonsoft.Json;\nusing System.Text;\n\nnamespace YourProjectHere\n{\n    /// &lt;summary&gt;\n    /// This adapter is required to get JavaScript objects logged by JSNLog to appear correctly in Serilog. Regretably it is required \n    /// because of a defficiency in JSNLog that hasn't been fixed in at least 5 years at the time of writing, and depends on slightly \n    /// customised versions of classes and methods taken from that library, included here in this one file for tidiness.\n    /// &lt;/summary&gt;\n    public class CustomLoggingAdapter : ILoggingAdapter\n    {\n        private readonly ILoggerFactory _loggerFactory;\n        public CustomLoggingAdapter(ILoggerFactory loggerFactory)\n        {\n            _loggerFactory = loggerFactory;\n        }\n        public void Log(FinalLogData finalLogData)\n        {\n            ILogger logger = _loggerFactory.CreateLogger(finalLogData.FinalLogger);\n            Object message = LogMessageHelpers.DeserializeIfPossible(finalLogData.FinalMessage);\n            switch (finalLogData.FinalLevel)\n            {\n                case Level.TRACE: logger.LogTrace(\"{@logMessage}\", message); break;\n                case Level.DEBUG: logger.LogDebug(\"{@logMessage}\", message); break;\n                case Level.INFO: logger.LogInformation(\"{@logMessage}\", message); break;\n                case Level.WARN: logger.LogWarning(\"{@logMessage}\", message); break;\n                case Level.ERROR: logger.LogError(\"{@logMessage}\", message); break;\n                case Level.FATAL: logger.LogCritical(\"{@logMessage}\", message); break;\n                default:\n                    break;\n            }\n        }\n    }\n    internal class LogMessageHelpers\n    {\n        public static T DeserializeJson&lt;T&gt;(string json)\n        {\n            T result = JsonConvert.DeserializeObject&lt;T&gt;(json);\n            return result;\n        }\n        public static bool IsPotentialJson(string msg)\n        {\n            string trimmedMsg = msg.Trim();\n            return (trimmedMsg.StartsWith(\"{\") &amp;&amp; trimmedMsg.EndsWith(\"}\"));\n        }\n        /// &lt;summary&gt;\n        /// Tries to deserialize msg.\n        /// If that works, returns the resulting object.\n        /// Otherwise returns msg itself (which is a string).\n        /// &lt;/summary&gt;\n        /// &lt;param name=\"msg\"&gt;&lt;/param&gt;\n        /// &lt;returns&gt;&lt;/returns&gt;\n        public static Object DeserializeIfPossible(string msg)\n        {\n            try\n            {\n                if (IsPotentialJson(msg))\n                {\n                    Object result = DeserializeJson&lt;Object&gt;(msg);\n                    return result;\n                }\n            }\n            catch\n            {\n            }\n            return msg;\n        }\n        /// &lt;summary&gt;\n        /// Returns true if the msg contains a valid JSON string.\n        /// &lt;/summary&gt;\n        /// &lt;param name=\"msg\"&gt;&lt;/param&gt;\n        /// &lt;returns&gt;&lt;/returns&gt;\n        public static bool IsJsonString(string msg)\n        {\n            try\n            {\n                if (IsPotentialJson(msg))\n                {\n                    // Try to deserialise the msg. If that does not throw an exception,\n                    // decide that msg is a good JSON string.\n                    DeserializeJson&lt;Dictionary&lt;string, Object&gt;&gt;(msg);\n                    return true;\n                }\n            }\n            catch\n            {\n            }\n            return false;\n        }\n        /// &lt;summary&gt;\n        /// Takes a log message and finds out if it contains a valid JSON string.\n        /// If so, returns it unchanged.\n        /// \n        /// Otherwise, surrounds the string with quotes (\") and escapes the string for JavaScript.\n        /// &lt;/summary&gt;\n        /// &lt;returns&gt;&lt;/returns&gt;\n        public static string EnsureValidJson(string msg)\n        {\n            if (IsJsonString(msg))\n            {\n                return msg;\n            }\n            return JavaScriptStringEncode(msg, true);\n        }\n        public static string JavaScriptStringEncode(string value, bool addDoubleQuotes)\n        {\n#if NETFRAMEWORK\n            return System.Web.HttpUtility.JavaScriptStringEncode(value, addDoubleQuotes);\n#else\n            // copied from https://github.com/mono/mono/blob/master/mcs/class/System.Web/System.Web/HttpUtility.cs\n            if (String.IsNullOrEmpty(value))\n                return addDoubleQuotes ? \"\\\"\\\"\" : String.Empty;\n            int len = value.Length;\n            bool needEncode = false;\n            char c;\n            for (int i = 0; i &lt; len; i++)\n            {\n                c = value[i];\n                if (c &gt;= 0 &amp;&amp; c &lt;= 31 || c == 34 || c == 39 || c == 60 || c == 62 || c == 92)\n                {\n                    needEncode = true;\n                    break;\n                }\n            }\n            if (!needEncode)\n                return addDoubleQuotes ? \"\\\"\" + value + \"\\\"\" : value;\n            var sb = new StringBuilder();\n            if (addDoubleQuotes)\n                sb.Append('\"');\n            for (int i = 0; i &lt; len; i++)\n            {\n                c = value[i];\n                if (c &gt;= 0 &amp;&amp; c &lt;= 7 || c == 11 || c &gt;= 14 &amp;&amp; c &lt;= 31 || c == 39 || c == 60 || c == 62)\n                    sb.AppendFormat(\"\\\\u{0:x4}\", (int)c);\n                else switch ((int)c)\n                    {\n                        case 8:\n                            sb.Append(\"\\\\b\");\n                            break;\n                        case 9:\n                            sb.Append(\"\\\\t\");\n                            break;\n                        case 10:\n                            sb.Append(\"\\\\n\");\n                            break;\n                        case 12:\n                            sb.Append(\"\\\\f\");\n                            break;\n                        case 13:\n                            sb.Append(\"\\\\r\");\n                            break;\n                        case 34:\n                            sb.Append(\"\\\\\\\"\");\n                            break;\n                        case 92:\n                            sb.Append(\"\\\\\\\\\");\n                            break;\n                        default:\n                            sb.Append(c);\n                            break;\n                    }\n            }\n            if (addDoubleQuotes)\n                sb.Append('\"');\n            return sb.ToString();\n#endif\n        }\n    }\n}\n</code></pre>","tags":["web development","dotnet","asp","c#","serilog","jsnlog"]},{"location":"web-dev/dotnet-csharp/pattern-matching/","title":"C# 8 Pattern Matching","text":"<p>2022-02-15</p> <p>This serves as introductory documentation to the pattern matching features introduced in C# 8, which I think are particularly useful.</p>","tags":["web development","dotnet","c#","pattern matching"]},{"location":"web-dev/dotnet-csharp/pattern-matching/#deconstructors-positional-patterns","title":"Deconstructors &amp; Positional Patterns","text":"<p>Adding a deconstructor to your class can be done as below. It must be named <code>Deconstruct</code> and be a <code>public void</code>. Any values we want deconstructed can be set as <code>out</code> parameters, then populated. Here all properties are being deconstructed, but a subset could be deconstructed if desired.</p> <pre><code>class Student\n{\n    public string Name { get; set; }\n    public int Year { get; set; }\n    public Teacher FormTutor { get; set; }\n\n    public void Deconstruct (out string name, out int year, out Teacher formTutor) \n    {\n        name = Name;\n        year = Year;\n        formTutor = FormTutor;\n    }\n}\n</code></pre> <p>To use positional parameters, I'll also set up a second model.</p> <pre><code>class Teacher\n{\n    public string Name { get; set; }\n    public string Subject { get; set; }\n\n    public void Deconstruct (out string name, out string subject) \n    {\n        name = Name;\n        subject = Subject;\n    }\n}\n</code></pre> <p>Here is an example positional pattern. The discards (<code>_</code>) are used to 'match all'.</p> <pre><code>public static class YourClassHere\n{\n    public static bool IsInYear9English(Student student)\n    {\n        return student is Student(\n            _,\n            9,\n            Teacher (_, \"English\")\n        );\n    }\n}\n</code></pre> <p>While an if statement would probably be easier to read and maintain in this example, positional patterns enable recursion to be used, which can be very useful.</p>","tags":["web development","dotnet","c#","pattern matching"]},{"location":"web-dev/dotnet-csharp/pattern-matching/#property-patterns","title":"Property Patterns","text":"<p>I've updated the Student model to add a School property.</p> <pre><code>class Student\n{\n    public string Name { get; set; }\n    public int Year { get; set; }\n    public string School { get; set; }\n    public Teacher FormTutor { get; set; }\n}\n</code></pre> <p>I prefer using property patterns to positional patterns as they are much more readable. Here's an example method to check whether a student attends a particular school and has a form tutor who teaches Maths.</p> <pre><code>public static class YourClassHere\n{\n    public static bool IsStudentInGreenAcademyWithMathsFormTutor(Student student)\n    {\n        return student is {\n            School: \"Green Academy\", \n            FormTutor: {\n                Subject: \"Maths\"\n            } \n        };\n    }\n}\n</code></pre> <p>This can be made more generic to accept an <code>object</code> rather than a <code>Student</code>, and check that object is a Student.</p> <pre><code>public static class YourClassHere\n{\n    public static bool IsStudentInGreenAcademyWithMathsFormTutor(object obj)\n    {\n        return obj is Student student &amp;&amp; \n            student is {\n                School: \"Green Academy\", \n                FormTutor: {\n                    Subject: \"Maths\"\n                } \n            };\n    }\n}\n</code></pre>","tags":["web development","dotnet","c#","pattern matching"]},{"location":"web-dev/dotnet-csharp/pattern-matching/#switch-expressions","title":"Switch Expressions","text":"<p>These can be used in place of standard switch cases. I'm using a discard (<code>_</code>) for catching the default case for unmatched patterns.</p> <pre><code>public static class YourClassHere\n{\n    public static string DisplayPersonInfo (object person)\n    {\n        string result = person switch\n        {\n            Student student =&gt; $\"Student at {student.School} in Year {student.Year}\",\n            Teacher teacher =&gt; $\"Teacher of {teacher.Subject}\",\n            _ =&gt; \"Person is not a student or a teacher\"\n        };\n\n        return result;\n    }\n}\n</code></pre> <p>The syntax can make it easier to read, and it can be a lot more powerful. You can also define recursive switch patterns.</p> <pre><code>public static class YourClassHere\n{\n    public static string DisplayPersonInfo (object person)\n    {\n        string result = person switch\n        {\n            Student student =&gt; student switch\n            {\n                _ when student.Year &lt; 10 and student.Year &gt; 6 =&gt; \"Student in Key Stage 3\",\n                _ when student.Year &gt;= 10 and student.Year &lt;= 13&gt; =&gt; \"Student in Key Stage 4\",\n                _ =&gt; $\"Student in Year {student.Year}\"\n            },\n            Teacher teacher =&gt; $\"Teacher of {teacher.Subject}\",\n            _ =&gt; \"Person is not a student or a teacher\"\n        };\n\n        return result;\n    }\n}\n</code></pre>","tags":["web development","dotnet","c#","pattern matching"]},{"location":"web-dev/dotnet-csharp/pattern-matching/#tuple-patterns-with-switch-expressions","title":"Tuple Patterns (with Switch Expressions)","text":"<p>You can also use Switch Expressions with Tuples to write even more useful code. For example, you could be creating a game with crafting, combining two items to make another.</p> <pre><code>public static class YourClassHere\n{\n    public static CraftingMaterial GetCraftingMaterial (CraftingMaterial item1, CraftingMaterial item2)\n    {\n        return (item1, item2) switch\n        {\n            // Match the items in both positions\n            (CraftingMaterial.MountainFlowers, CraftingMaterial.SoulGems) =&gt; CraftingMaterial.DwemerMetal,\n            (CraftingMaterial.SoulGems, CraftingMaterial.MountainFlowers) =&gt; CraftingMaterial.DwemerMetal,\n\n            (CraftingMaterial.Ore, CraftingMaterial.DwemerMetal) =&gt; CraftingMaterial.Ingots,\n            (CraftingMaterial.DwemerMetal, CraftingMaterial.Ore) =&gt; CraftingMaterial.Ingots,\n\n            // Handle both items being the same (discard for both, to match any)\n            (_, _) when item1 == item2 =&gt; item1,\n\n            // Default case (with discard)\n            _ =&gt; CraftingMaterial.Unknown\n        };\n    }\n}\n</code></pre> <p>This is lovely and easy to read, as well as very powerful.</p>","tags":["web development","dotnet","c#","pattern matching"]},{"location":"web-dev/dotnet-csharp/semantic-kernel-vector-database/","title":"Semantic Kernel (GPT with C#) - using a vector database to store embeddings","text":"<p>2023-05-25</p> <p>At the time of writing there isn't any documentation on using vector databases to store embeddings, so I'm sharing my notes from my time digging around in the semantic kernel code.</p>","tags":["web development","dotnet","c#","AI","GPT","SK","semantic kernel"]},{"location":"web-dev/dotnet-csharp/semantic-kernel-vector-database/#a-brief-explanation","title":"A brief explanation","text":"<p>So, what am I even talking about? As the global phenomenon that is Chat GPT has shown, the OpenAI GPT models are capable of making some impressively powerful tools. However, even as an experienced developer, working with LLM AIs is not something I nor, most of my colleagues have experience of. This is where the semantic kernel comes in. It's essentially a wrapper that allows you to use the GPT models in a more familiar way, using C# (or Python). It's a great tool for developers who want to use the power of GPT, but don't want to have to learn how to use it directly. </p> <p>There's some excellent resources available covering how you can create a text file that serves as a 'skill' - a set of prompts and responses that the AI can use to generate text - and for using volatile memory to store 'embeddings'. I won't go into those here, but the official Microsoft documentation is a great place to start.</p> <p>Embeddings are essentially a way of storing a representation of a piece of text. The semantic kernel uses them to store the prompts and responses that you provide, and then uses them to find the most similar text to a given input. This is how it can generate responses to a given input. When you start trying to add context to a skill, you'll quickly find that the amount of prompts and responses you need to add to cover all the possible inputs is huge. This is where a vector database comes in.</p> <p>A vector database is different to a relational or document database. I won't explain the differences here, but they can be used to store data in a way that best enables you to programmatically find the most similar data to a given input. In this case, I'm going to store contextual data that I want my skill to use to generate responses. I'll then use the vector database to provide this data to my skill, in a way that will not exceed the token limits of the GPT model.</p>","tags":["web development","dotnet","c#","AI","GPT","SK","semantic kernel"]},{"location":"web-dev/dotnet-csharp/semantic-kernel-vector-database/#setting-up-a-console-app-to-make-use-of-semantic-kernel","title":"Setting up a console app to make use of semantic kernel","text":"<p>There's already excellent documentation on how to do this, so I won't go into it here. I'll assume you've already got a console app set up and ready to go, and that you've added the semantic kernel NuGet package using <code>dotnet add package Microsoft.SemanticKernel --version 0.14.547.1-preview</code> - take a look at the official documentation if you haven't.</p>","tags":["web development","dotnet","c#","AI","GPT","SK","semantic kernel"]},{"location":"web-dev/dotnet-csharp/semantic-kernel-vector-database/#setting-up-the-vector-database-to-store-embeddings","title":"Setting up the vector database to store embeddings","text":"<p>This is remarkably easy thanks to some NuGet packages that have been provided alongside the semantic kernel. I'm going to use the SQLite database, but there are other, more robust options available.</p> <p>You'll need to install the relevant NuGet package, which in our case is the SQLite one - <code>dotnet add package Microsoft.SemanticKernel.Connectors.Memory.Sqlite --version 0.14.547.1-preview</code> will do that for you.</p> <p>First, set up a new kernel, enabling embeddings, and connecting to your SQLite database. You'll need to provide a connection string, which is just a path to the database file. I've used a relative path here, but you can use an absolute path if you prefer. I've left in a commented out line to demonstrate what setting this up with a standard in-memory database would look like. </p> <pre><code>var kernel = new KernelBuilder()\n    .Configure(c =&gt;\n    {\n        if (useAzureOpenAI)\n        {\n            c.AddAzureTextEmbeddingGenerationService(\"text-embedding-ada-002\", azureEndpoint, apiKey);\n            c.AddAzureTextCompletionService(model, azureEndpoint, apiKey);\n        }\n        else\n        {\n            c.AddOpenAITextEmbeddingGenerationService(\"text-embedding-ada-002\", apiKey);\n            c.AddOpenAITextCompletionService(model, apiKey, orgId);\n        }\n    })\n    .WithMemoryStorage(await SqliteMemoryStore.ConnectAsync(\"memory.sqlite\"))\n    // .WithMemoryStorage(new VolatileMemoryStore())\n    .Build();\n</code></pre> <p>Next I'm going to set up a dictionary of GitHub URLs and their descriptions. I'm going to use these to demonstrate how to store embeddings in the vector database. I've used a dictionary here, but you could use any collection of key-value pairs.</p> <pre><code>const string memoryCollectionName = \"SKGitHub\";\n\nvar githubFiles = new Dictionary&lt;string, string&gt;()\n{\n    [\"https://github.com/microsoft/semantic-kernel/blob/main/README.md\"]\n        = \"README: Installation, getting started, and how to contribute\",\n    [\"https://github.com/microsoft/semantic-kernel/blob/main/samples/notebooks/dotnet/02-running-prompts-from-file.ipynb\"]\n        = \"Jupyter notebook describing how to pass prompts from a file to a semantic skill or function\",\n    [\"https://github.com/microsoft/semantic-kernel/blob/main/samples/notebooks/dotnet/00-getting-started.ipynb\"]\n        = \"Jupyter notebook describing how to get started with the Semantic Kernel\",\n    [\"https://github.com/microsoft/semantic-kernel/tree/main/samples/skills/ChatSkill/ChatGPT\"]\n        = \"Sample demonstrating how to create a chat skill interfacing with ChatGPT\",\n    [\"https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/SemanticKernel/Memory/Volatile/VolatileMemoryStore.cs\"]\n        = \"C# class that defines a volatile embedding store\",\n    [\"https://github.com/microsoft/semantic-kernel/tree/main/samples/dotnet/KernelHttpServer/README.md\"]\n        = \"README: How to set up a Semantic Kernel Service API using Azure Function Runtime v4\",\n    [\"https://github.com/microsoft/semantic-kernel/tree/main/samples/apps/chat-summary-webapp-react/README.md\"]\n        = \"README: README associated with a sample starter react-based chat summary webapp\",\n};\n</code></pre> <p>Obviously these have to be stored in the vector database before they can be used. I'm going to use the <code>SaveReferenceAsync</code> method to do this. This method takes a collection name, a description, the text to be stored, and an external ID and source name. The external ID is a unique identifier for the piece of text, and the source name is a string that identifies the source of the text. I'm using the GitHub URL as the external ID, and the source name as 'GitHub'. You can use whatever you like here, but it's important that the external ID is unique. I'm also going to print out a message for each URL that's saved, so I can see what's happening.</p> <pre><code>Console.WriteLine(\"Adding some GitHub file URLs and their descriptions to a volatile Semantic Memory.\");\nvar i = 0;\nforeach (var entry in githubFiles)\n{\n    await kernel.Memory.SaveReferenceAsync(\n        collection: memoryCollectionName,\n        description: entry.Value,\n        text: entry.Value,\n        externalId: entry.Key,\n        externalSourceName: \"GitHub\"\n    );\n    Console.WriteLine($\"  URL {++i} saved\");\n}\n</code></pre>","tags":["web development","dotnet","c#","AI","GPT","SK","semantic kernel"]},{"location":"web-dev/dotnet-csharp/semantic-kernel-vector-database/#querying-the-vector-database","title":"Querying the vector database","text":"<p>Now that we've stored some data in the vector database, we can query it. I'm going to use the <code>SearchAsync</code> method to do this. This method takes a collection name, a query string, and a limit and minimum relevance score. The limit is the maximum number of results to return, and the minimum relevance score is the minimum relevance score that a result must have to be returned. I'm going to set the limit to 5, and the minimum relevance score to 0.77. I'm also going to print out the query string, so I can see what's happening.</p> <p>The reason I want to limit the number of results is that the vector database will return all results that have a relevance score above the minimum relevance score. If I didn't limit the number of results, I'd get all the results that have a relevance score above 0.77, which would be all of them. It is critical to ensure that we stay below the token limit for the OpenAI API for the 'ask' that we're going to perform, and finding only a limited set of the most relevant data to use for the 'ask' is a good way to do this.</p> <pre><code>string ask = \"I love Jupyter notebooks, how should I get started?\";\nConsole.WriteLine(\"===========================\\n\" +\n                    \"Query: \" + ask + \"\\n\");\n\nvar memories = kernel.Memory.SearchAsync(memoryCollectionName, ask, limit: 5, minRelevanceScore: 0.77);\n\ni = 0;\nawait foreach (MemoryQueryResult memory in memories)\n{\n    Console.WriteLine($\"Result {++i}:\");\n    Console.WriteLine(\"  URL:     : \" + memory.Metadata.Id);\n    Console.WriteLine(\"  Title    : \" + memory.Metadata.Description);\n    Console.WriteLine(\"  Relevance: \" + memory.Relevance);\n    Console.WriteLine();\n}\n</code></pre> <p>Running this ask gives the following results:</p> <pre><code>===========================\nQuery: I love Jupyter notebooks, how should I get started?\n\nResult 1:\n  URL:     : https://github.com/microsoft/semantic-kernel/blob/main/samples/notebooks/dotnet/00-getting-started.ipynb\n  Title    : Jupyter notebook describing how to get started with the Semantic Kernel\n  Relevance: 0.8678345730903347\n\nResult 2:\n  URL:     : https://github.com/microsoft/semantic-kernel/blob/main/samples/notebooks/dotnet/02-running-prompts-from-file.ipynb\n  Title    : Jupyter notebook describing how to pass prompts from a file to a semantic skill or function\n  Relevance: 0.8163350291107238\n\nResult 3:\n  URL:     : https://github.com/microsoft/semantic-kernel/blob/main/README.md\n  Title    : README: Installation, getting started, and how to contribute\n  Relevance: 0.8084409058091631\n</code></pre> <p>It has found the three GitHub URLs that are most relevant to the query string, and returned them in order of relevance. The first result is the Jupyter notebook that describes how to get started with the Semantic Kernel, which is exactly what I was looking for.</p>","tags":["web development","dotnet","c#","AI","GPT","SK","semantic kernel"]},{"location":"web-dev/dotnet-csharp/semantic-kernel-vector-database/#conclusion","title":"Conclusion","text":"<p>This is only an incredibly basic example, but hopefully it's enough to get you kick-started with the Semantic Kernel and using a vector database to store embeddings. </p>","tags":["web development","dotnet","c#","AI","GPT","SK","semantic kernel"]},{"location":"web-dev/js/node-js-typescript/","title":"How to set up a Node.js Typescript project well","text":"<p>2022-11-19</p> <p>This post covers how to set up a project with Node.js and Typescript using eslint and vscode devcontainers using Docker.</p> <p>This project makes use of eslint with airbnb styling and typescript support.</p> <p>All linting errors can be automatically fixed (usually) by running <code>npm run fix</code> before you commit your changes.</p>","tags":["development","nodejs","typescript","eslint","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-typescript/#getting-it-running-locally-for-development","title":"Getting it running locally for development","text":"","tags":["development","nodejs","typescript","eslint","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-typescript/#devcontainer-vscode","title":"Devcontainer + vscode","text":"<p>If you have <code>Visual Studio Code</code> and <code>Docker</code> installed then you can open this project using the <code>Remote - Containers</code> extension (ms-vscode-remote.remote-containers). Click the button at the bottom left of vscode, then 'Reopen in container'. You can now run the application using the vscode debugger.</p>","tags":["development","nodejs","typescript","eslint","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-typescript/#standard-local-install","title":"Standard local install","text":"<p>Running <code>npm run dev</code> will build the development version in the <code>dist</code> folder and enable hot reloading. It will also open the application in a new browser tab automatically.</p> <p>Running <code>npm run build</code> will create the production assets in the <code>dist</code> folder.</p>","tags":["development","nodejs","typescript","eslint","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-typescript/#devcontainer","title":"Devcontainer","text":"<p>In the <code>.devcontainer</code> folder you'll need three files.</p>","tags":["development","nodejs","typescript","eslint","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-typescript/#basedockerfile","title":"<code>base.Dockerfile</code>","text":"<pre><code># [Choice] Node.js version (use -bullseye variants on local arm64/Apple Silicon): 18, 16, 14, 18-bullseye, 16-bullseye, 14-bullseye, 18-buster, 16-buster, 14-buster\nARG VARIANT=16-bullseye\nFROM mcr.microsoft.com/vscode/devcontainers/javascript-node:0-${VARIANT}\n\n# Install tslint, typescript. eslint is installed by javascript image\nARG NODE_MODULES=\"tslint-to-eslint-config typescript\"\nCOPY library-scripts/meta.env /usr/local/etc/vscode-dev-containers\nRUN su node -c \"umask 0002 &amp;&amp; npm install -g ${NODE_MODULES}\" \\\n    &amp;&amp; npm cache clean --force &gt; /dev/null 2&gt;&amp;1\n\n# [Optional] Uncomment this section to install additional OS packages.\n# RUN apt-get update &amp;&amp; export DEBIAN_FRONTEND=noninteractive \\\n#     &amp;&amp; apt-get -y install --no-install-recommends &lt;your-package-list-here&gt;\n\n# [Optional] Uncomment if you want to install an additional version of node using nvm\n# ARG EXTRA_NODE_VERSION=10\n# RUN su node -c \"source /usr/local/share/nvm/nvm.sh &amp;&amp; nvm install ${EXTRA_NODE_VERSION}\"\n</code></pre>","tags":["development","nodejs","typescript","eslint","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-typescript/#devcontainerjson","title":"<code>devcontainer.json</code>","text":"<pre><code>// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:\n// https://github.com/microsoft/vscode-dev-containers/tree/v0.238.0/containers/typescript-node\n{\n    \"name\": \"Node.js &amp; TypeScript\",\n    \"build\": {\n        \"dockerfile\": \"Dockerfile\",\n        // Update 'VARIANT' to pick a Node version: 18, 16, 14.\n        // Append -bullseye or -buster to pin to an OS version.\n        // Use -bullseye variants on local on arm64/Apple Silicon.\n        \"args\": { \n            \"VARIANT\": \"16-bullseye\"\n        }\n    },\n\n    // Configure tool-specific properties.\n    \"customizations\": {\n        // Configure properties specific to VS Code.\n        \"vscode\": {\n            // Add the IDs of extensions you want installed when the container is created.\n            \"extensions\": [\n                \"dbaeumer.vscode-eslint\"\n            ]\n        }\n    },\n\n    // Use 'forwardPorts' to make a list of ports inside the container available locally.\n    \"forwardPorts\": [3000],\n\n    // Use 'postCreateCommand' to run commands after the container is created.\n    \"postCreateCommand\": \"npm install\",\n\n    // Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.\n    \"remoteUser\": \"node\"\n}\n</code></pre>","tags":["development","nodejs","typescript","eslint","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-typescript/#dockerfile","title":"<code>Dockerfile</code>","text":"<p>One important thing to note in this file is the installation of <code>inotify-tools</code> to the underlying Debian OS, as this is required for nodemon to be able to poll and do cold reloading.</p> <pre><code># [Choice] Node.js version (use -bullseye variants on local arm64/Apple Silicon): 18, 16, 14, 18-bullseye, 16-bullseye, 14-bullseye, 18-buster, 16-buster, 14-buster\nARG VARIANT=16-bullseye\nFROM mcr.microsoft.com/vscode/devcontainers/typescript-node:0-${VARIANT}\n\n# [Optional] Uncomment this section to install additional OS packages.\nRUN apt-get update &amp;&amp; export DEBIAN_FRONTEND=noninteractive \\\n    &amp;&amp; apt-get -y install --no-install-recommends inotify-tools\n\n# [Optional] Uncomment if you want to install an additional version of node using nvm\n# ARG EXTRA_NODE_VERSION=10\n# RUN su node -c \"source /usr/local/share/nvm/nvm.sh &amp;&amp; nvm install ${EXTRA_NODE_VERSION}\"\n\n# [Optional] Uncomment if you want to install more global node packages\n# RUN su node -c \"npm install -g &lt;your-package-list -here&gt;\"\n</code></pre>","tags":["development","nodejs","typescript","eslint","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-typescript/#visual-studio-code-debugging","title":"Visual Studio Code Debugging","text":"<p>Within the <code>.vscode</code> folder you'll need to create this <code>launch.json</code> file:</p> <pre><code>{\n    \"version\": \"0.2.0\",\n    \"configurations\": [\n        {\n            \"name\": \"Launch via NPM\",\n            \"request\": \"launch\",\n            \"runtimeArgs\": [\n                \"run\",\n                \"dev\"\n            ],\n            \"runtimeExecutable\": \"npm\",\n            \"skipFiles\": [\n                \"&lt;node_internals&gt;/**\"\n            ],\n            \"type\": \"pwa-node\",\n            \"stopOnEntry\": true,\n        }\n    ]\n}\n</code></pre>","tags":["development","nodejs","typescript","eslint","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-typescript/#eslint","title":"Eslint","text":"<p>Create the following files:</p>","tags":["development","nodejs","typescript","eslint","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-typescript/#eslintignore","title":"<code>.eslintignore</code>","text":"<p>Assuming you're using git, this will be the same as your <code>.gitignore</code> file.</p> <pre><code>dist\nnode_modules\n</code></pre>","tags":["development","nodejs","typescript","eslint","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-typescript/#eslintrc","title":"<code>.eslintrc</code>","text":"<pre><code>{\n    \"root\": true,\n    \"parser\": \"@typescript-eslint/parser\",\n    \"parserOptions\": {\n        \"ecmaVersion\": \"latest\",\n        \"sourceType\": \"module\"\n    },\n    \"env\": {\n        \"browser\": true,\n        \"node\": true\n    },\n    \"plugins\": [\n        \"@typescript-eslint\"\n    ],\n    \"extends\": [\n        \"airbnb-base\",\n        \"eslint:recommended\",\n        \"plugin:@typescript-eslint/eslint-recommended\",\n        \"plugin:@typescript-eslint/recommended\",\n        \"plugin:import/errors\",\n        \"plugin:import/warnings\",\n        \"plugin:import/typescript\"\n    ],\n    \"rules\": {\n        \"import/extensions\": [\n            \"error\",\n            \"ignorePackages\",\n            {\n              \"js\": \"never\",\n              \"jsx\": \"never\",\n              \"ts\": \"never\",\n              \"tsx\": \"never\"\n            }\n        ]\n    }\n}\n</code></pre>","tags":["development","nodejs","typescript","eslint","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-typescript/#ts-support-plus-cold-reloading","title":"TS support, plus cold reloading","text":"<p>The structure for this application is to have a <code>src</code> folder that contains all <code>.ts</code> files. The files will be compiled to a <code>dist</code> folder for deploying to a server.</p> <p>Create the following files:</p>","tags":["development","nodejs","typescript","eslint","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-typescript/#nodemonjson","title":"<code>nodemon.json</code>","text":"<pre><code>{\n    \"watch\": [\"src\"],\n    \"ext\": \".ts,.js\",\n    \"ignore\": [],\n    \"exec\": \"ts-node .src/index.ts\"\n}\n</code></pre>","tags":["development","nodejs","typescript","eslint","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-typescript/#tsconfigjson","title":"<code>tsconfig.json</code>","text":"<pre><code>{\n    \"compilerOptions\": {\n      \"target\": \"es2016\",\n      \"lib\": [\"es6\"],\n      \"module\": \"commonjs\",\n      \"rootDir\": \"src\",\n      \"resolveJsonModule\": true,\n      \"allowJs\": false,\n      \"outDir\": \"dist\",\n      \"esModuleInterop\": true,\n      \"forceConsistentCasingInFileNames\": true,\n      \"strict\": true,\n      \"noImplicitAny\": true,\n      \"strictNullChecks\": true,\n      \"strictFunctionTypes\": true,\n      \"strictPropertyInitialization\": true,\n      \"noImplicitThis\": true,\n      \"noUnusedLocals\": true,\n      \"noUnusedParameters\": true,\n      \"noImplicitReturns\": true,\n      \"noFallthroughCasesInSwitch\": true,\n      \"noImplicitOverride\": true,\n      \"skipLibCheck\": true\n    }\n}\n</code></pre>","tags":["development","nodejs","typescript","eslint","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-typescript/#packagejson","title":"<code>package.json</code>","text":"<pre><code>{\n  \"name\": \"your app name\",\n  \"type\": \"commonjs\",\n  \"version\": \"1.0.0\",\n  \"description\": \"\",\n  \"scripts\": {\n    \"dev\": \"nodemon --legacy-watch\",\n    \"lint\": \"eslint . --ext .ts\",\n    \"fix\": \"eslint . --ext .ts --fix\",\n    \"build\": \"rimraf ./dist &amp;&amp; tsc\",\n    \"start\": \"npm run build &amp;&amp; node dist/index.js\"\n  },\n  \"repository\": {\n    \"type\": \"git\",\n    \"url\": \"\"\n  },\n  \"keywords\": [],\n  \"author\": \"\",\n  \"license\": \"ISC\",\n  \"devDependencies\": {\n    \"@types/node\": \"^18.11.9\",\n    \"@typescript-eslint/eslint-plugin\": \"^5.43.0\",\n    \"@typescript-eslint/parser\": \"^5.43.0\",\n    \"eslint\": \"^8.28.0\",\n    \"eslint-config-airbnb-base\": \"^15.0.0\",\n    \"eslint-plugin-import\": \"^2.26.0\",\n    \"nodemon\": \"^2.0.20\",\n    \"rimraf\": \"^3.0.2\",\n    \"ts-node\": \"10.9.1\",\n    \"typescript\": \"^4.9.3\"\n  }\n}\n</code></pre>","tags":["development","nodejs","typescript","eslint","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-web-typescript/","title":"How to set up a Node.js Typescript web project well","text":"<p>2022-06-15</p> <p>This post covers how to set up a project with Node.js and Typescript using eslint, webpack, babel, source maps, SCSS and vscode devcontainers using Docker.</p> <p>This project makes use of eslint with airbnb styling and typescript support.</p> <p>All linting errors can be automatically fixed (usually) by running <code>npm run lint -- --fix</code> before you commit your changes.</p>","tags":["web development","nodejs","typescript","eslint","webpack","babel","sourcemaps","scss","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-web-typescript/#getting-it-running-locally-for-development","title":"Getting it running locally for development","text":"","tags":["web development","nodejs","typescript","eslint","webpack","babel","sourcemaps","scss","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-web-typescript/#devcontainer-vscode","title":"Devcontainer + vscode","text":"<p>If you have <code>Visual Studio Code</code> and <code>Docker</code> installed then you can open this project using the <code>Remote - Containers</code> extension (ms-vscode-remote.remote-containers). Click the button at the bottom left of vscode, then 'Reopen in container'. You can now run the application using the vscode debugger.</p>","tags":["web development","nodejs","typescript","eslint","webpack","babel","sourcemaps","scss","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-web-typescript/#standard-local-install","title":"Standard local install","text":"<p>Running <code>npm run dev</code> will build the development version in the <code>dist</code> folder and enable hot reloading. It will also open the application in a new browser tab automatically.</p> <p>Running <code>npm run build</code> will create the production assets in the <code>dist</code> folder.</p>","tags":["web development","nodejs","typescript","eslint","webpack","babel","sourcemaps","scss","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-web-typescript/#devcontainer","title":"Devcontainer","text":"<p>In the <code>.devcontainer</code> folder you'll need three files.</p>","tags":["web development","nodejs","typescript","eslint","webpack","babel","sourcemaps","scss","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-web-typescript/#basedockerfile","title":"<code>base.Dockerfile</code>","text":"<pre><code># [Choice] Node.js version (use -bullseye variants on local arm64/Apple Silicon): 18, 16, 14, 18-bullseye, 16-bullseye, 14-bullseye, 18-buster, 16-buster, 14-buster\nARG VARIANT=16-bullseye\nFROM mcr.microsoft.com/vscode/devcontainers/javascript-node:0-${VARIANT}\n\n# Install tslint, typescript. eslint is installed by javascript image\nARG NODE_MODULES=\"tslint-to-eslint-config typescript\"\nCOPY library-scripts/meta.env /usr/local/etc/vscode-dev-containers\nRUN su node -c \"umask 0002 &amp;&amp; npm install -g ${NODE_MODULES}\" \\\n    &amp;&amp; npm cache clean --force &gt; /dev/null 2&gt;&amp;1\n\n# [Optional] Uncomment this section to install additional OS packages.\n# RUN apt-get update &amp;&amp; export DEBIAN_FRONTEND=noninteractive \\\n#     &amp;&amp; apt-get -y install --no-install-recommends &lt;your-package-list-here&gt;\n\n# [Optional] Uncomment if you want to install an additional version of node using nvm\n# ARG EXTRA_NODE_VERSION=10\n# RUN su node -c \"source /usr/local/share/nvm/nvm.sh &amp;&amp; nvm install ${EXTRA_NODE_VERSION}\"\n</code></pre>","tags":["web development","nodejs","typescript","eslint","webpack","babel","sourcemaps","scss","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-web-typescript/#devcontainerjson","title":"<code>devcontainer.json</code>","text":"<pre><code>// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:\n// https://github.com/microsoft/vscode-dev-containers/tree/v0.238.0/containers/typescript-node\n{\n    \"name\": \"Node.js &amp; TypeScript\",\n    \"build\": {\n        \"dockerfile\": \"Dockerfile\",\n        // Update 'VARIANT' to pick a Node version: 18, 16, 14.\n        // Append -bullseye or -buster to pin to an OS version.\n        // Use -bullseye variants on local on arm64/Apple Silicon.\n        \"args\": { \n            \"VARIANT\": \"16-bullseye\"\n        }\n    },\n\n    // Configure tool-specific properties.\n    \"customizations\": {\n        // Configure properties specific to VS Code.\n        \"vscode\": {\n            // Add the IDs of extensions you want installed when the container is created.\n            \"extensions\": [\n                \"dbaeumer.vscode-eslint\"\n            ]\n        }\n    },\n\n    // Use 'forwardPorts' to make a list of ports inside the container available locally.\n    \"forwardPorts\": [3000],\n\n    // Use 'postCreateCommand' to run commands after the container is created.\n    \"postCreateCommand\": \"npm install\",\n\n    // Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.\n    \"remoteUser\": \"node\"\n}\n</code></pre>","tags":["web development","nodejs","typescript","eslint","webpack","babel","sourcemaps","scss","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-web-typescript/#dockerfile","title":"<code>Dockerfile</code>","text":"<p>One important thing to note in this file is the installation of <code>inotify-tools</code> to the underlying Debian OS, as this is required for webpack to be able to poll and do hot reloading.</p> <pre><code># [Choice] Node.js version (use -bullseye variants on local arm64/Apple Silicon): 18, 16, 14, 18-bullseye, 16-bullseye, 14-bullseye, 18-buster, 16-buster, 14-buster\nARG VARIANT=16-bullseye\nFROM mcr.microsoft.com/vscode/devcontainers/typescript-node:0-${VARIANT}\n\n# [Optional] Uncomment this section to install additional OS packages.\nRUN apt-get update &amp;&amp; export DEBIAN_FRONTEND=noninteractive \\\n    &amp;&amp; apt-get -y install --no-install-recommends inotify-tools\n\n# [Optional] Uncomment if you want to install an additional version of node using nvm\n# ARG EXTRA_NODE_VERSION=10\n# RUN su node -c \"source /usr/local/share/nvm/nvm.sh &amp;&amp; nvm install ${EXTRA_NODE_VERSION}\"\n\n# [Optional] Uncomment if you want to install more global node packages\n# RUN su node -c \"npm install -g &lt;your-package-list -here&gt;\"\n</code></pre>","tags":["web development","nodejs","typescript","eslint","webpack","babel","sourcemaps","scss","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-web-typescript/#visual-studio-code-debugging","title":"Visual Studio Code Debugging","text":"<p>Within the <code>.vscode</code> folder you'll need to create this <code>launch.json</code> file:</p> <pre><code>{\n    \"version\": \"0.2.0\",\n    \"configurations\": [\n        {\n            \"name\": \"Launch via NPM\",\n            \"request\": \"launch\",\n            \"runtimeArgs\": [\n                \"run\",\n                \"dev\"\n            ],\n            \"runtimeExecutable\": \"npm\",\n            \"skipFiles\": [\n                \"&lt;node_internals&gt;/**\"\n            ],\n            \"type\": \"pwa-node\",\n            \"stopOnEntry\": true,\n        }\n    ]\n}\n</code></pre>","tags":["web development","nodejs","typescript","eslint","webpack","babel","sourcemaps","scss","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-web-typescript/#eslint","title":"Eslint","text":"<p>Create the following files:</p>","tags":["web development","nodejs","typescript","eslint","webpack","babel","sourcemaps","scss","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-web-typescript/#eslintignore","title":"<code>.eslintignore</code>","text":"<p>Assuming you're using git, this will be the same as your <code>.gitignore</code> file.</p> <pre><code>dist\nnode_modules\n</code></pre>","tags":["web development","nodejs","typescript","eslint","webpack","babel","sourcemaps","scss","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-web-typescript/#eslintrcjson","title":"<code>.eslintrc.json</code>","text":"<pre><code>{\n    \"env\": {\n        \"browser\": true,\n        \"es2021\": true\n    },\n    \"extends\": [\n        \"airbnb-base\",\n        \"plugin:import/errors\",\n        \"plugin:import/warnings\",\n        \"plugin:import/typescript\"\n    ],\n    \"parser\": \"@typescript-eslint/parser\",\n    \"parserOptions\": {\n        \"ecmaVersion\": \"latest\",\n        \"sourceType\": \"module\"\n    },\n    \"plugins\": [\n        \"@typescript-eslint\"\n    ],\n    \"rules\": {\n        \"import/extensions\": [\n            \"error\",\n            \"ignorePackages\",\n            {\n              \"js\": \"never\",\n              \"jsx\": \"never\",\n              \"ts\": \"never\",\n              \"tsx\": \"never\"\n            }\n        ]\n    }\n}\n</code></pre>","tags":["web development","nodejs","typescript","eslint","webpack","babel","sourcemaps","scss","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-web-typescript/#webpack-with-scss-and-ts-support-plus-hot-reloading","title":"Webpack with SCSS and TS support, plus hot reloading","text":"<p>The structure for this application is to have a <code>src</code> folder that contains: all <code>.html</code> and <code>.ts</code>; <code>assets</code> folder for images; <code>styles</code> folder for <code>.scss</code> files. The files will be compiled to a <code>dist</code> folder for deploying to a web server.</p> <p>Create the following files:</p>","tags":["web development","nodejs","typescript","eslint","webpack","babel","sourcemaps","scss","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-web-typescript/#customdts","title":"<code>custom.d.ts</code>","text":"<pre><code>declare module '*.svg' {\n    const content: any;\n    export default content;\n}\n</code></pre>","tags":["web development","nodejs","typescript","eslint","webpack","babel","sourcemaps","scss","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-web-typescript/#tsconfigjson","title":"<code>tsconfig.json</code>","text":"<pre><code>{\n  \"compilerOptions\": {\n    \"preserveConstEnums\": true\n  },\n  \"include\": [\"src/**/*\", \"src/custom.d.ts\"],\n  \"exclude\": [\"node_modules\", \"**/*.spec.ts\"]\n}\n</code></pre>","tags":["web development","nodejs","typescript","eslint","webpack","babel","sourcemaps","scss","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-web-typescript/#webpackconfigjs","title":"<code>webpack.config.js</code>","text":"<pre><code>const path = require('path');\n\nmodule.exports = {\n  mode: 'development',\n  entry: {\n    bundle: path.resolve(__dirname, 'src/index.ts'),\n  },\n  output: {\n    path: path.resolve(__dirname, 'dist'),\n    filename: '[name].[contenthash].js',\n    clean: true,\n    assetModuleFilename: '[name].[contenthash][ext]',\n  },\n  devtool: 'source-map',\n  devServer: {\n    static: {\n      directory: path.resolve(__dirname, 'dist'),\n    },\n    port: 3000,\n    open: true,\n    hot: true,\n    compress: true,\n    historyApiFallback: true,\n  },\n  watchOptions: { poll: true },\n  module: {\n    rules: [\n      {\n        // Use these loaders for any matching scss file types\n        test: /\\.scss$/,\n        use: [\n          'style-loader',\n          'css-loader',\n          'sass-loader',\n        ],\n      },\n      {\n        // Add backwards compatibility\n        test: /\\.(js|ts)$/,\n        exclude: /node_modules/,\n        use: {\n          loader: 'babel-loader',\n          options: {\n            presets: ['@babel/preset-env', '@babel/preset-typescript'],\n          },\n        },\n      },\n      {\n        // Add support for images\n        test: /\\.(png|svg|jpg|jpeg|gif)$/i,\n        type: 'asset/resource',\n      },\n    ],\n  },\n  resolve: {\n    // Specify the order in which to resolve files by their extension\n    extensions: ['*', '.js', '.ts'],\n  },\n};\n</code></pre>","tags":["web development","nodejs","typescript","eslint","webpack","babel","sourcemaps","scss","vscode","devcontainer","docker"]},{"location":"web-dev/js/node-js-web-typescript/#packagejson","title":"<code>package.json</code>","text":"<pre><code>{\n  \"name\": \"your app name\",\n  \"version\": \"1.0.0\",\n  \"description\": \"\",\n  \"main\": \"index.js\",\n  \"scripts\": {\n    \"build\": \"webpack\",\n    \"dev\": \"webpack serve\",\n    \"lint\": \"eslint --ignore-path .eslintignore --ext .js,.ts .\"\n  },\n  \"repository\": {\n    \"type\": \"git\",\n    \"url\": \"\"\n  },\n  \"keywords\": [],\n  \"author\": \"\",\n  \"license\": \"ISC\",\n  \"devDependencies\": {\n    \"@babel/core\": \"^7.18.5\",\n    \"@babel/preset-env\": \"^7.18.2\",\n    \"@babel/preset-typescript\": \"^7.17.12\",\n    \"@typescript-eslint/eslint-plugin\": \"^5.28.0\",\n    \"@typescript-eslint/parser\": \"^5.28.0\",\n    \"babel-loader\": \"^8.2.5\",\n    \"css-loader\": \"^6.7.1\",\n    \"eslint\": \"^8.17.0\",\n    \"eslint-config-airbnb-base\": \"^15.0.0\",\n    \"eslint-plugin-import\": \"^2.26.0\",\n    \"html-webpack-plugin\": \"^5.5.0\",\n    \"sass\": \"^1.52.3\",\n    \"sass-loader\": \"^13.0.0\",\n    \"style-loader\": \"^3.3.1\",\n    \"typescript\": \"^4.7.3\",\n    \"webpack\": \"^5.73.0\",\n    \"webpack-cli\": \"^4.10.0\",\n    \"webpack-dev-server\": \"^4.9.2\"\n  }\n}\n</code></pre>","tags":["web development","nodejs","typescript","eslint","webpack","babel","sourcemaps","scss","vscode","devcontainer","docker"]}]}