Version History and Background

Brian McCrory
Jul 27, 2023
Share

The history of this website and newsletter since 2018

Version History

2018 Jan - 2021 May: “J Jazz: Modern Jazz From Japan”

  • Initial website v1.0 with 150+ Album posts

  • Ran at jjazzist.com using WordPress/AWS/Google App Engine

2021 May - 2022 Jan: “J Jazz Substack”

  • Website v2.0 & newsletter with 200+ Album posts and emails

  • Ran at jjazz.substack.com using Substack

2022 Jan - 2023 May: “Jazz of Japan”

  • Website v3.0 with 300+ Album, Clubs, and Preview posts

  • Ran at jazzofjapan.com using GitHub/Jekyll

2023 May - present: “Jazz of Japan”

  • Website v4.0 & newsletter with optional membership

  • Running at jazzofjapan.com using Substack

The rest of this article discusses the reasons for and technical details about moving from WordPress to Substack, to GitHub, and back to Substack.

WordPress, Substack, and GitHub

Notes and Notebooks (2005)

This online project was first published online in January 2018, but I began writing notes on this topic after visiting Japanese jazz clubs and buying Japanese jazz albums for the first time in 2005. Those dozen or so notebooks practically make up version 0.1 of this project, and I continue to take handwritten notes to this day.

Starting with WordPress (January 2018)

It was easy to set up WordPress on a laptop and start writing posts. I published the site online using the custom domain jjazzist.com (no longer in use) with AWS and Google App Engine at various times. The WordPress system made it easy to switch layouts, write draft posts, and experiment and test changes on a laptop before updating the website online. However, over time I found that I stuck to a fairly simple design and didn’t make use of many WordPress plugins, except for functionality like media playlists, analytics, and backup tools.

WordPress started to become a hassle with frequent updates of plugins and occasional technical glitches. In addition to running on a laptop for drafting posts and testing, it also required installing, running, and maintaining the WordPress system on a hosted server. Over time, this became more effort than it was worth, and I wanted to concentrate more on writing and less on system and website maintenance. By this time I had written over a hundred posts including hundreds of image and audio files, and I wanted to make sure that all of this work was secure and stable.

Moving from WordPress to Substack (May 2021)

Substack offered an attractive solution with a nice, simple design that focused on newsletter functionality with a web archive. The writer dashboard was also refreshingly simple and clean, a great interface to focus on writing and organizing posts. While I didn’t initially have the goal of starting a newsletter, I thought I would give it a try. But the main attraction for me at this time was to find a way to use the Substack web archive as a replacement for the WordPress posts I had been writing. The system seemed simple, clean, and included extra benefits like an archive of posts and search functionality, all of which I got from Substack without having to maintain any systems myself.

I used the Substack migration tool with some manual finessing to import all the posts from WordPress to Substack. I published the new site on jjazz.substack.com (no longer in use) and began to send newsletters and experiment with Substack’s features. After importing, cleaning up, and organizing all of my previous posts, I began to send out new newsletter emails for “Album” articles on a semi-regular basis.

Moving from Substack to GitHub (January 2022)

Using Substack was great for free-form posts, but there was still some extra functionality that I was looking for. Now that I had 200+ articles (and more in drafts), I wanted to better organize my text, musician data, images, links, and audio files in a database, or some sort of flexible data format for fine-grained control of the data and presentation.

Since most of my posts are about albums, I was using a regular layout, a template with a specific format and well-defined sections:

  • A main body section (album description)

  • Images of the album

  • Musician names, instruments, names in Japanese, and website links

  • Video and audio links

I wanted to be able to validate that certain details were correctly displayed and consistent across all the articles. Specifically, musician names should be shown using the same spelling and related information on any pages on which they were mentioned. This would also provide an easy way to link musicians from any one album to other albums that they participated on.

What I needed was a database with a musician key or unique identifier having certain data specified only once for each musician entry: For example, musician_key → {“English name”, “Japanese name”, instrument, website_url} at a minimum.

At the same time, I wanted to be able to override certain detail from page to page as necessary. For example, a musician’s primary instrument may be “piano” but on a specific album they are listed as playing “Hammond B3”. I wanted to show the musician’s default instrument generally but override that easily when necessary.

Also, using this data, I wanted to automatically generate an index page with musicians’ names in English and Japanese, with instruments, website links, and a list of the albums they played on.

Importantly, with each new album post that I published, I wanted the index to be automatically updated using the latest data from the new articles, driven and constrained by the underlying data. I also wanted to have flexibility in the design of the generated files as tables or lists, with thumbnail images, or anything else.

In other words, I wanted to:

  • define a custom data structure with a persistent database or version-controlled data files

  • use the data to generate indexes and to perform other analyses and validation of the data used throughout the articles

GitHub provided a good fit with a versioning system integrated with GitHub Pages, a static website publishing system with a Jekyll/Liquid templating system. One main benefit was a flexible markdown file structure which allowed me to define the data structure I wanted for each “Album” article. I used a program to import all of my posts in Substack to generate my first GitHub Pages site this way. It was also easy to link my custom domain jazzofjapan.com to the GitHub Pages site.

In addition to the simple markdown file format, Jekyll also provided support for data files. I used some CSV files to maintain a database of musicians and other data, which ended up being a very straightforward and convenient solution to the data problem. Being in stored in GitHub, all of this data and the markdown files were easily versioned and regularly backed up.

The combination of data stored in structured markdown files and CSV data also made it easy for me to supplement the Jekyll website with custom layouts (for the auto-generated Musicians index page, for instance) and other programs for data validation and analysis.

One limitation of GitHub Pages was, to make the site available to the public, the repository needed to be public. This meant that all of the data and code would be available for anyone to clone or copy very easily. Fortunately, I found a way to protect the content in a private GitHub repo, while still being able to publish the site to the public repo using a custom GitHub Actions script.

With this, posts were written and tested using markdown, Jekyll. and some programming on a laptop, then committed and pushed to GitHub online, then published using GitHub Pages with my custom domain. It was a simple and convenient all-in-one solution.

Moving back to Substack (May 2023)

Now that the entire system was stored in GitHub (posts, images, audio, layouts, and programming), I returned to Substack to reintroduce the newsletter functionality and update the website’s look and feel. While GitHub Pages was functional and straightforward, it had some limitations, and I was being drawn back to Substack’s ease of use, clean interface, and tools. I imported all posts from GitHub pages to Substack in May 2023 and continued from there, with one important clarification: While using Substack for the newsletter and website, I would continue to maintain markdown and datafiles in GitHub for certain background tasks, in effect maintaining two systems.

The current v4.0 website has public posts, subscriptions, and newsletter emails all handled by Substack, while the data structure, tooling (data analysis and validation), and automatic content generation are handled using GitHub data files which mirror the Substack posts, with some additional programming.

This hybrid solution means that I use Substack for the elegant website design, smooth writing tools, and newsletter functionality, and I use GitHub as a data repository for index file generation and other utilities. The Jekyll markdown format also makes it easy to cut and paste the data-driven sections (Musicians, Audio and Video, etc) into the articles like using a consistent template.

It’s also reassuring that GitHub serves as a secondary source backup of all my content, should anything disastrous happen.

Finally, something I’ve noticed that may be the biggest benefit of Substack is that it’s such a convenient place to write. The interface is clean and welcoming, the tools are simple and pared down, and there is a wealth of support and community. Most importantly, the system itself just seems to encourage you to write, much more than the WordPress editor or utilitarian GitHub markdown files do.