Details on website migration and plans for the future

With my last post I promised to give some more detailed information on the migration of the crowbyte website and blog as well as my plans for the future of crowbyte.org and I live up with my promise in this post.

Migration details

After over half a year of absence I had the time to rethink the direction into which the blog and website headed lately.

I used Nikola as a page generator for my blog and websites. This was a decision which I made after a lot of consideration and testing of available systems. In the end it was Nikola which best fitted my requirements: it offers a good performance and all features I wanted. This is why I enjoyed Nikola a lot. All you needed was a simple text editor for writing posts and pages. But after some time I had to realise that this concept has a few shortcomings as well.

The Nikola paradox

Ironically one of Nikola's strongest points is at the same time one of it's weakest points: pregenerated HTML files.

The benefit of serving pregenerated HTML is: HTML files are served lightning fast by the webserver and you have no PHP code running which might be vulnerable to any attacks. But the downside is a logical side effect: it means as well that you have to generate HTML files each and every time you make changes to the markup language files which are used to generate the HTML files from. And the use of markup language files means, your pages or posts are rendered with big and unpleasant error messages as soon as you have a syntax error in your markup file.

So in the end, publishing a post or page was a very easy task on the one hand side, meaning: Open a new text file, write your article or whatever in your preferred markup language, save the file, copy it to your server, connect to the server, generate the HTML files with a simple nikola build, publish your posts and pages by coping them into your webserver's document root folder. Done.

But this means as well that changing a few little typos or fixing some markup syntax errors boils down to the exact same procedure: Open the text file, fix the typos and syntax errors, save the file, copy the file to your server, connect to your server, generate the HTML files again, publish them by coping them into your webserver's document root folder - and you are finally done.

So, as you can see there are a lot of steps to do to only fix a few little mistakes. Additionally, even though Nikola is smart and only generates HTML files for changed markup files, Nikola nonetheless takes some time to generate your pages. In the end you spend quite some time for a simple task, repeating simple steps which sum up to a bit of overhead for something so simple as of fixing a few small typos and alike.

This overhead took some of the fun out of publishing new posts and slowed down my writing of posts although I intended to ease the publishing process with Nikola to become more productive instead of less productive. Therefore I made the decision to testdrive October CMS again which was the closest contender to Nikola that time. And as it has matured and now finally offeres multi language pages I decided to use it to run crowbyte.org with it.

Concept for the new website

The migration gave me the opportunity to decide what to migrate and what to change and to adjust the concept for the website and blog as well.

First of all it is needless to say that all posts needed to be migated over from Nikola to October CMS. I planned to keep the structure for the blog post calls (www.crowbyte.org/posts/.. and www.crowbyte.org/de/posts/..) so that links to older posts remain functional. But in the end I decided to convert this into a more standardized form as of www.crowbyte.org/blog/post/.. and www.crowbyte.org/de/blog/post/... This means breaking old links but offering a bit more logical structure as the blog is intended to be only a part of the crowbyte.org website.

Furthermore I planned to add a few more features for the blog than I had available with Nikola. Besides the standard of categories and tags I wanted to feature related articles and share links for the most common social networks. As a nice to have addition I would like to add a search function as well, which is not yet available but intended to follow soon.

Although I already added and will add even a few more features in the future, I got rid of one feature for this time: the comments section. The reason is simple. Nikola had no comments function and to add this to the blog I had to use an external service like Disqus or, as I did, IntenseDebate. Not only that I want to avoid external services this time I made the experience that this features remained almost completely unused. Most of the discussion of posts happened in the social networks. And I would like to keep discussions there because I think that social networks are the right place for it.

Last but not least I wanted to reduce the top bar menu to only a few relevant menu entries, make a redesign of the whole website to have it look a bit more modern and beautiful and have it using HTTPS to offer a secure connection.

Software, plugins and tools

I decided to go with the following software, plugins and tools:

Implementation

After getting the server and finishing the installation of Arch Linux I added a normal user, restricted SSH to deny access for the root user, added my SSH public key, did a system update and installed Hiawatha, PHP with a few extensions, MariaDB and Let's Encrypts certbot from the pacman package repositories. If you need more details on these steps, contact me or use google to find a HowTo as there are plenty available on this already.

MariaDB can easily be set up by running

mysql_secure_installation

and following the instructions.

I then configured Hiawatha and the virtual host for October CMS. It should look like:

VirtualHost { 
        Hostname = domain.tld
        WebsiteRoot = /path/to/website/root/domain.tld/
        StartFile = index.php
        TLScertFile = /path/to/tls_cert/domain.pem
        RequireTLS = yes
        ExecuteCGI = yes
        TimeForCGI = 5
        UseToolkit = october
}

Additionally you need to uncomment the CGIhandler = /bin/php-cgi:php entry to make PHP work with Hiawatha and your virtual host.

And as you can see it uses a URL Toolkit which is not yet available.

Therefore I added the rewrite rules for October CMS to the config file. The rules can be easily found on the Hiawatha website. Hit Support, navigate to HOWTO and select 10. URL rewrite rules for CMSes, wiki's, webmails, etc and then scroll down until you find the October CMS entry. Unfortunately there is a mistake in it as RequestURI file Return must be RequestURI isfile Return.

The working URL Toolkit looks like this:

UrlToolkit {
        ToolkitID = october
        Match /themes/.*/(layouts|pages|partials)/.*.htm Rewrite /index.php
        Match /uploads/protected/.* Rewrite /index.php
        RequestURI isfile Return
        Match .* Rewrite /index.php
}

Now we are almost ready to go. I only had to get the certificate. For this I used certbot. Be aware that certbot needs your DNS entries being configured correctly. This means you should have a DNS A record in your DNS pointing to the IP address of your server where Hiawatha is supposed to be running.

If this is true for you, you can run certbot to obtain a certificate with

sudo certbot certonly --standalone -d domain.tld

Certbot tells you where it saves your certificates and whether you received them successfully or not. You will find them in /etc/letsencrypt/live/domain.tld. As this folder is protected only root has access to it. Change into your root account with sudo su.

As Hiawatha expects the certificate to be of a quite special form I had to generate it to be useable for Hiawatha.

I generated the needed certificate by running

cat privkey.pem cert.pem chain.pem > /path/to/tls_cert/domain.pem

You can save the certificate where ever it suits you best, but for security reasons is should never be in a folder which can be globally read or even accessed from the internet.

Dont forget to change the access permissions:

chmod 400 /path/to/tls_cert/domain.pem

Now everything should be up and running. Time to start the Hiawatha webserver:

sudo systemctl start hiawatha

If it starts correctly everything is fine. If you configured something wrong Hiawatha will tell. You can see what went wrong with sudo systemctl status hiawatha.

All that was left was downloading October CMS, extract the archive into the configured document root for the virtual host and install it as described on the October CMS documentation.

Migrating the content

As there is no database behind Nikola but only plain text files the migration over to October was a bit exhausting. October's blog plugin has an import function for blog posts but needs a CSV file. I could have aggregated the content and other values for the posts in a CSV file and used that for the import but decided to do everything manually. I used this opportunity to translate a few posts which were not translated yet and read each post to find some of the left typos. As the blog plugin uses Markdown for writing posts I used pandoc to convert the RestructuredText files I used for Nikola into Markdown files and then copied the content of each file into October's New post form.

As I restructured the other pages I rewrote almost each text to fit into the new layout and to be up-to-date again.

Other than the technical part this sounds quite easy but took more time than getting everything up and running, adjusting the CSS, writing the HTML page structures and whatever else had to be done.

Conclusion

This post was intended to give a more detailed look behind the scenes. Actually it has now become much longer and much more detailed than I originally intended.

On the way I had some trouble with adjusting the blog plugin to render the post list as I intended. I found out that it is easier to first follow the video guides for setting up a blog and such and to adjust things afterwards (like replacing the components with partials etc.).

Plans for the future

As already mentioned I plan to add a search function. Additionally I want to fix the only remaining known bug where after changing the language to German blog post teaser texts are still in English. And last but not least I plan to make the contact form work and fill the project page with upcoming or current projects. But as soon as this happens I will update you all with a blog post about the changes and details on the added projects.

Questions?

Do you still have questions to parts of this article or in general? Don't hesitate to contact me via Facebook, Twitter, Diaspora, Google+ or by E-Mail.

Thanks for reading!

pb

Published by

Pierre Böckmann

Born 1985 in Berlin, I am Software Developer, Blogger, Author, Open Source Enthusiast, openSUSE member and passionate openSUSE Leap user.