Paul Johnson

Automating Netlify Redirects in Quarto Using Lua

Paul Johnson — Wed, 04 Jun 2025 00:00:00 GMT

Last year, I wrote a blog post about my process for generating Netlify redirects. It was a similar process to that used by many others, adapted slightly to work with my slightly different setup. It involved housing a couple chunks of code in the homepage (index.qmd in the root directory) that would generate redirects every time the homepage is rendered.

This worked fine. It got the job done. However, it bothered me that I had to remember to re-render the homepage every time I needed to update the redirects file. This relied too heavily on my remembering to do something. So, I’ve finally had a crack at fixing this and setting something up that will run without my intervention.

I’ve set up a Lua script that is called before the whole project is rendered. I figured a brief post outlining what I’ve done and how it works might be helpful to others trying to figure this out, and it should be relatively easy to adapt to your own needs.

The Old (Manual-ish) Setup

I won’t spend too long going over the old approach (you can find it in the blog post I wrote last year). It worked, and I never actually forgot to re-render the homepage to update the redirects. However, I didn’t love having to go through that extra step, and I certainly didn’t trust myself to remember this detail.

Creating dependencies is an unnecessary mess when all I want to do is generate some redirects from a relatively simple website structure.

The Automated Solution

Instead, I wrote a Lua script¹ that is set to run before any Quarto document renders (using Quarto’s pre/post rendering functionality), which generates all my redirects in a file that is stored at the root for Netlify to pick up and use in deployment.

It’s essentially the same process I followed before. Still, by shifting it into a separate script that runs when Netlify renders the site (using the Quarto Netlify plugin), I no longer have to worry about running this before pushing a new post.

I chose Lua because it is already packaged up in Quarto’s installation, as Pandoc uses Lua as its filter language, and Netlify recognises it when deploying the site. That means no extra work is required with a Lua script. Lua is relatively simple, and if you have experience with R and/or Python, it shouldn’t take long to pick it up².

There are three types of redirects handled by this script:

Manual Redirects - These are hand-written rules stored in a _manualredirects file, intended to redirect old pages when the website used a different structure.
Blog Post Redirects - Automatically generated rules that redirect from simple URLs (/blog/my-post-title) to the full dated URLs (/blog/2025-05-26-my-post-title).
Category Redirects - Dynamic redirects that map tag-style URLs (/tags/lua) to filtered blog views (/blog/#category=lua).

Redirects Using Lua

If you want to skip the details, here is the script in full.

Lua Code (Click to Expand)

Testing Transfermarkt’s Squad Market Values

Paul Johnson — Sun, 11 May 2025 00:00:00 GMT

Setup Code (Click to Expand)

How to Write About Data Science (Effectively)

Paul Johnson — Mon, 14 Apr 2025 00:00:00 GMT

Many data scientists will have been kids who excelled at math throughout school. Some would have viewed writing with the fear that the rest of the world views their beloved numbers. I’m not like most data scientists, though. I’m much dumber. My instinct is to express myself using words instead of fancy letters, and as someone who struggled with math in school, I am a little more familiar with the intimidation many feel when faced with notation. I’ve always enjoyed writing, and I think I’m at least pretty good at it for someone in a field full of number nerds. I’m the bridge between the nerds and the normies. I have successfully spun my own stupidity as a niche.

This post is intended to offer some tips for writing about data science, conceding my only competitive advantage in the process. I am not arrogant enough to think I have anything to offer to a general audience¹, but I hope I can give some valuable advice to other data scientists. I will focus on the approach rather than the technicalities of writing about data science. The technical elements of good writing remain consistent across context and domain, so I will defer to the subject matter experts on this².

Everything that follows is neither definitive nor authoritative. It may even be nonsense. You might be right if you think anything I’ve said here is wrong or stupid. This is only intended to share some of what has helped me become a better writer. It’s important to remember I’m just some dumb bozo on the Internet, and whatever you do, don’t be mad at me.

Writing the Right Way

If you’re thinking without writing, you only think you’re thinking.

- Leslie Lamport

I really like this quote³. It’s one of those pithy things you hear people say. They’re usually nonsense, but I think it is true in this case. It’s a sentiment shared by Larry McEnerney in his talk on academic writing at the University of Chicago⁴.

Writing is essential to good science because it helps you organise your thoughts, identify the weaknesses in your argument, and turn disparate, muddled thoughts into something resembling a coherent idea. The same benefits to thinking also apply to learning, which is a big reason for maintaining this blog. I build on what I know when I have to write it down for others. Partly because I don’t want to make a fool of myself but also because I want to do a good job. And if you are going to write? Well, you might as well do it well, right?

Writing is Not an Afterthought

There are plenty of excellent data scientists who might struggle as writers. It’s natural to prioritise the skills you excel at—such as math, statistics, or programming—and when those skills become a career, it becomes a strong positive feedback loop. Writing can quickly become an afterthought.

Even so, communicating your findings shouldn’t be a lower priority than the findings themselves. Instead of treating the writing stage as the means to an end, treat writing as central to everything you do. If you want to improve your writing, afford it the same weight you do to the other parts of the process and treat it as a part of the journey and a part of the fun. A well-written analysis is just as satisfying as the analysis itself, and a scientist has many tools at their disposal for illuminating their writing⁵.

If you care about your writing like you care about your analysis, you’ll invest a similar amount of time iterating over the results to maximise the performance. Zinsser (2001) talks about the time he invests in his writing, pouring over every sentence, reading and rereading passages just to identify and replace the word that makes it feel clunky, and playing with draft after draft so that the end result can be the best version of his work. Data scientists will recognise this process. You apply it to the modelling stage. You should also do it with your writing.

Simple is Smart

William Zinsser (2001) lists “four articles of faith” in On Writing Well: clarity, simplicity, brevity, and humanity⁶. I think these are excellent guiding principles for writing that is accessible and engaging.

First and foremost, the goal should be to distil complex ideas as simply as possible. Data science is complicated. Why make it more difficult by overcomplicating how you communicate about it? The Pulitzer Prize winner Cormac McCarthy recommends “minimalism to achieve clarity” (Savage and Yeh 2019). He advises writers to constantly ask themselves whether words and punctuation marks need to be there, removing anything superfluous. He also suggests that simple, short sentences are effective when communicating about science because they help the reader focus on the key message. Finally, perhaps his best advice relates to theme and structure, recommending a focus on two or three points you want the reader to take away from reading your work⁷.

I think honest intent is vital, too. Write to show the reader something cool, teach people about an interesting method, or play around with a fun new tool. There are lots of good reasons for writing about data science. But don’t write to demonstrate your intelligence. Going in looking to prove how smart you are will encourage you to overcomplicate things. Complexity is a distraction. “Jargon is the enemy of clarity” (Plaxco 2010). Data science is complicated enough, so why make it harder for your readers by taking the scenic route?

There is nothing wrong with wanting to give a good impression. My natural state is the class clown, so I’m more concerned that readers laugh at my dumb jokes than think I’m a clever little boy. But I still don’t want everyone to think I’m talking nonsense. I think it’s important that these goals are secondary, though. Writing about a complex subject with clarity, simplicity, brevity, and humanity will achieve these secondary goals. Readers will be a lot more impressed if you can do this.

So much of this advice ultimately comes down to making your audience a central consideration. Write for yourself, but communicate with your readers. Write about subjects you are passionate about, methods you love, or study questions to which the answer interests you. But when you write, do so with close consideration to who will read your work, and adjust your approach to make it as accessible and engaging as possible. That doesn’t mean that your work can’t be complex. There is plenty of room for deep, complex analyses. But write in simple terms. Write in terms that allow any reader to at least read the words and understand the sentence they just read. If they do not have the necessary prerequisite knowledge to understand what it all means, that’s not on you. But it shouldn’t be your writing that stops someone from following along.

The reality is that data science is not always straightforward, and there are plenty of times when it won’t be possible to write something in simple terms. However, I think the goal should always be to try and simplify what you’re saying without losing its meaning. How far is that? It’s hard to say, and the answer will always vary based on subject, context, and audience. Still, simplicity is generally a good goal for writing, especially when writing about data science.

Practice Makes Perfect Less Bad

The most important step you can take in improving your writing is to write. This blog post forced me to think harder about writing as a craft; otherwise, my writing has improved simply through practice. I’ve spent many years writing for a variety of audiences, including writing a lot of admittedly relatively trivial⁸ articles for the Borussia Dortmund fan site, Fear the Wall, and this has helped me find and hone my voice without giving it much thought. This experience has made it so much easier for me now.

It’s a little cliché, but the best advice for improving your writing is to carve out the time to write regularly. Like any craft, getting better at writing requires doing more of it. The more you do it, the easier it comes, and the easier it comes, the more you can try to figure out what works for you and what doesn’t. With practice, you will get more comfortable writing in your own voice, making your writing more approachable. This idea is perhaps undervalued when writing about data science, particularly in professional settings. There is still value in finding a consistent voice when communicating findings at work, in academic research, and even (and especially) for your silly little blog posts. Consistency makes your writing flow better, familiarity will make your readers’ lives easier, and a unique voice is more pleasing and engaging.

Your voice will differ from my voice. My voice is stupid. I’m a profoundly unserious person. You might not be that person, and that’s okay (good, even). Finding your voice is essential in your writing journey⁹. Explore what works for you, what you enjoy, and what feels like an authentic representation of your personality.

Many excellent writers stress the importance of writing like a human being (Zinsser 2001) or writing like you talk (Graham 2015). How do you communicate with people when you are comfortable? How would you explain something to a friend who likes you enough not to disown you for talking at them about data science? This is how you should write. The closer you get to writing like your spoken voice, the closer you are to sounding like an actual human using everyday words in a regular order. This will also make your work much easier to follow. Cormac McCarthy argues that “spoken language and common sense” are the best guides when writing your first draft because it is more important that the reader can understand you than that you write to the letter of the grammatical law (Savage and Yeh 2019). And Cormac McCarthy knows a lot more about writing than I do.

Getting good at writing is often about practice, but it’s also about challenging yourself to improve, taking the time and effort to think about your writing and where it does and does not work, and considering how you can grow as a writer. Writing more will help you improve.

Final Thoughts

Writing is hard. But it can also be fulfilling when you get it right. I’m wary of seeming like the breezy hobbyist claiming that writing is a joyous experience, knowing that the professionals will read this and laugh at me¹⁰. Still, I think this point is worth reiterating because I get the sense it doesn’t get recognised when talking about writing about data science (or science generally). As much as a clean, well-executed analysis is satisfying, clear and concise findings can make your work shine.

You need to start writing to find those nuggets of joy. To whatever extent I am a good writer, it is because I have plenty of practice. I have developed my voice and learned how to achieve what I want with words. That, and the lower standard expected of a data scientist trying to communicate with words instead of numbers. Whatever your abilities, write honestly, with humanity and some humility. If your intentions are good but your writing is terrible, the latter can at least be fixed with practice.

Acknowledgments

Preview image by Patrick Fore on Unsplash.

Support

If you enjoyed this blog post and would like to support my work, you can buy me a coffee or a beer or give me a tip as a thank you.

References

Alexander, Rohan. 2023. Telling Stories with Data: With Applications in r. Chapman; Hall/CRC. https://tellingstorieswithdata.com.

Gopen, George D, and Judith A Swan. 1990. “The Science of Scientific Writing.” American Scientist 78 (6): 550–58. https://www.popecol.org/wp-content/uploads/2020/09/Gopen-Swan-1990.pdf.

Graham, Paul. 2015. “Write Like You Talk.” Paul Graham. https://paulgraham.com/talk.html.

Graham, Paul. 2024. “Writes & Write-Nots.” Paul Graham. https://paulgraham.com/writes.html.

King, Stephen. 2020. On Writing: A Memoir of the Craft. Scribner.

Plaxco, Kevin W. 2010. “The Art of Writing Science.” Protein Science 19 (12): 2261. https://onlinelibrary.wiley.com/doi/10.1002/pro.514.

Savage, Van, and Pamela Yeh. 2019. “Novelist Cormac McCarthy’s Tips on How to Write a Great Science Paper.” Nature 574 (7777): 441–43. https://gwern.net/doc/science/2019-savage.pdf.

Strunk, William, and Elwyn Brooks White. 1999. The Elements of Style. Pearson. https://archive.org/details/TheElementsOfStyle4thEdition/.

Zinsser, William Knowlton. 2001. On Writing Well: The Classic Guide to Writing Nonfiction. Quill/A Harper Collins Books.

Footnotes

For general advice on becoming a better writer, you are probably better off seeking the counsel of some actual professional writers. I’d recommend William Zinsser’s (2001) On Writing Well and Stephen King’s (2020) On Writing.↩︎
Regarding the technical aspects of writing, I’d highly recommend William Strunk & E.B. White’s (1999) Elements of Style and George Gopen & Judith Swan’s (1990) The Science of Scientific Writing.↩︎
I tried to track down a proper source for this quote. I could only find it attributed to Leslie Lamport, but not where he initially said it. I discovered it via Paul Graham’s Writes and Write-Nots (2024).↩︎
I have Mark Thompson to thank for this one, which he shared in a recent post.↩︎
The writing itself is only one component. Analytics gives you a few other tools that are not always available to other writers. Rohan Alexander’s (2023) Telling Stories With Data goes into detail about this stuff in the Communication section of the book. If your audience is comfortable with notation, there is no more succinct and precise way to express your model, and causal diagrams are an excellent way to lay out your theory intuitively and visually. The most powerful tool is data visualisation. If a picture is worth a thousand words, surely a plot is worth at least a couple hundred?↩︎
This section focuses on the first three principles, but humanity is no less critical. Writing with humanity means putting human beings at the centre of your writing. Tell human stories, discuss what you love about a particular topic, and highlight your work’s effects on real people. Speak to readers in terms that make them want to keep reading.↩︎
McCarthy focuses on academic writing, but this approach can be applied to writing in many contexts. It will help when dealing with a complex topic, as often in data science.↩︎
I think the trivial stuff was most helpful in improving my writing. When tasked with writing several hundred words about some absolutely abysmal game I had nothing to say about, I started experimenting. In those moments of desperation, I found a style that worked for me.↩︎
Your voice doesn’t have to be serious for your work to be taken seriously. There’s nothing wrong with having fun. Suppose you read my posts and are unconvinced that being a silly goose does not hurt your legitimacy. In that case, I’d point to Danielle Navarro and Dan Simpson as counters to your understandable reticence. Both are very engaging writers and don’t take themselves too seriously. Both are also excellent at this stuff and are bonafide experts.↩︎
Don’t laugh at me on the Internet. That’s illegal.↩︎

Reuse

CC BY-SA 4.0

Citation

For attribution, please cite this work as:

Johnson, Paul. 2025. “How to Write About Data Science (Effectively).” April 14. https://paulrjohnson.net/blog/2025-04-14-writing-data-science/.

Predicting Bob’s Burgers Episode Ratings

Paul Johnson — Fri, 07 Feb 2025 00:00:00 GMT

Setup Code (Click to Expand)

Resolutions for the Poster’s Reformation

Paul Johnson — Fri, 10 Jan 2025 00:00:00 GMT

The microblogging church has fractured. The dominant domain faces challenges from all corners as new groups offer salvation and prosperity to new followers. A familiar, smug order rises from the rubble amid the chaos. Fuelled by technological advances like ~~the printing press~~ Quarto, a Calvinist commitment to the content mines, and the rebellious act of drinking delicious, hopped-up beers¹, a new age of blogger has rejected the opulence and ritualism of the old way, instead choosing to go back to basics.

If you’re feeling lost, you’re not alone. This joke started to get away from me several sentences ago, and now I’m adrift². What I’m saying is Mark Thompson is correct - Blogging is Cool Again.

The recent resurgence of the humble blog³ is possibly a consequence of social media’s fragmentation. Shifting away from shorter-form content packed into threads is a dizzying new freedom. But too much freedom has consequences, as evidenced by all this slop. It’s nothing that can’t be fixed by some planning and a tiny shred of shame, though, and that’s why we’re here. If 2025 is the year of the blogsman, I intend to seize the moment and post my way to the moon. If I want to remain among the cool kids, I will need a plan of attack. Some goals for the new year. Resolutions, even. Hasn’t that worked out well?

Posting My Way to Happiness in 2025

I had much fun working on this blog over the last year. I wrote several pretty hefty posts that I’m still pleased to look back at now, and I learned plenty in the process. The dominant thread through the year was football analytics. Half of my blog posts this year were standalone football analytics projects. The intention isn’t that this should be the blog’s focus, but that’s how it shook out in 2024.

I enjoyed working on this blog over the last year and think it was worth my time. No one emailed me to tell me they were mad at me, so I’d call this a success!

Post Fast, Post Often

Like a child who left home to escape the tyranny of bedtime, this website is evidence of my living completely untethered from the shackles of a word count, refusing to admit I need adult supervision. Many of my posts last year were approximately the length of great works of Russian literature. Coming from academia, my natural instinct is to go long, but there are plenty of cases where this is unnecessary.

I do my best work in these extensive long-form analyses, but just like any writing, a quick analysis that does something interesting and doesn’t leave the reader with many unanswered questions is a learned skill. Over the next year, I want to practice this skill⁴.

Not all posts need to involve analysis either⁵. I would like to mix things up a bit, from sharing cool stuff I’ve found to writing about experiences in my career. I just want to spend 2025 keeping the haters guessing.

Be Good, Not Perfect

This one sounds incredibly self-satisfied, but that’s only because I couldn’t think of another subheading that doesn’t make me sound like a smug idiot. Nothing I’ve ever written is perfect, and some of it might not even be good, but the old adage, “perfect is the enemy of good,” is something I struggle with often in my posting. I frequently spend several days, weeks, or even months trying to get everything right. While that might be defensible in some cases (I think it was a worthwhile effort for the analysis of the effect of money in football), it is usually just a case of completely overthinking things.

If anyone comes to this blog expecting perfection, that’s on them. The goal should be to do a good job and write something interesting and fun, at least for the kind of nerd who reads my posts. Julia Evans’ post dispelling common blogging myths is fantastic, and I would highly recommend anyone read it (I think it applies beyond the blogging context too). I would like to do a better job of following her advice this year⁶. Pursuing perfection is unrealistic and a little silly for what is ultimately just a fun little blog.

Undertaking With Vigour The Task of Posting

In the end, whether you are writing long wordy threads about football tactics on Twitter or Bluesky, or debating the specifics of political language deep in the Discord trenches, or you’re just a simple blogsman seeking salvation from such indulgences, we are all just posting⁸. It’s not my place to judge. I’m just focused on post-maximising.

I started this blog because it seemed like something I should be doing. These posts are all relevant to my professional development. I’m learning stuff that will benefit my career, and as silly as the blog’s tone often is, it’s still a portfolio of sorts. Nonetheless, the main reason I have kept the blog going is because I enjoy it, and the goal in 2025, above all else, is to keep enjoying it. If I can create something others find useful and avoid anyone emailing me telling me to log off, even better.

Acknowledgments

Preview image by Tom Hill on Unsplash.

Support

If you enjoyed this blog post and would like to support my work, you can buy me a coffee or a beer or give me a tip as a thank you.

Footnotes

Listen, if I want to make my soul sad and weigh down my inner organs, that’s my business.↩︎
I haven’t studied the Reformation, but I assume it was just a bunch of lovely lads who travelled around Europe being very normal about Christianity.↩︎
Are Substacks blogs? From a content perspective, what’s really the difference between a newsletter and a blog? I think they’re the same. Is this a heathen’s take?↩︎
I suspect that writing quick, interesting analyses is as much about the setup as it is about the writing. Identifying questions interesting enough to bother answering but simple enough to tackle quickly is not easy!↩︎
I’ve heard posts don’t have to be about football, either, though I’ve yet to see any evidence to support this claim.↩︎
I am guilty of falling for all of the myths Julia mentions, several of them regularly.↩︎
An Introduction to Statistical Learning is one of those books I’ve read a decent chunk of over the years but never start to finish.↩︎
Unless you are a LinkedIn person, in which case you are a demonic creature.↩︎

Reuse

CC BY-SA 4.0

Citation

For attribution, please cite this work as:

Johnson, Paul. 2025. “Resolutions for the Poster’s Reformation.” January 10. https://paulrjohnson.net/blog/2025-01-10-resolutions-posters-reformation/.

Analysing Money’s Effect on Football Using Multilevel Regression

Paul Johnson — Wed, 09 Oct 2024 00:00:00 GMT

Note

An earlier version of this post included a couple errors and insufficient detail in the section translating the model’s outputs into real-world findings. While I’ve fixed these issues, I won’t be as bold as to claim there are now zero errors in this post. There are fewer than before, though.

Setup Code (Click to Expand)

Calculating a Football Team’s Strength of Schedule

Paul Johnson — Tue, 02 Apr 2024 00:00:00 GMT

Setup Code (Click to Expand)

Missing Data Matters (Part II) - Solutions

Paul Johnson — Tue, 26 Mar 2024 00:00:00 GMT

Note

This is the second post in the Missing Data Matters series. The first post, about missing data mechanisms, can be found here.

Setup Code (Click to Expand)

Missing Data Matters (Part I): Mechanisms

Paul Johnson — Thu, 14 Mar 2024 00:00:00 GMT

Note

This is the first post in the Missing Data Matters series. The second post, about missing data solutions, can be found here.

Setup Code (Click to Expand)

Setting up Netlify Redirects With Quarto

Paul Johnson — Fri, 01 Mar 2024 00:00:00 GMT

Note

I have updated my process for generating Netlify redirects. If you are interested in an automated process, check out the blog post discussing my updated approach.

If you are using Netlify to host your Quarto website there is a ton of helpful functionality that can smooth out the process. They even give you a fancy deployment status badge, and who among us can really claim they’re better than a nice status badge?

One of the features that Netlify offers is a simple process for handling redirects. Whether you need to redirect traffic from old URLs from your previous site URL structure (it’s me) or you want to shorten URLs to make it easier to share links to your blog posts, Netlify’s _redirects handles all of this very easily.

It’s pretty easy to add a _redirects file to your site if you manually publish from your local machine or link your Netlify site to a Git repository. However, if you have continuous deployment setup with GitHub and/or Netlify or if you want to automate the process of grabbing URLs and generating their redirects (or both), then it’s not immediately obvious how to do this in a painfully manual way with a Quarto website.

There are a few resources out there that will help you figure this out, but my needs were a little different. I had to adapt these guides to fit what I was trying to do, so I thought I’d put a quick explainer together to help others in the same situation.

Standing on the Shoulders of Giants

Much of the process detailed in this blog post has been influenced by or borrowed from other online guides. Rather than try and pitch this as the product of my unique, beautiful mind¹, I want to give credit where it is due.

These were the resources I found most useful:

Danielle Navarro - Porting a Distill Blog to Quarto
Silvia Canelón - Hello Quarto: Porting my Website from Hugo Apéro
Aster Hu - Migrating from Jekyll to Quarto
Jadey Ryan - Intermediate Guide to Publish a Quarto Website with GitHub & Netlify

These are much more detailed blog posts about creating a website with Quarto, and there’s a lot more good information in these than just setting up redirects. This blog post will have a shorter, more specific focus, but my solution leans heavily on several of these guides, so I want to dish out the appropriate credit right away. I will also try to highlight throughout the post where my code is adapted from one of these resources.

Setting Up Manual Redirects

First, I created a _manualredirects file that I’ve added to the assets directory (following a similar process to Silvia Canelón). It contains all the blog posts I wrote a couple years ago because the old blog used a slightly different URL structure. I’ve also added a couple other pages that no longer exist, and they redirect to either the homepage or other pages that are successors.

I’m sure it would be possible to automate most of this, but I haven’t figured out an easy fix². Wherever you’ve got multiple redirects that don’t follow the same pattern and aren’t going to change in the future (so wherever you’re fixing redirects from an old site or old site structure), it’s probably easier to specify these manually.

Each line in a redirects file refers to a different redirect rule, with the original path listed first, followed by the path being redirected to. Your redirects might look something like this:

/portfolio/                               /projects/
/2023/08/good-but-long-post/              /blog/2023-08-good-but-long-post/
/2023/10/okay-post-that-no-one-needed/    /blog/2023-10-okay-post-that-no-one-needed/
/2024/01/boring-post-you-regret/          /blog/2024-01-boring-post-you-regret/

There are quite a few ways you can customise the behaviour of a redirect rule, and for anything more complex I would recommend reading through the Netlify documentation.

Automating Redirects

To automate the process of creating a redirects file you need to add some code somewhere that it can be run regularly. I’ve added these code chunks to the index.qmd file in the project root folder, and I’ve added echo: false to the YAML frontmatter so that the code doesn’t appear on my homepage. This is more or less the same process that Danielle Navarro and Silvia Canelón follow, but the difference is that I am not setting freeze: false in the YAML frontmatter (for reasons explained in Section 4).

There are multiple sources from which redirect rules will be derived, which requires several steps to bring them together in the _redirects file, but the first step is to import the _manualredirects file.

# import redirects
manual_redirects <- readr::read_lines(here::here("assets", "_manualredirects"))

Redirecting Blog Posts

Besides making sure no one gets lost in the ether using your old URLs, one of the benefits of redirects is creating shorter URLs that are easier to share. In my case, I’m really just removing the date slug from my blog posts, but this still looks a little neater and is easier to remember.

The code chunk below is also adapted from Silvia Canelón’s process, making a few changes to meet my needs, and it defines a function that gets the file paths to every blog post and then defines redirect rules for each.

# get file paths for all blog posts
get_file_paths <-
  function(folder) {
    blog_posts <-
      list.dirs(
        path = c(here::here(folder)),
        full.names = FALSE,
        recursive = FALSE
      ) |>
      tibble::as_tibble_col(column_name = "path") |>
      dplyr::mutate(folder = folder)
  }

# create & extract shortened redirects
blog_posts <-
  # get blog post file paths
  purrr::map("blog", get_file_paths) |>
  # row-bind file paths
  purrr::list_rbind() |>
  dplyr::mutate(
    # remove date slug
    blog_title = stringr::str_remove(path, "(?!\\-)\\d{4}-\\d{2}-\\d{2}-"),
    # create short path
    short_path = paste0(folder, "/", blog_title),
    # combine old path and new path to create redirect rule
    redirects = paste0("/", short_path, " ", "/", folder, "/", path)
  )

blog_redirects <-
  blog_posts |>
  dplyr::pull(redirects)

Redirecting Categories

Another useful trick that is possible using a similar process is redirecting old tags or categories for blog posts. This probably isn’t necessary for my site because I doubt I have written enough posts or drawn enough traffic to my blog posts that the old tags are being used that often, but it is nice to be thorough. This might be more useful to those of you who are a much bigger deal than me!

The code chunk below is adapted from Danielle Navarro’s process, again with some minor adaptations specific to my use case, defining a function to get all categories listed in blog posts³, and defines redirect rules by specifying the old (tags) and new paths (categories).

# get categories from blog posts
get_categories <-
  function(post) {
    file <- here::here("blog", post) |> fs::path("index.qmd")
    if (!fs::file_exists(file)) {
      return(character(0))
    }
    file |>
      readr::read_lines() |>
      stringr::str_subset("^categories:") |>
      stringr::str_extract("\\[.*\\]") |>
      stringr::str_remove_all("[\\[\\]]") |>
      stringr::str_split(", ") |>
      purrr::list_c()
  }

# extract categories
categories <-
  # blog post categories
  purrr::map(blog_posts |> dplyr::pull(path), get_categories) |>
  # concatenate categories into a vector
  purrr::list_c() |>
  unique() |>
  stringr::str_replace_all(" ", "%20")

# extract tags
tags <-
  categories |>
  stringr::str_to_lower() |>
  stringr::str_replace_all("%20", "-")

# write category redirects
category_redirects <-
  paste0("/tags/", tags, " ", "/blog/#category=", categories)

The previous code chunks have compiled all the redirect rules for the site, and most of it uses an automated process. This saves time and limits my opportunities to be an idiot, which I always appreciate.

Bringing it All Together

Finally, we can combine everything into a single vector and save this as _redirects in the project root, for reasons that should become clear in just a moment.

# write the _redirects file
readr::write_lines(
  x = c(manual_redirects, blog_redirects, category_redirects),
  file = here::here("_redirects")
)

We now have a _redirects file that contains all the redirect rules we want to use on our site, from manual redirects to automated blog posts and category redirects.

Including Redirects in Netlify Builds

The solutions I have used as a reference point all run this code every time the site renders, saving the _redirects file in the _site directory. This works in those cases because they don’t appear to be using continuous deployment. They are pushing the _site directory to GitHub and using this to deploy their website via Netlify. There is nothing wrong with doing it this way, and if you want to save yourself some trouble I’d recommend taking this easier (and probably more sensible) route instead. However, I really like Netlify’s build tools, and there are some Netlify plugins that I want to use.

Given that I want to use continuous deployment, I don’t want to check my _site directory into version control, so I have stored the _redirects file in the root. Instead, I need to move the file to the build directory once everything has been built, which leads me to the other sticking point. I can’t set freeze: false because this will cause the R code to run every time the site is rendered, including when it is being built on Netlify. This would require Netlify to have R installed, and, as far as I am aware, that is not possible. I needed a solution that would freeze all computations and make it possible for Netlify to build the site.

My solution was to save the redirects file to the root and include it as a resource for the build process, by adding resources: _redirects to _quarto.yml, nested under projects. This ensures that when Netlify builds the site, it moves the redirects to your build directory, where Netlify can see it! I have also kept freeze: auto for the file containing the redirects code, to avoid Netlify trying to run it and deployment failing. That means that when Quarto renders the site it stores the results of running these code chunks in the _freeze folder, which needs to be checked into version control so that Netlify can use these outputs.

This is an imperfect solution because freezing this code means that it won’t run unless the source code of the file this code is in changes, which probably won’t happen that often. My solution is to run quarto render index.qmd whenever the redirects need updating. This isn’t ideal, and I’m sure this will result in me forgetting to update the redirects, but I’ve yet to think of a better alternative⁴.

Summary

Netlify offers some valuable features. Netlify’s build tools and redirect capabilities are great, but they don’t necessarily play nice with Quarto, depending on how you want to combine them.

There are guides available online for getting redirects working with Quarto, but things get a little trickier if you want to use continuous deployment with Netlify’s build tools. This post sought to build on the good work others have done and explain how I got redirects working for my particular use case. I suspect there are better ways to approach this. My solution is very much cobbled together. However, it addresses a slightly different problem than the resources I’ve found about setting up Netlify’s redirects with Quarto.

If you have come up with a better way of doing this than me, please let me know!

Support

If you enjoyed this blog post and would like to support my work, you can buy me a coffee or a beer or give me a tip as a thank you.

Footnotes

Please understand that I definitely have a unique, beautiful mind though.↩︎
I think this should be relatively simple for older blog posts, but for the ad-hoc pages that are being redirected, I think a manual approach is probably the only option.↩︎
I really like this function. It would have taken me a long time to come up with something much worse than this, so thanks to Danielle for her work!↩︎
If you have any suggestions, I’d love to hear them.↩︎

Reuse

CC BY-SA 4.0

Citation

For attribution, please cite this work as:

Johnson, Paul. 2024. “Setting up Netlify Redirects With Quarto.” March 1. https://paulrjohnson.net/blog/2024-03-01-setting-up-netlify-redirects-with-quarto/.

Developing Offensive Ratings Using Desirability Functions

Paul Johnson — Thu, 22 Feb 2024 00:00:00 GMT

Setup Code (Click to Expand)

A Cheatsheet for Building Multilevel Models in R

Paul Johnson — Tue, 01 Nov 2022 00:00:00 GMT

Note

Some minor changes have been made to this blog post, cleaning up some unclear and confusing language/phrasing, and adding additional resources for learning about multilevel regression.

Multilevel models (MLMs) are hard. Some part of the difficulty is due to the fact that multilevel data structures add an extra layer of complexity conceptually and mathematically, and therefore the appropriate model structure for this kind of data is also more complex. However, there’s plenty about them that just seems needlessly hard.

Oh would you like to make a multilevel model or a hierarchical model? But have you considered a generalized linear mixed effects model? What about a variance component model? WHAT DO YOU MEAN THEY’RE THE SAME THING?¹! That can’t possibly be true. Why would anyone in statistics or the sciences make anything needlessly convoluted and use different names for the same concepts? Thankfully things are much simpler when it comes to discussing fixed and random effects. Just a couple concepts with precise meaning and usage in every field… Wait, what? Oh you’ve got be fucking kidding me.

So I thought I’d try and make things a little easier for anyone taking their first steps into this fresh hell, and for anyone (like me) that has a mushy, liquefied brain and needs a refresher any time they haven’t looked at something every single day for the last two years.

I’m not going to write a full introduction, or a detailed how-to for constructing multilevel models, because a) that sounds long as hell and b) there are much smarter people than I that have already done these things (resources are linked at the end of the post). This will just serve as a quick explainer of what MLMs are, when you need them, and the syntax for constructing the most basic (and most common) types of MLMs in R. The goal is to keep this relatively short and make it accessible for folks without intensive statistics backgrounds. Therefore, I will attempt to keep the notation to the minimum, explaining model structures and using examples to illustrate. I have included some notation, but I’ve kept it hidden to make this cheatsheet² a little bit less intimidating. If you fancy your chances, you can expand the notation sections.

I won’t go into the details about fitting, interpreting, and validating multilevel models here, but I will possibly turn this into a series, and if so I will get in to it in the next blog post. LFG!!!

What are Multilevel Models?

A common feature of many real-world questions that practitioners might study is that there are multiple “levels” around which the data is structured. A dataset may contain observations at a population-level, with a higher-level variable that specifies a group that those observations belong to (for example, patients being treated by a certain doctor or in a certain hospital, or political parties operating in different countries). Figure 1 below is a visual representation of how multilevel data might be structured.

graph TB
  A([Group 1]) --> B[Unit 1]
  A --> C[Unit 2]
  A --> D[Unit 3]
  E([Group 2]) --> F[Unit 4]
  E --> G[Unit 5]
  E --> H[Unit 6]
  I([Group 3]) --> J[Unit 7]
  I --> K[Unit 8]
  I --> L[Unit 9]

Figure 1: A Hierarchical Grouping Structure

Assuming that the grouping variable is relevant to the outcome being studied, this means that some observations will be correlated with each other, or clustered around the groups they belong to, which invites bias into any model that doesn’t account for this clustering. Multilevel models are regression models that handle data which is structured in groups³.

Notation

Rendering Mermaid Diagrams on a Hugo Website Using Quarto

Paul Johnson — Sat, 17 Sep 2022 00:00:00 GMT

Note

Having ported my site over to Quarto in entirety, this blog post no longer applies to the site as is. There are some inconsistencies with some of the issues it talks about and how the site looks now (for example, there is no longer an option to toggle a dark theme), but the advice in this post should hopefully still work for anyone that is using Quarto with Hugo!

I find a lot of the methods for drawing diagrams and flow charts using code to be a bit of a nightmare. I’m not sure what it is, perhaps I’m just a bit stupid, but they always seem a little more convoluted than I can handle. It’s possible that the design is actually of some value when it comes to drawing diagrams that are much larger and more complicated than I am generally dealing with, but in my case, it always seems to be a lot of work for what little I’m trying to do.

The easiest, most intuitive way that I’ve come across is using a JavaScript library called Mermaid. It’s just very easy, and the diagrams look great. Even better is the fact that it works with a lot of Markdown formats, and it’s even one of the diagram formats that Quarto can handle.

One issue that I came across, however, is that Mermaid diagrams don’t render when you’re using Quarto to create Hugo Markdown files for a Hugo website. You do have some simple options to resolve this, such as setting the mermaid-format to PNG or SVG in the article front matter, but with just a little extra work, you can get Mermaid working perfectly on your Hugo website.

Setting up Hugo to Support Mermaid

Steps for setting up Hugo to support Mermaid are detailed in the Hugo documentation, however that didn’t seem to work for me (producing errors when Netlify tries to publish the site). Instead, I had to take a slightly different approach, which was inspired by Kevin Wu’s blog post about using Mermaid with Hugo. This should work for others, but if it doesn’t, please lets just assume that it is either Kevin’s fault, or your fault, but definitely not my fault.

First, create a HTML file /layours/partials/mermaid.html, and add the following code, which specifies where to find Mermaid and how it should be loaded (in this case I’ve specified the dark theme, but there’s a few different options):

<script src="https://cdn.jsdelivr.net/npm/mermaid/dist/mermaid.min.js">script>
<script>mermaid.initialize({startOnLoad:true, theme:"neutral" });script>

Following this, a snippet needs to be added to one of the partial layouts that will be used in your content pages. I added it to the extend_head.html file (which, as the name helpfully suggests, is just an extension of head.html). This snippet just tells Hugo that, when Mermaid is called in the frontmatter, it needs to use the mermaid.html partial you just created.

{{ if .Page.Params.mermaid }}
{{ partial "mermaid.html" . }}
{{ end }}

This can either be added to the existing theme partial, located in themes//layouts/partials, or it can be in the global layouts/partials directory, which will override your theme.

Now, you just have to add mermaid: true to the YAML frontmatter for any page that contains a Mermaid code block, and you’re good to go!

Rendering Mermaid Using Quarto

Once you’ve set Hugo up to handle Mermaid code blocks, you would think it should then render like other code chunks in Quarto, and you would be right. Mermaid is very good.

I find that I sometimes have to render the Quarto document a couple times for Mermaid diagrams to show up, or for changes to render properly, but that’s about the only issue that I’ve come up against.

Flowcharts

The diagram I will get the most use out of is flowcharts, and Mermaid makes them really easy.

```{mermaid}
%%| label: fig-mermaid-flowchart
%%| fig-cap: Wow look how it flows.
%%| fig-alt: An example flowchart, drawn using Mermaid.

graph LR
    A([Oooh]) --> B([Would You])
    B --> C([Look at This])
    C --> D[Very]
    C --> E[Fancy]
    C --> F[Flowchart]
```

graph LR
    A([Oooh]) --> B([Would You])
    B --> C([Look at This])
    C --> D[Very]
    C --> E[Fancy]
    C --> F[Flowchart]

Figure 1: Wow look how it flows.

Gantt Charts

If Figure 1 isn’t enough to convince you, how about some Gantt charts?

```{mermaid}
%%| label: fig-mermaid-gantt
%%| fig-cap: Oh you real fancy huh?
%%| fig-alt: An example Gantt chart, drawn using Mermaid.

%%{init: {'theme': 'default', 'themeVariables': { 'textColor': '#798189'}}}%%
gantt
    title Big Ol' Gantts
    dateFormat  YYYY-MM-DD
    section Section
    Some Normal Tasks : done, a1, 2022-09-01, 15d
    But Wait, There's More! :after a1, 20d
    section And ANOTHER
    Deary Me, It's Never Ending : crit, 2022-10-01, 20d
    I Hate This Project : 14d
```

%%{init: {'theme': 'default', 'themeVariables': { 'textColor': '#798189'}}}%%
gantt
    title Big Ol' Gantts
    dateFormat  YYYY-MM-DD
    section Section
    Some Normal Tasks : done, a1, 2022-09-01, 15d
    But Wait, There's More! :after a1, 20d
    section And ANOTHER
    Deary Me, It's Never Ending : crit, 2022-10-01, 20d
    I Hate This Project : 14d

Figure 2: Oh you real fancy huh?

The dark and light theme my Hugo site uses doesn’t handle the Gantt chart as well as the other Mermaid diagrams. If I use the neutral Mermaid theme, it doesn’t show as clearly when using the dark Hugo theme, and vice versa with the dark Mermaid theme and light Hugo theme. This isn’t ideal, but you can customise the appearance of your Mermaid diagrams too.

I’ve stuck with the neutral Mermaid theme for the rest of the site because it seems to be the best fit for flowcharts with the two different Hugo themes, and flowcharts are the diagrams I’m going to be using Mermaid for most often on this site. However, for Figure 2 I’ve customised the appearance a little, using %%init%%, specifying the default theme and a grey text colour that works slightly better (though it’s still not brilliant).

Git Graphs

So you’ve got Figure 1 to wow people with your idea and Figure 2 to manage the project, but now you need some git graphs to show how you broke everything.

```{mermaid}
%%| label: fig-mermaid-git
%%| fig-cap: Git a load of this!
%%| fig-alt: An example Git graph, drawn using Mermaid.

gitGraph
  commit
  commit
  branch develop
  checkout develop
  commit
  commit
  checkout main
  merge develop
  commit
  commit
```

gitGraph
  commit
  commit
  branch develop
  checkout develop
  commit
  commit
  checkout main
  merge develop
  commit
  commit

Figure 3: Git a load of this!

Summary

And that’s about all you need to do. Think of all the amazing diagrams you’re going to create and show the world? I, on the other hand, am going to continue to make minimal diagrams that are completely uninspiring. Have fun!

Acknowledgments

Preview image by Daniella Garcia on Unsplash.

Support

If you enjoyed this blog post and would like to support my work, you can buy me a coffee or a beer or give me a tip as a thank you.

Reuse

CC BY-SA 4.0

Citation

For attribution, please cite this work as:

Johnson, Paul. 2022. “Rendering Mermaid Diagrams on a Hugo Website Using Quarto.” September 17. https://paulrjohnson.net/blog/2022-09-17-rendering-mermaid-diagrams-using-quarto-and-hugo/.

Setting up a Hugo Website with Quarto

Paul Johnson — Sun, 11 Sep 2022 00:00:00 GMT

Note

Having ported my site over to Quarto in entirety, this blog post no longer applies to the site as is. This post should hopefully still work for anyone that is using Quarto with Hugo!

Like every other nerd that likes R and has used RMarkdown more than one time in their entire life, I am now in the process of transitioning to using Quarto, and I want to let the world know! I wanted to make good use of Quarto, including utilising the really nice job it does with rendering R, Python, Julia, and JavaScript (not that I’m writing a lot of Julia or JavaScript) and outputting it in a HTML document. However, I didn’t want to give up this lovely website I had already put together using Hugo. What a conundrum… Well, it turns out it’s relatively easy to use Quarto with a Hugo site. These folks really have thought of everything!

This isn’t a tutorial for building a Hugo website or building a website from scratch with Quarto. These are more involved processes (though neither are incredibly difficult). If that’s what you’re looking for, here are some links that might point you in the right direction.

Hugo Tutorials:

Quarto Tutorials:

No, my ambitions with this blog post are considerably less ambitious. I just want to show you how to set your existing Hugo website up so that you can use Quarto to write your blog posts (and any other pages).

Setting Up Hugo for Quarto Files

The first step is to make some changes to your ‘config.toml’ file so that it doesn’t try and publish your Quarto documents, whether they are in qmd or ipynb format. You need to add the following code to the file:

ignoreFiles = [ "\\.qmd$", "\\.ipynb$" ]

I added it to the opening section of the config file, with the baseUrl, title, theme etc.

You also need to add some configuration to the markup section of the config file, in order to make sure Hugo handles raw HTML appropriately, because R and Python will sometimes give HTML as an output rather than Markdown.

[markup.goldmark.renderer]
  unsafe= true

This is essentially everything you need to get your Hugo website set up to deal with Quarto without losing its mind.

Creating Blog Posts

This is the part that I expected to at least be a little bit harder than it turned out to be. Once you have set your config file up, you can now create new posts (and any other pages) in pretty much the same way you did in the before times. In my case, all the blog posts are stored under ‘content/posts’, and it’s the same deal now in this exciting new Quarto world.

Blog posts are written in a qmd file, with the YAML front matter looking much the same as the typical Markdown posts for Hugo sites, with tags, categories, keywords etc., and format: hugo added so that Quarto knows how to render the file. This can be also be added to a ’_quarto.yml’ file, as I have done, if you are treating the site as a Quarto project.

Once you’ve done all this, you can render your Quarto files and it will produce Markdown files that Hugo will publish as blog posts.

Don’t Forget to Render

The biggest problem I’ve faced with this process, in the short time since making the change, is that I am a stupid person and that I sometimes do stupid things.

When you’ve written your posts in a qmd file you have to remember to render them as and when you make changes that you want to deploy. I have already managed to forget to do that on a couple occasions, pushing changes that I haven’t rendered, resulting in qmd files that don’t line up with the md file, and therefore don’t show on the site. Think of all the funny things you might write that people won’t get to see, and it’s all your fault. Get in the shame cube!

Wrapping Up

The rendering issue isn’t ideal, because as much as I’d like to believe that I will eventually remember that I need to render before I publish posts, I happen to know that I am a buffoon, so I am certain that I will forget enough that it’s going to be a bit of a nuisance.

This should be something that I can resolve by setting up a GitHub Actions workflow to render files when pushed to the repository. It’s not something I’ve done just yet, mostly because it is 8pm on a Sunday and I can’t be bothered to take that extra step right now. But maybe I’ll write another post about that when I get round to it. Because everything is content and everybody is posting.

However, everything else is exceptionally smooth, and very easy to set up. It takes a matter of minutes to make the change and now your Hugo site can be used with Quarto, making it so much easier to write about and incorporate R, Python, and other programming languages in your blog posts. Great success!

Support

If you enjoyed this blog post and would like to support my work, you can buy me a coffee or a beer or give me a tip as a thank you.

Reuse

CC BY-SA 4.0

Citation

For attribution, please cite this work as:

Johnson, Paul. 2022. “Setting up a Hugo Website with Quarto.” September 11. https://paulrjohnson.net/blog/2022-09-11-setting-up-hugo-website-with-quarto/.

Resources for Learning Bayesian Statistics

Paul Johnson — Tue, 10 May 2022 00:00:00 GMT

As a social scientist turned data scientist, my graduate school training taught me a lot of frequentist statistics that has served me well in my career since. However, there’s one thing that frequentist statistics can’t solve, and that’s my lack of a personality. For that, we’ve got Bayesian statistics, the perfect substitute.

I’ve spent the last year or so gradually trying to become more Bayesian in my approach to statistical reasoning and analysis, and while there’s quite a few really good resources for supporting that journey, it can be a little difficult to know where to start. I wanted someone to give me a short reading list that would save me trying to work out which books to use and in what order. In an attempt to fill that gap, I’ve put together a list of resources that I’ve come across and found to be pretty solid. I’ve not worked through each and every one of these cover to cover, but I’ve interacted with them enough to think they’re useful and that someone might be able to learn a lot from them.

Where possible I’ve tried to find stuff that is free and open-source, but that hasn’t been possible in every single case. But in each case I have at least found a freely available resource that is made available by the authors, whether it be the book itself, a video series, or a series of articles.

Becoming More Bayesian

These first resources are good starting points for anyone looking to start learning more about Bayesian statistics. They’ve all got plenty of detail, but they focus on making the intuition of Bayesianism a little more approachable.

Statistical Rethinking

I think the starting point for everybody, and an all-round great resource for anyone interested in statistics, is Richard McElreath’s Statistical Rethinking. If you read only one of these books, make it Statistical Rethinking. It’s a great read for anybody interested in statistics and quantitative research.

The focus of Statistical Rethinking is utilizing Bayesian statistics for scientific reasoning/thinking and causal inference. It’s a really great read, in part because it is really accessible while also being pretty thorough too. I particularly like his consistent focus on how these tools apply to scientific models.

Not only is it available as a book, but Richard also makes a lecture series available on YouTube which can serve as an accompaniment to the book, or can even stand on its own as a good learning resource.

Student’s Guide to Bayesian Statistics

Ben Lambert’s A Student’s Guide to Bayesian Statistics is a really good dive into the statistics that underpin Bayesian analysis, and the implementation of the statistical methods. It’s not light on notation, but it does introduce new concepts by first explaining the intuition before laying it out mathematically. I think this book pairs very well with Statistical Rethinking, as they take slightly different approaches, and between the two you cover tons of ground.

You can buy the book on Amazon but it isn’t freely available anywhere (that I’ve found). Ben does offer a free YouTube course and lecture notes that mirror the book’s content though. The YouTube course doesn’t offer the kind of detail that the book does, but it is still pretty good.

Introduction to Empirical Bayes

If you’re looking for something to help you apply Bayesian statistics to a real-world situation, or you’re the kind of learner that needs a relatable hook to help you make sense of things (this is me), then David Robinson’s Introduction to Empirical Bayes is the perfect starting point. David applies Empirical Bayes methods (Bayesian approaches where the prior is estimated from the data) in a baseball context, specifically simulating batting averages. Baseball isn’t really the focus of David’s work here, but it serves as a really good medium for couching some quite complicated concepts in terms that people can understand, and thanks to this (and David’s excellent communication of the concepts), it is a really good opening gambit for anyone interested in Bayesian statistics.

The Introduction to Empirical Bayes book started out as a series of blog posts on David’s website, Variance Explained, and those blog posts are all still freely available, but his e-book also includes an extra chapter, some additional materials across several other chapters, and some minor edits and changes to help simplify things. The book is also available at a “pay-what-you-want” price, which means you can choose to pay nothing, if you can’t afford to pay more, but for those that can, the suggested price is $9.95 (~£8).

I don’t think that David’s e-book/blog posts are the most thorough of the resources listed here, but they’re also not trying to be. Instead, they’re a really good plain(ish)-English explanation of Bayesian concepts, applied to examples that should help a lot of people intuit them a little easier. The use of Empirical Bayes serves as a good first step in the direction of Bayesian approaches, and it is also a good way to learn some really useful methods in real-world data analysis. If you find that some of the other resources are a little hard to wrap your head round, I’d read all of his posts (or buy his book if you can afford to) first, because it might help lay the groundwork for your learning journey.

Think Bayes

The vast majority of the resources listed here use R for implementation. I think that’s because the packages for computing Bayesian models in R are much easier to navigate than their alternatives in Python. However, if you’re coming at this as someone with limited or no knowledge of R and no interest in changing that, then Allen Downey’s Think Bayes is worth a look. Think Bayes takes a code-first approach to teaching Bayesian concepts, rather than using notation and illustrating ideas using concepts like calculus. For anyone that uses Python and would learn better through concepts being explained through code, this book is a great start.

Building a Stronger Understanding

The following resources would serve as great next steps for someone looking to build on their understanding of Bayesianism, whether it be an extremely thorough exploration of the mathematics underpinning the methods, or detailed documentation of the code for the computation of Bayesian methods.

Bayesian Data Analysis

Everything Andrew Gelman does is good, and Bayesian Data Analysis (co-authored with John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin) is yet more proof of that. It’s a really extensive book covering the fundamentals of Bayesian inference, analysis, and methods.

I think Bayesian Data Analysis might be a little heavier going if you’re just starting out, but the book is also supplemented by an online course made available by Aki Vehtari, with lecture slides and videos to help bridge any gaps in understanding.

The book and the course are excellent resources for those looking to build on foundational knowledge (perhaps developed using the earlier resources). It’s probably the most thorough of all the resources here, so if you’re comfortable with mathematical notation and only want to read one book, this might be your best bet.

Stan Documentation

Stan is a probabilistic programming language for implementing Bayesian statistical models. It’s the language that underpins the computation of Bayesian methods, and there are interfaces that make it accessible in a number of different languages, including R, Python, and Julia (for those of you looking to intimidate your peers).

The Stan documentation, and the documentation for the R interface to Stan, RStan, and the Python equivalent, PyStan are good resources for learning how to implement a variety of Bayesian models.

Beyond Bayes

Finally, there are books that are worth a mention because they are really good resources that, while not focused entirely on Bayesian statistics, do a great job of placing Bayesian approaches in the context of wider statistical methods.

Regression and Other Stories

More Gelman. I have no regrets! Regression and Other Stories is co-authored with Jennifer Hill and Aki Vehtari, and it will serve you brilliantly if you’re looking to learn about Bayesian approaches while also covering the frequentist methods too. It is a book that focuses on the application of regression to real-world problems and the challenges users will face in the process.

While the focus of the book is regression, it incorporates Bayesian inference and methods across the book, and this might be useful to anyone wanting a broader education in statistical analysis that includes Bayesian approaches.

Probabilistic Machine Learning

If you’re looking to apply your newly developed Bayesian principles to machine learning, then Probabilistic Machine Learning: An Introduction by Kevin Patrick Murphy is a great start.

It’s not specifically a Bayesian machine learning book, but probabilistic machine learning incorporates a lot of Bayesian principles, and there’s plenty of explicitly Bayesian tools in the book too. If you’ve cracked several of the resources in this list and are looking to take your next step, I think this would be a great direction.

Conclusion

I think that having a good understanding of Bayes rule is really important for anyone working in or around statistics, because it helps inform the way you reason about statistics, and it helps you interrogate your approach to analysis in a more thorough manner. Getting a good intuition for Bayesian thinking is relatively simple, but if you’re looking to go a little further, it does require a bit of work. I think there are tons of really good resources out there to take those next steps, but I don’t think they’re as easy to track down as they could be. Hopefully this list will bridge that gap for a few people.

Support

If you enjoyed this blog post and would like to support my work, you can buy me a coffee or a beer or give me a tip as a thank you.

Reuse

CC BY-SA 4.0

Citation

For attribution, please cite this work as:

Johnson, Paul. 2022. “Resources for Learning Bayesian Statistics.” May 10. https://paulrjohnson.net/blog/2022-05-10-learning-bayesian-statistics/.

Combining Tidymodels & Caret for Machine Learning in R

Paul Johnson — Sat, 07 May 2022 00:00:00 GMT

The two main approaches to building machine learning models in R are caret and tidymodels. Having tried both, I found that I struggled to pick my favorite. There’s elements of both that made more intuitive sense to me than the other. I think it’s a product of having become very familiar with the tidyverse, particularly dplyr, for data wrangling, but still using a lot of Base R functions for statistical modeling.

The process for prepping the data for a machine learning model seems to make a ton of sense to me when done in tidymodels (using recipes and rsample), but the equivalent process using caret felt a little clunky. However, specifying and training models using caret made a lot of sense to my broken brain.

Anyway, I recently discovered something that is probably entirely unremarkable to everyone else, and that probably shouldn’t have taken me by surprise… You can just combine the two! You can split and preprocess your data using the tidymodels framework before defecting to caret for the next steps. What a time to be alive.

Predicting Diabetes Using Random Forest

Because I’m not a savage, I won’t leave you without a simple worked example. We’ll use Gary Hutson’s really useful MLDataR package to grab a toy diabetes dataset, cleaning the variable names using janitor, and converting the target variable, diabetic_class, to a factor.

# import packages
suppressPackageStartupMessages({
  library(dplyr)
  library(caret)
  library(recipes)
  library(randomForest)
})

# load data
diabetes_raw <- MLDataR::diabetes_data

# clean data
df <-
  diabetes_raw |>
  janitor::clean_names() |>
  mutate(diabetic_class = as.factor(diabetic_class))

Train/Test Splits

Having done this, we can use rsample to split the data into a train and test set.

# set random seed
set.seed(456)

# split train/test data
train_test_split <-
  rsample::initial_split(df, strata = diabetic_class, prop = 0.7)

# create train/test sets
train_df <- rsample::training(train_test_split)
test_df <- rsample::testing(train_test_split)

Data Preprocessing

The next step is a little more involved, and is where I think tidymodels really excels. Using the recipes package, we can specify all the preprocessing steps needed for the dataset, such that the data will then be ready for training a machine learning model.

# preprocessing
model_recipe <-
  recipe(diabetic_class ~ ., data = train_df) |>
  # combine low frequency factor levels
  step_other(all_nominal(), threshold = 0.05) |>
  # remove predictors with zero variance
  step_nzv(all_predictors()) |>
  # normalize numeric variables (sigma = 1, mu = 0)
  step_normalize(all_numeric()) |>
  # convert nominal variables to numeric binary variables
  step_dummy(all_nominal(), -all_outcomes(), one_hot = TRUE)

You can check that all the preprocessing steps are working as expected by using prep() and juice().

# check preprocessing results
model_recipe |>
  prep() |>
  juice() |>
  glimpse()

Rows: 364
Columns: 32
$ age                     -0.67696788, -1.18732268, -1.52755921, 1.619628…
$ diabetic_class          Negative, Negative, Negative, Negative, Negativ…
$ gender_Female           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
$ gender_Male             1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,…
$ excess_urination_No     1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,…
$ excess_urination_Yes    0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
$ polydipsia_No           0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1,…
$ polydipsia_Yes          1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0,…
$ weight_loss_sudden_No   0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1,…
$ weight_loss_sudden_Yes  1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0,…
$ fatigue_No              0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0,…
$ fatigue_Yes             1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1,…
$ polyphagia_No           1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1,…
$ polyphagia_Yes          0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0,…
$ genital_thrush_No       1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 0,…
$ genital_thrush_Yes      0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 1,…
$ blurred_vision_No       0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1,…
$ blurred_vision_Yes      1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,…
$ itching_No              0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0,…
$ itching_Yes             1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1,…
$ irritability_No         1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1,…
$ irritability_Yes        0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0,…
$ delay_healing_No        1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0,…
$ delay_healing_Yes       0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1,…
$ partial_psoriasis_No    0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1,…
$ partial_psoriasis_Yes   1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0,…
$ muscle_stiffness_No     0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1,…
$ muscle_stiffness_Yes    1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,…
$ alopecia_No             1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0,…
$ alopecia_Yes            0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1,…
$ obesity_No              1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1,…
$ obesity_Yes             0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0,…

If everything looks alright, you can take the model_recipe object that you’ve created and use it as the model formula that you would otherwise have to specify in the caret train() function.

Model Training

For the rest of the process, you can switch over to caret, first using the trainControl() function to specify the training parameters and then the train() function for the model training.

# set random seed
set.seed(456)

# control parameters for model training
ctrl <-
  trainControl(
    method = "cv",
    number = 5,
    classProbs = TRUE,
    summaryFunction = twoClassSummary
  )

# train random forest model
rf_mod <-
  train(
    model_recipe,
    data = train_df,
    method = "rf",
    tunelength = 10,
    metric = "ROC",
    trControl = ctrl,
    importance = TRUE
  )

Having trained the random forest model, you can check the performance, and see what parameters were chosen in the tuning process.

# check results
print(rf_mod)

Random Forest 

364 samples
 16 predictor
  2 classes: 'Negative', 'Positive' 

Recipe steps: other, nzv, normalize, dummy 
Resampling: Cross-Validated (5 fold) 
Summary of sample sizes: 292, 291, 291, 291, 291 
Resampling results across tuning parameters:

  mtry  ROC        Sens       Spec     
   2    0.9947186  0.9500000  0.9372727
  16    0.9865188  0.9428571  0.9326263
  31    0.9839791  0.9428571  0.9235354

ROC was used to select the optimal model using the largest value.
The final value used for the model was mtry = 2.

Not bad! The best performing model has an ROC of 0.995 and both the sensitivity and specificity are ~0.95. Pretty solid for a quick and easy model.

Test Predictions

To really test the model’s performance, we want to see how it copes with the test data that it hasn’t seen.

# make predictions on test data
rf_predict <- predict(rf_mod, newdata = test_df, type = "prob")
rf_class <- predict(rf_mod, newdata = test_df, type = "raw")

preds <-
  cbind(rf_predict, rf_class) |>
  mutate(
    Positive = round(Positive, digits = 2),
    Negative = round(Negative, digits = 2)
  )

Finally, we can produce a confidence matrix for a more intuitive look at how the model is performing on the test set.

cm_class <- test_df[, names(test_df) %in% c("diabetic_class")]

confusionMatrix(
  rf_class,
  as.factor(cm_class$diabetic_class),
  positive = "Positive"
)

Confusion Matrix and Statistics

          Reference
Prediction Negative Positive
  Negative       57        1
  Positive        3       95
                                         
               Accuracy : 0.9744         
                 95% CI : (0.9357, 0.993)
    No Information Rate : 0.6154         
    P-Value [Acc > NIR] : <2e-16         
                                         
                  Kappa : 0.9455         
                                         
 Mcnemar's Test P-Value : 0.6171         
                                         
            Sensitivity : 0.9896         
            Specificity : 0.9500         
         Pos Pred Value : 0.9694         
         Neg Pred Value : 0.9828         
             Prevalence : 0.6154         
         Detection Rate : 0.6090         
   Detection Prevalence : 0.6282         
      Balanced Accuracy : 0.9698         
                                         
       'Positive' Class : Positive

The results are pretty good for a very quick model. How exciting. Lets pretend that it’s because I’m a brilliant data scientist rather than it being due to the very clean, balanced toy dataset we used.

Wrapping Up

So there you have it, if you’re in the same position as me and you’re struggling to pick between tidymodels and caret, because both frameworks offer something you like, you can just combine the two and make Frankenstein’s framework.

Ultimately, despite this blog post, I’m probably going to stick with tidymodels (why am I like this?). I think that I’m going to force myself to get used to the tidymodels framework end-to-end because a) it is receiving tons of development so it’s probably going to continue to get better and bigger, and will be leading the way for the foreseeable future, and b) because in reality I think the explicit way that you structure each step is probably sensible, even if it confuses me a bit.

But it’s nice to know that I’ve got options.

Acknowledgments

Preview image by Rock’n Roll Monkey on Unsplash.

Support

If you enjoyed this blog post and would like to support my work, you can buy me a coffee or a beer or give me a tip as a thank you.

Reuse

CC BY-SA 4.0

Citation

For attribution, please cite this work as:

Johnson, Paul. 2022. “Combining Tidymodels & Caret for Machine Learning in R.” May 7. https://paulrjohnson.net/blog/2022-05-07-combining-tidymodels-and-caret/.

Make More Packages!

Paul Johnson — Sat, 19 Feb 2022 00:00:00 GMT

Having learned R in an academic context, I wasn’t exposed to programming in a linear path that taught me the basics of programming theory and application. I learned how to carry out statistical analysis using R, and my interaction with packages was that they would help me get the job done without having to do everything manually. As a result, package development has always intimidated me a little bit. It seemed like the responsibility of much smarter people that know what they’re doing, and that can build packages that can help me figure out what I’m doing too.

To be fair, that was probably an accurate description of my situation for… a good few years of my time using R. But practice makes you less bad at things, and over time I’ve gone from being someone that uses R like it’s a boiling pot of oil that they’re about to chuck a fistful of frozen chips at for the first time, to someone that has full confidence that they can’t do any damage because they’ve already burned the house down anyway.

That is (mostly) a lie. I’m now (mostly) good at what I do. But that didn’t stop package development being a process that intimidated me.

However, I recently made the jump into some basic package development as a result of working on an internal package that I am helping to develop. While it wasn’t being built entirely from scratch, this package was a good opportunity for me to play around with some cool functions and learn about package infrastructure. Having done so, I realised that the process is actually pretty painless.

To follow-up on what I’d learned, I decided to build my own (stupid but funny) package from scratch, and made nice.

Now, I thought I would write this short blog post detailing the process for creating a very basic package, in the hope that it might help someone else overcome an imagined barrier to entry. I will run through the process of setting up a package, adding some simple functions, and then building out the basic infrastructure with documentation and testing, using nice as an example.

Initial Setup

In order to build a basic package from scratch, you should only need to rely on four different packages that will aid you in setting everything up, building the correct infrastructure, and documenting and testing your functions.

The packages you need:

devtools is responsible for the majority of the functions you’ll need to build and maintain a package and usethis handles the initial setup of the project, while roxygen2 helps you document your package, and testthat simplifies unit testing.

Functions

Packages are effectively just a way of housing and distributing functions. If you’re making a package, you’ll need some functions! They don’t need to be wildly complex. They just need to work, and ideally they would be reasonably useful too. However, in the absence of useful functions, we’ve got an example of a funny function instead:

check <- function(x) {
  if (x %in% c(69, 0.69)) {
    print("Nice!")
  } else {
    print("Not very nice.")
  }
}

This is very simple. All it is doing is checking if an object x matches either the number 69 or 0.69. If it does, then it returns “Nice!”, and if it doesn’t, it returns “Not very nice.”. If you’re just looking to build a basic package as a means of learning how to do so, this is all you need! You can build on this iteratively, as you become more confident with package development, or come up with other useful functions your package should include.

I think it is generally best practice to dedicate a different file to each individual function, though it’s not absolutely necessary. If you choose to, you can store all your functions in one file, and this might be a sensible approach if you’re creating a package that contains all your miscellaneous functions. However, for anything a little more meaty than that, I think the best approach is to split out everything into files for each function. That makes it easier for someone (likely including yourself) to navigate the source code.

Documentation

The next step is to document your package, so anyone using it will have some guidance on what they’re doing and why. Documentation doesn’t need to be complicated, though if the function it is documenting is pretty complex, it should probably go into plenty of detail to help anyone trying to use it. In the nice example, the functions are super simple, so there’s no sense in overdoing the docs either.

You need to add comments at the beginning of your function script which explain what the function does, details any of the required parameters for the function to run, and give some examples of usage. In order for roxygen2 to recognise the comments that need to be turned into documentation, you have to add #’ in front of them.

Below is a basic example of what is needed, corresponding to the nice check function above:

#' Checking whether your R output is really nice
#'
#' @keywords nice
#' @export
#' @examples
#' check(69)
#'

Having done this, devtools will process the documentation so that it all works as expected:

devtools::document()

You can get a look at your documentation by calling ?function like you normally would. In this case, calling ?check returns the documentation for the nice check function.

Testing

In order to reduce the likelihood that your package becomes a mess of errors, it is important to test it. The best way of doing this is automating your tests, to make everything reproducible, and reduce your own opportunities to do anything stupid!

To set up your tests, the following creates a new script for all tests of the particular function(s) you’re working on.

usethis::use_test("name")

An example of a basic test:

test_that("check returns correct type of response", {
  nice <- check(69)
  expect_type(nice, "character")
})

This test is, as it describes, checking that check() is returning the right type of output when it runs. The test creates an object called nice that contains the output from the check() function, when the input it receives is 69. The test expects the object to be of type character, and if that is so, then it will pass the test. If you want to build further tests for this particular function, you could check that the function correctly recognises whether the input is nice or not, and returns the correct response.

This would look as follows:

test_that("check returns correct positive response", {
  nice <- check(69)
  expect_equal(nice, "Nice!")
})

Once you have set up some tests that you want to use to check your functions, you can run those tests by calling on devtools:

devtools::test()

This will run through each test and check that it returns the expected output. If it doesn’t it will show that it has failed, and show you where exactly that failure has occurred, in order to help you fix it. If it has worked, then you’re all good!

Acknowledgments

Preview image by Jiawei Zhao on Unsplash.

Support

If you enjoyed this blog post and would like to support my work, you can buy me a coffee or a beer or give me a tip as a thank you.

Reuse

CC BY-SA 4.0

Citation

For attribution, please cite this work as:

Johnson, Paul. 2022. “Make More Packages!” February 19. https://paulrjohnson.net/blog/2022-02-19-make-more-packages/.

Making Myself Accountable

Paul Johnson — Mon, 17 Jan 2022 00:00:00 GMT

One of my goals when setting up this site was to use the blog section reasonably frequently to share some of my work and share anything particularly interesting that I’ve learned in my development as a data scientist. There hasn’t been much sharing going on. I’d like to think that isn’t because I haven’t done or learned anything interesting (…), I just haven’t found the time to share it.

I’m the ideas man. No one told me I’d also have to be the one to finish the ideas too. That seems deeply unfair.

But having realized that everything I’ve thought about sharing either hasn’t made it out of my head, or has been shared with friends in the form of a dumb joke, I’m looking to set myself some goals that will make the blog a little more active. So I’m using this space as an opportunity to hold myself accountable to my own fantastical intentions.

Over the course of the next 12 months, I’m going to try and average a blog post a month, around data science generally, and anything cool that I’m either learning or doing.

Given that everything is posting and everyone is a poster, it’s time I put some hours in at the content factory. New Year, New Me.

Acknowledgments

Preview image by Jack Lucas Smith on Unsplash.

Support

If you enjoyed this blog post and would like to support my work, you can buy me a coffee or a beer or give me a tip as a thank you.

Reuse

CC BY-SA 4.0

Citation

For attribution, please cite this work as:

Johnson, Paul. 2022. “Making Myself Accountable.” January 17. https://paulrjohnson.net/blog/2022-01-17-making-myself-accountable/.

Paul Johnson

Automating Netlify Redirects in Quarto Using Lua

The Old (Manual-ish) Setup

The Automated Solution

Redirects Using Lua

Testing Transfermarkt’s Squad Market Values

How to Write About Data Science (Effectively)

Writing the Right Way

Writing is Not an Afterthought

Simple is Smart

Practice Makes Perfect Less Bad

Final Thoughts

Acknowledgments

Support

References

Footnotes

Reuse

Citation

Predicting Bob’s Burgers Episode Ratings

Resolutions for the Poster’s Reformation

Posting My Way to Happiness in 2025

Post Fast, Post Often

Be Good, Not Perfect

Keep On Learning (and Sharing)

Undertaking With Vigour The Task of Posting

Acknowledgments

Support

Footnotes

Reuse

Citation

Analysing Money’s Effect on Football Using Multilevel Regression

Calculating a Football Team’s Strength of Schedule

Missing Data Matters (Part II) - Solutions

Missing Data Matters (Part I): Mechanisms

Setting up Netlify Redirects With Quarto

Standing on the Shoulders of Giants

Setting Up Manual Redirects

Automating Redirects

Redirecting Blog Posts

Redirecting Categories

Bringing it All Together

Including Redirects in Netlify Builds

Summary

Support

Footnotes

Reuse

Citation

Developing Offensive Ratings Using Desirability Functions

A Cheatsheet for Building Multilevel Models in R

What are Multilevel Models?

Rendering Mermaid Diagrams on a Hugo Website Using Quarto

Setting up Hugo to Support Mermaid

Rendering Mermaid Using Quarto

Flowcharts

Gantt Charts

Git Graphs

Summary

Acknowledgments

Support

Reuse

Citation

Setting up a Hugo Website with Quarto

Setting Up Hugo for Quarto Files

Creating Blog Posts

Don’t Forget to Render

Wrapping Up

Support

Reuse

Citation

Resources for Learning Bayesian Statistics

Becoming More Bayesian

Statistical Rethinking

Student’s Guide to Bayesian Statistics

Introduction to Empirical Bayes

Think Bayes

Building a Stronger Understanding

Bayesian Data Analysis

Stan Documentation

Beyond Bayes

Regression and Other Stories