Conor Sheehan

Leaving Netlify Free Tier

2024-03-07T00:00:00+00:00

Background

I’ve had 3 sites on netlify since 2022.

URL	Date created
https://spelling-b.netlify.app	2022-08-15
https://beach-litriochta.netlify.app	2022-07-12
https://conorscocktails.netlify.app	2022-01-06

I liked the service, although I wasn’t thrilled with their default flow for deployments. I wrote a blogpost about how to deploy without giving netlify read and write access to all of your public and private repositories.

Leaving Netlify

Then I read in the news last week that a random developer using the same supposedly free tier on netlify got hit with a >$100k bill due to a DDOS attack. https://news.ycombinator.com/item?id=39520776

This is because the netlify free tier is not free. https://www.netlify.com/pricing

Their landing page says start for free but if you scroll down you’ll see there’s a 100GB bandwidth limit, and a fee for any traffic over the limit.

Normally this would be fine, but when you hit their limit they don’t stop service. They might not even inform you that you’ve exceeded the free limit until you get a bill at the end of the month.

They also do not protect against DDOS. They just continue service with their high rate of $55 per 100GB. For comparison AWS is around $9 per 100GB, and that’s still high IMO. https://www.hostdime.com/blog/data-egress-fees-cloud

I’ve been stretched thin lately and not working on personal projects, but the thought of a potentially infinite bill definitely kicked me into gear. Last week I copied all my netlify sites to cloudflare pages.

Today I deleted all my netlify sites and my netlify account. I managed to run those 3 sites on netlify for the last 2 years with ~20Gb bandwidth usage, but I’m not willing to stay given the risk.

Moving to cloudflare pages

I found the dev experience really good moving to cloudflare pages. They give you preview sites based on pr builds for free. The github access is very easily controlled. They only ask for read access and you can choose the repositories they get access to. They do ask for write access to metadata so they can put status messages on PRs, but I like that.

I only had 2 tiny issues:

I needed to set an environment variable YARN_VERSION to get the build to work
I had to delete and recreate a site to change the pages.dev url https://developers.cloudflare.com/pages/platform/known-issues/#build-configuration

If you need to change your *.pages.dev subdomain, delete your project and create a new one.

Other than that it was seamless. Definitely easier than the github action setup I needed for netlify.

Most importantly, on the free plan if you exceed their limits, they shutdown the service instead of continuing and charging you! https://developers.cloudflare.com/pages/functions/pricing/#free-plan

The icing on the cake is

On both free and paid plans, requests to static assets are free and unlimited

This reminds me of the good old days on heroku, where you had a free limit on compute each month, and if you exceeded it the service just stopped.

Conclusion

Although the Netlify CEO said they’d waive this bill, I wouldn’t recommend you host any site on netlify, unless you’re absolutely sure 100Gb of requests will make you more than netlify charge. For me I don’t run ads or make money per request, so I’ll be leaving entirely.

It seems like this tactic of keeping service up no matter what and charging high fees later isn’t limited to netlify. Vercel do something similar. https://serverlesshorrors.com/all/vercel-23k Generally I think this pattern is very dangerous. As a commenter on hackernews pointed out

It’s unbounded liability. Not to mention the strong conflict of interest for netlify, who stands to gain from their customers being attacked.

If you’re looking for an alternative, I’d recommend cloudflare pages. I found it great so far.

Archiving Youtube Timestamp Reddit Bot

2023-07-01T00:00:00+00:00

I have been meaning to write another article about the YouTubeTimestampRedditBot I made for a while now. My last article was nearly 2 years ago! Unfortunately this post will mostly be about why I’m archiving it. But first, some interesting bits about it:

Using Reddit as a makeshift database

I found lots of other bots used databases to keep track of things, like what threads they’d already seen or commented on. I managed to work around this by having the bot pm itself, using Reddit itself as a makeshift database. I’d use these pms like logs to debug when things went wrong, and to generally keep track of posts the bot interacted with.

I thought this technique might benefit other bot developers and lower costs for hosting bots, but that seems a bit pointless now that Reddit is starting to charge quite a lot for API requests that were previously free. The API pricing has caused a lot of controversy.

While the bot could continue to use the API for free based on https://www.redditinc.com/blog/apifacts I don’t really see any point in working to get it online again.

Effective July 1, 2023, the rate limits to use the Data API free of charge are 100 queries per minute per OAuth client id if you are using OAuth authentication and ten queries per minute if you are not using OAuth authentication.

Getting into the top 100 ranked bots

Before the Heroku free tier shutdown the bot was running fairly regularly, and managed to just edge into the top 100 ranked bots on botranks.com!

I don’t remember exactly what rank it started at, but it was very low. The lowest screenshot I could find was ranked 350 on 2021-11-24.

It had climbed to 150 by 2022-05-10, and 99 by 2022-09-20. I think it kept a pretty good pace. By the end it had around 1k karma and 90 “good bot” votes.

Unfortunately the botranks site itself is in danger of shutting down, since it uses the Pushshift API, which is also probably shutting down due to the API charges!

Decision to archive

A few things have influenced my decision to archive the bot:

The Heroku free tier ending
Reddit API changing from free to expensive
Lack of interest in working on the bot, and Reddit in general

I was a redditor for nearly a decade. I was even a moderator on a few small subreddits (just setting up automod and editing some CSS). Reddit has always had its problems but this really feels like a tipping point. As of today I won’t be using reddit, and I’m archiving this bot. Lots of third party reddit apps like RIF and Apollo stopped working today, so it seems like a fitting time to fully archive my reddit bot and move on. I’m going to go touch some grass.

Jekyll Plugins On Github Pages

2022-09-08T00:00:00+00:00

I finally got tags and categories pages working for this website! https://github.com/ConorSheehan1/ConorSheehan1.github.io/releases/tag/v3.0.0

To get the build working I had to use the jekyll-deploy-action.
Github doesn’t support the jekyll-tagging plugin I’m using.

I had some issues along the way. I had the usual problems with browser cache where I wasn’t sure if my css was really broken or just cached. What ended up working for me was adding baseurl: "" to _config.yml.

You can see the tags and categories pages here.

Buying And Configuring My First Domain

2022-08-04T00:00:00+00:00

Recently I made an Irish version of the New York Times spelling bee game. I wanted to host it on a domain that was easy to remember, but I’d never bought or configured a domain before. Here’s what I learned along the way.

Buying the domain

First I had to choose a company to buy the domain from. I wanted a ‘.ie’ domain since it’s an Irish game, so that limited the options anyway. Initially, I was thinking of using GoDaddy, but they require a tax ID. The only Irish tax ID I know of is a PPSN, which you’re not meant to share with anyone but the government. After asking around on r/DevelEire I decided to go with blacknight. It’s an Irish company so it was simpler to buy a ‘.ie’ domain from them, and they had similar prices anyway.

Next, I had to choose a domain. I named the game beach litríochta, which is a rough translation of spelling bee. Initially I wanted https://beach.ie because it’d be super easy to remember, makes sense in both English and Irish, and is kind of funny since “Beach” means bee in Irish.

I thought the domain was free, but it just had a slow redirect. It resolves to https://beachawards.ie, so I couldn’t get it. Instead I went with https://www.beacha.ie. Beacha is the plural of beach, and it’s still short and hopefully easy to remember.

Buying the domain was pretty straight forward. I had to upload a picture of my ID to blacknight to prove I’m an Irish citizen and that was it really.

Configuring the domain

This is where I ran into some issues. Blacknight is changing it’s control panel, so a lot of its support forum FAQs won’t work for new customers. The old control panel is at https://cp.blacknight.com, but I was a new customer so I had to use https://cp.blacknighthosting.com.

I already had my game hosted with Netlify, so all I needed was to forward the traffic from my new domain to Netlify. I found this blacknight support article on domain forwarding, but it was using the old blacknight control panel so I couldn’t use it.

I managed to get things working by following the Netlify docs, some back and forth with blacknight support, and some trial and error. Here’s what I did to set up forwarding:

Forwarding traffic

Go to Netlify and select your site -> site settings -> domain management
Add your custom domain(s)
Click the options dropdown -> Go to DNS panel
Copy the 4 DNS entries from Netlify
Go to the blacknight control panel.
Click the domains dropdown -> your domain
Click Nameservers Management
Select custom nameservers and paste in the values you copied from Netlify.
Now you have to wait “/. I left it overnight, and in the morning my domain was redirecting correctly!

If you’re interested in learning Irish or spelling games in general please check out https://beacha.ie! It’s free and the code is open source https://github.com/ConorSheehan1/beach-litriochta.

Reverse Engineering Uhabits Datamodel

2022-04-18T00:00:00+00:00

OK, that title sounds fancy and complicated. What did I actually do? First I should probably explain why I did anything in the first place.

Background

I’ve been using uhabits for years. I noticed a new numeric habit type was added in v2.0.0. Previously you could only track habits in a boolean way. Now instead of a checkbox per day, you had a numeric input to track how many times per day you did something.

I had lots of habits in the old boolean format that would benefit from the new format. E.g. I was tracking my coffee and alcohol consumption using days per week, rather than number of units per week. I looked around, but couldn’t find a way to convert between the habit types. I found this feature request, but no replies yet. I had a quick look through the codebase, and realized it’d be a lot of work to make a pr since I didn’t know the main language, kotlin. So I decided to try and make my own little utility to convert between the habit types.

I know uhabits supports importing / exporting it’s internal sqlite database. I don’t know kotlin, but I do now SQL, so instead of trying to figure out the logic from the sourcecode, I created a few habits with the new numeric type, and exported everything. I opened up the exported db and compared the differences between the old and new habits.

Reverse engineering the datamodel

I could see the tables and fields pretty easily, but that didn’t tell me what they were actually used for. I figured out that the habits and repetitions tables were what I needed to work on, because the others were either empty (events), or metadata tables (android_metadata, sqlite_sequence).

I needed to start messing with the data to see what would happen. I started with the habits table. I noticed type is 0 for boolean habits and 1 for numeric habits. freq_num only changes for boolean habits. For numeric habits, it’s always 1. That makes sense since target_value is generally 0 for boolean habits, but can be any number for numeric habits. freq_den seems to refer to the time unit. e.g. If I make a boolean habit with a target of 5 times per week I’d get the following for each type:

// boolean, aim to do something 5 days every 7 days
{freq_den: 7, freq_num: 5, target_value: 0, type: 0}

// numeric, aim to do something 5 times every 7 days
{freq_den: 7, freq_num: 1, target_value: 5, type: 1}

Next I looked at the repetitions table. I noticed boolean habits generally got a value of 2 when checked, while numeric habits went up in multiples of 1000. It looks like 1000 is used to represent 1.00, so decimals can be added too. e.g. 1.5 would be 1500.

Once I had identified the differences between habit types in the 2 main tables, I was able to create some python functions to convert from the old boolean type to the new numeric type.

After converting and re-importing my habits, I noticed that the graphs for boolean and numeric habits didn’t quite match. I decided to add 2 options; 1 to try to preserve the graphs, and another to preserve the data the way uhabits would generate it.

Creating a CLI to convert between types

After I was confident my converter worked, I made a little CLI layer to call it from, and published it on github. https://github.com/ConorSheehan1/uhabits_converter

The end results look like this:

Before	After

Stackoverflow Dev Story Sunset

2022-03-31T00:00:00+00:00

Sadly stackoverflow is sunsetting multiple services including jobs and developer story tomorrow. I’ve archived my developer story here.

Hopefully it’s an april fools joke. They’ve made plenty in the past but this seems more serious so I’ve archived my page anyway.

Edit from the future: it wasn’t a joke :(

Netlify Deployments From Github Without Giving Write Access

2022-02-21T00:00:00+00:00

Netlify is a great hosting service, but there’s one thing that bothers me about it. The default deployment path involves giving the service read and write access to all of your public and private repositories on GitHub :scream:

https://docs.netlify.com/site-deploys/create-deploys/#deploy-with-git

They skip over the authorization step in the video on their docs, but here’s what it looks like.

I found some questions on their forums asking about this. There’s even an open issue on their GitHub about it. Despite all the links I visited, I didn’t find a clear way to automate deploys to Netlify without giving them full access to my GitHub, so here’s how I managed to do it.

Initial drag and drop deploy

Before we can automate our deploys, we need a site ID. Netlify provides a drag and drop feature, so we can drag the output of a build, or even a folder with an empty index.html to create a new site. https://app.netlify.com/drop

GitHub action

Now that we have a Netlify site created, we can automate deploys to it. I used https://github.com/jsmrcaga/action-netlify-deploy, which requires an auth token and a site ID.

Generate auth token

We can generate a Netlify auth token by going to https://app.netlify.com/user/applications#personal-access-tokens. Click New Access Token, then give it a description and click generate. Copy the value, it won’t be displayed again.

To make the value accessible to the GitHub action, go to your GitHub repository and click settings -> secrets -> new repository secret. I named mine NETLIFY_AUTH_TOKEN and pasted in the value I copied from Netlify.

Get site ID

You can find your Netlify site ID by going to your Netlify site overview and clicking site settings and copying the APP ID. Again, to make it accessible to the GitHub action, click settings -> secrets -> new repository secret. I named mine NETLIFY_SITE_ID and pasted in the value I copied from Netlify.

GitHub action

Now that we have our secrets set up, we can create our GitHub action. Mine looks something like this:

# .github/workflows/deploy.yml
name: deploy

on:
  # enable manual deploys
  # https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/
  workflow_dispatch:
  # deploy tags and commits to master automatically
  push:
    tags:
      - "*"
    branches:
      - "master"

jobs:
  deploy:
    runs-on: ubuntu-latest
    name: "Deploy to Netlify"
    steps:
      - uses: actions/checkout@v2
      - uses: jsmrcaga/action-netlify-deploy@v1.7.2
        with:
          # pass secrets in to the action
          NETLIFY_SITE_ID: $
          NETLIFY_AUTH_TOKEN: $
          # add the GitHub ref to the deploy message so we can trace back what version is deployed from the Netlify side
          NETLIFY_DEPLOY_MESSAGE: "Prod deploy v$"
          NETLIFY_DEPLOY_TO_PROD: true
          # this bit should be custom to your project. I'm deploying a vuepress project that uses yarn, so these are my settings.
          install_command: yarn install
          build_command: yarn build
          build_directory: src/.vuepress/dist
          node_version: 14.18.2

That’s it! Now you can deploy to Netlify automatically from your GitHub repo, and you haven’t given up any access rights.

Example site

I figured this out while building this project https://github.com/ConorSheehan1/conors-cocktails which is deployed here https://conorscocktails.netlify.app if you’re interested.

Add Bootstrap Vue To Vuepress

2022-01-25T00:00:00+00:00

Create a VuePress project

The first thing we need to do is create a new VuePress project. It’s really easy with the latest version of npm or yarn. From the VuePress docs

yarn create vuepress-site $optionalDirectoryName

Install BootstrapVue

Next we install BootstrapVue

yarn add bootstrap bootstrap-vue

Configuration

Now we need to import BootstrapVue in .vuepress/enhanceApp.js, where we have access to the vue instance.

// .vuepress/enhanceApp.js
import { BootstrapVue, IconsPlugin } from "bootstrap-vue";

export default ({ Vue, options, router, siteData }) => {
  // Make BootstrapVue available throughout your project
  Vue.use(BootstrapVue);
  // Optionally install the BootstrapVue icon components plugin
  Vue.use(IconsPlugin);
}

Finally we need to load the bootstrap css. VuePress ships with stylus by default now, but we can still import css into our stylus file at .vuepress/styles/index.styl

/**
 * Custom Styles here.
 *
 * ref：https://v1.vuepress.vuejs.org/config/#index-styl
 */

@require '~bootstrap/dist/css/bootstrap.css'
@require '~bootstrap-vue/dist/bootstrap-vue.css'

That’s it! Now you can use BootstrapVue components in your VuePress app.

Example

Vuepress lets you embed components directly in markdown, so you can do something like this

<!-- src/index.md -->
## Hi from bootstrap-vue
<b-button>Hello world!</b-button>

Here’s an example app I’ve deployed to netlify which uses various BootstrapVue components including b-carousel and b-table: https://conorscocktails.netlify.app/

You can find the sourcecode here: https://github.com/ConorSheehan1/conors-cocktails

3 Patterns For Cookiecutter Templates

2022-01-07T00:00:00+00:00

Intro

If you’ve heard of cookiecutter you can skip this part.

Cookiecutter is a command-line utility that creates projects from templates. There’s a list of templates maintained by the cookiecutter team and plenty of community awesome lists. It’s built with python and uses the jinja templating framework (found in python web frameworks like flask). You can use it to make a template for pretty much anything! All you need to get started is pip install cookiecutter.

Hooks

Cookiecutter provides pre and post generate scripts. They are Python or Shell scripts that run before and/or after your project is generated.

They can be really useful. For example, if you want to get the absolute path to the generated project, you can use a post generate script to replace a specific piece of text with the absolute path. e.g.

# cookiecutter-$your-project/hooks/post_gen_project.py 
abs_path = os.getcwd()

for root, dirs, files in os.walk(abs_path):
    for filename in files:
        with open(os.path.join(root, filename)) as f:
            content = f.read()
        content = content.replace('replace_me.base_dir', abs_path)
        with open(os.path.join(root, filename), 'w') as f:
            f.write(content)

Here’s an example in a cookiecutter I made. See https://github.com/cookiecutter/cookiecutter/issues/955#issuecomment-444864537

Tests

There are a few ways to test cookiecutters.

Putting tests inside the template

This approach has the advantage that when someone generates a project using your template, they already have tests set up. e.g.

# {{cookiecutter.repo_name}}/{{cookiecutter.repo_name}}.py
__version__ = "0.1.0"


def {{cookiecutter.repo_name}}(version=False):
    if version:
        return __version__
    else:
        # do some cli stuff

# {{cookiecutter.repo_name}}/tests/test_{{cookiecutter.repo_name}}.py 
import unittest

class Test{{cookiecutter.repo_name}}(unittest.TestCase):
    def test_version(self):
        assert {{cookiecutter.repo_name}}(version=True) == "0.1.0"

Here’s an example in a cookiecutter I made.

Putting tests outside the template

This approach is useful if it doesn’t make sense to include tests in the generated project, but you still want to test what is generated. Note: this doesn’t mean trying to test cookiecutter itself!

Normally Cookiecutter opens a prompt to get user input to be injected into your template. You can bypass this with the --no-input argument. It also allows you to pass values required by cookiecutter.json as arguments. e.g.

// cookiecutter-$your-project/cookiecutter.json
{
  "project_name": "alphabet",
}

# this will generate a project named foo instead of alphabet
cookiecutter . --no-input project_name="foo"

I’ve used this approach when creating cookiecutters that contain scripts rather than full projects. To test the scripts I generate a project, import and run functions from the scripts, and test the output. e.g.

# cookiecutter-$your-project/{{cookiecutter.project_name|lower}}/script.sh
#!/bin/bash

{{cookiecutter.project_name|lower}}_repo_dir="{{cookiecutter.repo_dir}}"

goto_{{cookiecutter.project_name|lower}}_repo() {
  cd "${{cookiecutter.project_name|lower}}_repo_dir" || return 1
}

# cookiecutter-$your-project/tests/test_helper.bash
setup() {
    # we expect foo/script.sh to be generated
    load "foo/script.sh"
}

# cookiecutter-$your-project/tests/script.bats
#!/usr/bin/env bats

load "test_helper"

# we expect a function named goto_foo_repo in foo/script.sh
@test "goto_foo_repo" {
  goto_foo_repo
  assert_equal "$foo_repo_dir" "$PWD"
}

Here’s an example in a cookiecutter I made.

CI

Now that you have tests set up, you can set up continuous integration! The important bit of here is

cookiecutter 
. # create a project using the current directory as a template
--overwrite-if-exists # if the destination directory exists overwrite it
--no-input # don't prompt for user input. 
# since there are no other args, use default values from cookiecutter.json

Here’s an example with github actions

# .github/workflows/ci.yml
name: ci

jobs:
  build:

    runs-on: macos-latest
    strategy:
      matrix:
        python: [3.6, 3.7, 3.8]

    steps:
      - uses: actions/checkout@v1
      - name: Set up Python ${{ matrix.python }}
        uses: actions/setup-python@v1
        with:
          python-version: ${{ matrix.python }}
      - name: Install Poetry
        run: |
          python -m pip install --upgrade pip
          pip install poetry
      - name: Install python packages
        run: |
          poetry install
      # here's the important bit!
      # generate a new project using the cookiecutter template
      # use the default values in cookiecutter.json with --no-input
      # if the directory already exists, overwrite it
      - name: Generate package using cookiecutter
        run: |
          poetry run cookiecutter . --overwrite-if-exists --no-input
      # now inside the generated project, install dependencies and run tests
      - name: Install python packages (in cookiecutter dir)
        working-directory: example_cli
        run: |
          poetry install
      - name: Run tests (in cookiecutter dir)
        working-directory: example_cli
        run: |
          poetry run task tests

Here’s an example in a cookiecutter I made.

Install from GitHub

Cookiecutter provides a really easy way to use templates hosted on github. All you need is cookiecutter gh:$username/$repo

Hopefully now you should be able to create a Cookiecutter template with hooks, tests, and CI, all easily installable from GitHub!

Dev.to

2022-01-06T00:00:00+00:00

I’ve started blogging over at https://dev.to/conorsheehan1.
I guess this site could be considered a blog, but I like a lot of the features dev has and wanted to post there too. I’ll still document my projects here, but general blog posts will be on dev.to too.

Edit from the future: I’ve also started posting at https://conorsheehan1.hashnode.dev. All posts can be found here, but there may be conversations on the other sites.

I followed this great post to set up a RSS feed so a subset of the posts I make here with a specific tag are push out to dev.to and hashnode as well.