On June 19, 2023, I setup scaling for the environment. My experience doing this is with AWS; however, I host this service with DigitalOcean. The good news is I now understand how DigitalOcean Load Balancers work. The bad news is I had intermittent outages throughout the day.

Sorry, I’ll do better.

  • jgrim
  • jgrim of SublinksOPMAEnglish
    2·
    2 years ago

    I created a new webhead image that was slimmed down and only had the front end. I set up scaling with DigitalOcean to create new instances from a snapshot. I didn’t know that when a snapshot creates an instance, it reset the /etc/hosts file, breaking private network traffic. I tried to use DigitalOcean’s tools to persist the host file. However, none of these options worked. I have to find a new way to map traffic.

    I put the site behind a load balancer with 2 web heads. The rate limiting by Lemmy kicked and started blocking traffic. I tried raising the limits too high to see if that’d fix it. It didn’t work. That’s when I realized that the Nginx configuration for Lemmy has additional rate-limiting configured. After I reconfigured the Nginx configuration, I was able to get traffic flowing again.

    This took a big part of my day. Sorry for those that missed out.