t/suki host migration and sso

Meta
1

t/suki host migration upcoming on 2026-01-23

Before the migration, in an abundance of caution, please back up anything important such as git repositories from git.tsuki.games. Make plans to not have access to these services during the migration time.

If you have any questions about the migration, please ask them here. If anything needs to be asked of me during the migration, you can email me or contact me on Discord (DM me for these contact details now if you don’t have them).

As mentioned in the chatplace for t/suki topic, I am going to replace the Discourse chat service with Matrix. However, before this can be done I must first:

  • Migrate forum.tsuki.games and git.tsuki.games to a new host. (I am planning on migrating to Hetzner, for their lower costs. If you have an alternative suggestion, I am open to considering it.)
  • Migrate the account systems to use SSO, so that you are automatically signed into all services after logging in once.

These things need to be done in order to reduce the cost of maintaining these services as well as make them easier to access for members.

During the migration, the forum and the gitplace will be shut down, and will not be available again until migration is completed. Since I have never completed such a migration before, I cannot make a good estimate of when it will be completed. I am hoping it can be performed within 24 hours, but I am reserving four days in case it takes longer. Hopefully, everything goes smoothly and I can bring the site back up in a prompt manner. :crossed_fingers:

5 Likes
3

Best of a luck for a smooth migration and Godspeed!

1 Like
4

I realized that I haven’t explained everything:

This migration is currently planned to move us from Atlantic to Hetzner, in order to reduce the cost of running t/suki.

At Atlantic, I currently pay:

  • ~$24/mo for two of the lowest VPS tier + backups

At Hetzner, I can potentially pay:

  • ~€8/mo for two of the lowest VPS tier + backups

I desire cost reductions because I anticipate an increase in computing needs when Matrix is added and I do not have the ability to bankroll that all on my own at the moment.

Unfortunately, I have recently become aware (thanks to @outfrost) that Hetzner apparently has a history of hostility to the LGBTQ+ community and has shut down sites arbitrarily (and on very short notice, due to a German law requiring that of them):

There’s a few ways we can deal with this:

  • We find a different host that is both cheaper than Atlantic and more ethical than Hetzner. Note that I’m not willing to consider AWS or GCP.
  • We stay with Atlantic and I receive more funding from others to help pay for the website. Note that I unfortunately don’t currently have a non-profit incorporated for t/suki, so you’d have to take me at my word.
  • We go with Hetzner anyway, knowing the risk.

I will do some research of my own as well, but I would be happy to hear any opinions y’all may have.

1 Like
5

Assuming we go with the route of staying with Atlantic and you field some funding to support t/suki, how much would you need per month to be stable?

6

That’s a great question. The total cost will either be ~$24/mo or ~$36/mo.

This is because after we add Matrix, we will either find a need to increase computing capacity or we will find that we don’t need it. Right now we have two G3.2GB servers, both of which cost $10/mo, and the next tier G3.4GB costs $20/mo (and backups cost +20%).

I could technically continue paying $24/mo out of pocket, though I still desire a reduction in cost… but $36/mo is too much for me. So, I would like at least $12/mo in contributions in such a scenario. However, to be honest, I’m not terribly excited to take contributions at the moment because I’d rather we set up some kind of formal, reliable system for this that makes it easier to track where the money is going and how it is being spent.

To describe how much of a cost reduction Hetzner is, I want to point out that they provide more computing power at their lowest tier compared to Atlantic, so I think it’s possible we won’t even need the next tier when Matrix is added. But, even if we did, the price when considering the next tier is still ~€11/mo due to a finer gradient of computing options, which is significantly cheaper than Atlantic and is a bigger cost savings than having $12/mo in contributions from members.

Hetzner also gives us the option of downgrading in-place, but Atlantic does not. So, if we ever upgrade to G3.4GB on Atlantic and find ourselves not needing the extra compute, I’m stuck with the cost unless I do some work to migrate the instance – which is a huge pain (it would be like switching hosts).

For what it’s worth, I did some additional research on Hetzner, and it seems that the anti-LGBT activity never broke the news or made it onto the Wikipedia page. It’s possible that the posts I mentioned earlier misrepresented or over-sensationalized the problem and it’s not as widespread as it might first appear.

1 Like
7

After more consideration, I have decided to not migrate to Hetzner and to stay on Atlantic.

I found more instances of Hetzner shutting down servers arbitrarily:

Including the bullshit abuse report and the woem.men shutdown I mentioned earlier, this makes for five instances where I believe Hetzner acted unfairly and in a manner that would be hostile to us.

In addition, all except for the kiwix story appear to be related to Hetzner’s Abuse Team threatening to shut down a server for “pornographic content” being hosted on Hetzner servers, where the pornographic content is either:

  • Not actually pornographic.
  • Not actually hosted on a Hetzner server.
  • Being hosted elsewhere by someone else unrelated to the parties in question.

Even when the Hetzner customer is in the right, it appears that Hetzner’s customer service and abuse teams don’t really communicate well and leave the customers in the dark.

Not to mention that we literally host pornographic content, since the gitplace hosts a team that is developing an adult game.

t/suki services will still be down in order to implement SSO.

2 Likes
8

I’ve laid the groundwork to run an idP on our moonlight server, where I also host the gitplace and plan to eventually host the matrix server. I also took some time during the downtime to fully update all services. Since I decided to not migrate hosts, the maintenance that needed to be performed was much shorter than planned, and you can consider all services to be available until further notice.

Though no more downtime is necessary to start hosting an idP and get SSO working, SSO work has not been completed. I was going to use Authentik, but I’m running into a problem where it uses up too much memory even when idle.

For the technically inclined, here’s a sample of what top says when Authentik isn’t running:

%Cpu(s):  0.3 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   1646.1 total,    586.8 free,    597.9 used,    630.3 buff/cache

And when Authentik is idle:

%Cpu(s): 17.1 us,  7.7 sy,  0.0 ni, 74.6 id,  0.7 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   1646.1 total,     77.4 free,   1388.8 used,    372.4 buff/cache

Once I start doing something with Authentik, the memory usage increases even more, which activates kswapd (in order to use swap space) and that ends up using the rest of our CPU resources.

I think this is unsuitable given the cost constraints, so I’m going to do some research into alternatives. I’ll keep everyone posted.

2 Likes
10

In light of Discord’s mandatory age verification announcement, I feel a report on how this is going so far is in order.

I explored Authelia for a few days to see if it would be an cheaper alternative to Authentik. It is in fact much cheaper, but it doesn’t come with an account system. So, I tried using one of the recommended LDAP implementations named LLDAP, which was interesting to me because it also prioritized performance over having a full feature set that we don’t need. However, the application is developed by one person, they don’t scrutinize the security of their program (which is understandable under the circumstances, I think), and they have admitted to using LLMs to generate code for the repository which I don’t think is great for an application where security is important.

I feel that the gains from using Authelia are quickly lost when the required LDAP is added, as it increases deployment complexity and, judging by the system requirements of other LDAP solutions I looked into, the runtime cost seems only marginally better at best. So, I decided to stick with Authentik.

I pulled the trigger on increasing the VPS size and got Authentik working today. @MyriadMinds and I investigated how to migrate everything over properly. However, there’s one sticking point: Invitations.

This is an invite-only community, but Authentik apparently only lets admins create invites. This means that I would have to be responsible for creating all the invites. We considered switching our focus back to Authelia, but it doesn’t support invitations at all. Our options are then:

  • admins (just me, for now) create all the invites
  • i keep trusting distrust, and you need to create an account on discourse then log into authentik before you can access other t/suki services
  • find out how to let authentik act as a transparent authentication gateway, so that your discourse login can be used for the other services

However, in the last two cases, authentik is a completely unnecessary overhead that will just make it harder for us to get to the applications we want to use and costs me more money for no added benefit. We might as well continue using distrust to turn Discourse into an OIDC provider.

So, that’s the plan. We’re going to give up on SSO and continue operating in the way we have been, with Discourse and distrust as our identity provider. Tomorrow, I’ll delete Authentik and set up Matrix.

4 Likes
11

I’ve been deploying netbird at my job and as a form of authentication they support a selfhosted solution called Pocket ID. I haven’t looked into the service too far yet (planning to do this in the near future) however, from a quick glance it looks promising.
However, the “down” side of pocket ID is that it doesn’t support username/password authentication. Instead it opts for a more secure (but less average user friendly) passkey only. Regardless I do think it could be interesting to look at.

1 Like
12

Ah, I’ve seen this before in my research as well. Unfortunately, I think passkey only might be too big of a technical literacy challenge for lots of users and it has the same lack of support for invitation-only user management:

Creating users in Pocket ID can be handled in two ways:

  1. Manually create users via the admin interface.
  2. Sync users from an LDAP source.

See: User Management

1 Like
13

I’m closing this topic in favor of moving further discussion to chatplace for t/suki?