Mikoto is completely bootstrapped, without a monetization strategy so far. This means that every cent counts to maximize our runway. Prior to this migration, we were spending around 48.08 USD per month on a single-node GKE cluster. While this isn't much for most startups, this is a major cost for a project ran by a fresh graduate with little sources of other income.

Our analysis for the costs:

• Nodes were much more expensive than what you could get from other clouds.

• Kubernetes creates Load Balancers per ingress, which eats into a lot of the costs.

Eventually, we've decided the cost was too much, so we'd have to move.

We Have Kubernetes at Home

I've the other Big 3 (AWS, Azure) clouds; while they offered slightly cheaper prices on their cluster. Another option could be to on-prem with our own hardware; this was not feasible due to our geographical location, as well as that we are a fully remote team.

The solution? Run our own Kubernetes cluster. Luckily, K3s exists, which is a small k8s distribution that can be installed with a single command. We've installed it on a Hetzner VPS.

Installing the Necessary Services

I set up a VPS on Hetzner and set up a k3s node on it. Wasn't particularly difficult.

Even out of the box, k3s comes with Traefik + ServiceLB out of the box. Using a software-based load balancer means that you do not need to pay for fees incurred by additional LoadBalancers (something that most cloud seem to create per Ingress!). Traefik is also a very nice Ingress controller to work with, supporting more feature than the NGINX-based one that comes with many other k8s distros.

One of the first things I set up in any k8s cluster is ArgoCD. It handles GitOps - watching a git repository for changes in your Kubernetes manifest, which will be applied to the Kubernetes cluster. We use an app of apps pattern to create an App containing other ArgoCD apps (even ArgoCD itself is managed with it), I was able to quickly replicate the old services from the cluster on Google Cloud into the Hetzner cluster.

Some of the services that power Mikoto now

After that, I've installed Cert-Manager, so that every service in the cluster can get HTTPS with minimal work on the container-level. Previously, we used HTTP01 challenge (which was less effort to set up), but we moved to DNS01 challenge, so that one Let's Encrypt Certificate could be shared for the entire infrastructure, and that we didn't need to wait for the certificate issuance every time we add a new service.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-dns
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-dns-private-key
    solvers:
      - selector:
          dnsZones:
            - "mikoto.io"
        dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: API_TOKEN
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: mikoto-wildcard
spec:
  secretName: mikoto-certificate
  issuerRef:
    name: letsencrypt-dns
    kind: Issuer
  commonName: "*.mikoto.io"
  dnsNames:
    # ... the domains go here
---
apiVersion: traefik.containo.us/v1alpha1
kind: TLSStore
metadata:
  name: default
spec:
  defaultCertificate:
    secretName: mikoto-certificate

Just adding this manifest will automatically make every Ingress in your K8s project use HTTPS!

Getting Rid of CockroachDB

Now, one of the things that we had problems with was moving out of CockroachDB Serverless. Mikoto keeps the DB close to the server for performance, but they did not offer CockroachDB serverless on Hetzner.

We previously used CockroachDB because it was free and it advertised itself as PostgreSQL-compatible. However, this is misleading, as the compatibility was surface level deep. Sure, the wire protocol is the same, and much of the syntax will work on both databases as well. However, no extension will work on CockroachDB, like PGVector, which blocked us from releasing our AI-powered search features.

Even more, CockroachDB cannot take advantages of the wide range of tools built for PostgreSQL (ORMs, Migration tools, Database Viewers), because the metadata-related queries for retrieving table information work in different ways.

Another main issue was the licensing: CockroachDB is licensed under BSL, a source-available license. This is problematic for us, in that Mikoto is open-source and running Mikoto would require using non-OSS software. While I have nothing against the CockroachDB team, but it was increasingly clear that their product direction did not fit my needs.

PostgreSQL on Kubernetes

Because of this, we've decided to bite the bullet and migrate out of CockroachDB and back into PostgreSQL. But even that was difficult; again, CockroachDB is only compatible with PostgreSQL on surface level. CockroachDB handles backups by its BACKUP statements.

The solution we settled on was to use cockroach dump from an old version of CockroachDB (the newer versions had deprecated the command), to dump just the table data, then use a few scripts to patch up the data to be imported into the PostgreSQL server running on my Kubernetes cluster.

We've used CloudNative-PG operator for PostgreSQL, which manages the installation of the software. We have it configured to continuously archive the cluster using Barman into a S3-compatible storage.

The Future

Our move brought our costs to 15.34 USD per month (nearly 70% in savings), even when using a node that was almost twice as powerful!

From our move from GCP to Hetzner, we've saved cut our costs by over 2/3 and doubled our capacity, while keeping all the benefits of running your services on a cloud. Of course, the big clouds have a reason to be more expensive, and while I haven't had to deal with them, I assume they have more support regarding scaling up and support. However, modern computers are stupidly powerful, and you can get decently far with just optimizing your code - our current infrastructure is expected to support at least a few hundred thousand concurrent users!

This blog post was written somewhat hastily and might contain many mistakes - please let me know if you're interested in some parts, or if you want me to expand on the parts I haven't covered, and I will update this post. Sorry for not keeping up with my own blogging schedules. Expect a lot more blog posts from me now!

This tweet inspired me to write this blog post:

What if we gave everyone their own internet?

We can easily tell that the internet (at least the social web) is broken. It's difficult to find any good content, 99% of the things you see are spam (to the point where people think "Dead Internet Theory" is real!), and everything is focused around quick dunks over productive discussions.

Still, I'm not a doomer. I think I have a good enough plan to fix at least some of the problems with it.

We need mixed-use skyscrapers, not public squares

Elon Musk has described that he is planning on making X (formerly known as Twitter) into a public square of the internet.

The thing is, no one wants to live in a public square: Imagine if you were living in a public square. It's way too noisy. There's some guy on the end of the street yelling about conspiracy theories. There are two drunk guys fighting it out under one of the trees. No one wants to be homeless. Maybe the public square isn't right for you. What you need isn't a public square, but a home.

No one wants an internet where they see every political discourse, nor every low-effort meme repost, or every "NUDES IN MY BIO". We think we do, but not because we want to: because that's what we're addicted to. That's what all the websites today do. This is inevitable, because they're all paid by how long they can keep the users on the app, not by how much value they're providing to the user. THEY DON'T WANT TO PROVIDE VALUE. THEY WANT TO EXTRACT YOURS. The incentives are misaligned from the start.

Some of the cracked people in a twitter DM that I'm it

In my personal experience over 10 or so years on the internet, the best experiences on any website, whether that be Skype, Reddit, Twitter, Discord, or various forums, wasn't interacting with the majority of the users. The best parts happened when some level of credibility was built, and you started getting invited into private groups, usually filled with anime pfp anons. It's arguably one of the parts with the highest alpha on the internet, yet there are almost no startups catering to this.

If other websites are all focused on the gold rush of building the "public square", Mikoto is skyscrapers full of residential and commercial units, to borrow their analogy. We want to build mostly-private part of the internet that you can call to be your home, while still maintaining the capability to cross-reference parts of the internet and only publish the parts that are actually valuable.

Our (not-so) secret plan

Step 1: Build a community app that's really fucking good

Mikoto achieves this by ruthlessly optimizing for signal-to-noise ratio. Every UI component means something; There are remappable hotkeys for every action; the layout is highly malleable. Many people have made comparisons of Mikoto with text editors/IDEs in terms of UI: This was exactly what I was going for - instead of optimizing for the lowest common denominator, we optimize for the superuser.

To do this: I've taken inspirations from text editors/IDEs like VS Code and JetBrains software, Which were always designed with superusers in mind first.

Top: VS Code, with the Mikoto codebase

Bottom: the main Mikoto space

See the similar UI idioms, but translated to a communication software from a developer software? The idioms of tabs, a tree-view of subjects, and split-screen window managements are all there. and it's immediately intuitive.

End of the blog (for now)

That was shorter than I expected, and I will have more parts to this blog coming out soon. Mikoto builds for the internet of the future; we're not here to restore it to its original form, or preserve the current flawed models of today.

Mikoto solves a very narrow section of the problem, and it's not enough to save all of the internet. We might even discover new problem as it grows. We might even be completely wrong regarding this hypothesis. I refer to building a social platform as "One of the hardest problems in the world" - because it is.

Still, I'm giving myself a pat in the back because this is a problem that even Elon Musk failed to solve, but I've managed to make some dent in the problem, and I'm proud of me and my community for getting at least that far.

Mikoto is already available as public alpha at https://alpha.mikoto.io: we will be entering public beta soon.