Infrastructure Q&A¶

Four Questions — Answered¶

Q1: What is the current server estate and how loaded is each box?¶

Before any consolidation or migration decisions can be made, we need a clear picture of what's running where and how much capacity each box has. The figures below are estimates — no live monitoring is in place yet. Actual numbers need to come from monitoring tools once deployed.

Server overview¶

PlaceCal production — estimated RAM usage¶

PlaceCal is a Ruby on Rails application. The following processes are expected to be running:

PlaceCal staging — estimated RAM usage¶

Same stack as production, lower traffic. RAM usage expected to be lower but same process set. Actual figure unknown without monitoring.

shaw — estimated RAM usage¶

shaw runs several services. Mastodon is the most RAM-intensive.

Kamal box — unknown¶

Spec: [INPUT NEEDED] Services: donna-bot, musicwall RAM usage: Cannot estimate — we don't know what either service does, how they're built, or the box spec. [INPUT NEEDED: what do these services do? What language/framework?]

Q2: Use GitHub instead of git.coopcloud.tech for Co-op Cloud deployments?¶

The Co-op Cloud config lives on git.coopcloud.tech (Gitea). [INPUT NEEDED: URL]. It has been copied to GitHub, so the config is already readable there without needing a Gitea account.

This largely resolves the visibility problem. The source of truth remains git.coopcloud.tech — changes are made there — but the GitHub copy means contributors can read the config without a separate login.

Switching the actual deployment tooling away from Gitea would require forking or mirroring Co-op Cloud's own tooling, which is built around git.coopcloud.tech. That's meaningful work and isn't necessary given the current direction in Q4.

Q3: Best path to migrate the Ghost blog?¶

Ghost does things a static site can't — it stays as a dynamic CMS. The problem is where it's hosted. Moving it off Digital Ocean onto one of the existing Hetzner servers is straightforward in principle.

The server review in Q1 should be completed first to confirm which box has capacity. Once that's known, Ghost is a good candidate for an early migration — the process is low-risk and well-documented, and it would also serve as a first demonstration of the build/mirror/switch method before applying it to more complex services.

Migration path: 1. Confirm which server has headroom (from Q1 monitoring) 2. Provision Ghost as a Docker container on that server 3. Export Ghost content from Digital Ocean — Admin → Settings → Labs → Export 4. Import into the new instance and verify 5. Lower DNS TTL on gfsc.community to 5 minutes (do this a day before cutover) 6. Switch DNS to new Hetzner IP 7. Monitor for 24–48 hours 8. Cancel Digital Ocean droplet

Effort: 2–4 hours. Email: Mailgun (£12/mo) stays — Ghost officially only supports Mailgun for transactional email. Risk: Low — Ghost export/import is well-documented and the content is self-contained.

Q4: Co-op Cloud or standardise on Kamal?¶

Two options. Both are viable — the right choice depends on the current management arrangement with Autonomic and what's actually on shaw after the audit.

Option A — Keep Co-op Cloud on shaw, clarify the management arrangement

Continue using shaw and Co-op Cloud tooling. Understand and document the current arrangement with Autonomic. Clear the dead services. Bring everything under GFSC's direct oversight.

• For: shaw already exists and is paid for; Co-op Cloud provides tested recipes for the services running on it; no migration work needed for Mastodon or HedgeDoc in the short term
• Against: management arrangement is currently unclear; Co-op Cloud adds a tooling layer that differs from how PlaceCal is managed; git.coopcloud.tech is a separate system from GitHub

Option B — Exit Co-op Cloud, standardise on Kamal

Migrate all shaw services to Kamal deployment (the same method used for PlaceCal). Could run on shaw repurposed, or on existing Hetzner boxes if capacity allows.

• For: one deployment method across all services; GitHub-native; same toolchain as PlaceCal; simpler mental model for future volunteers
• Against: meaningful migration work for each service; Mastodon in particular is complex to migrate; loses Co-op Cloud's recipes and any remaining Autonomic support

[INPUT NEEDED: confirm current arrangement with Autonomic — this affects which option is practical]

Mastodon — Current Situation and Short-Term Position¶

Mastodon/Hometown is currently self-hosted on shaw and under GFSC's control. It runs five processes simultaneously (web, two Sidekiq workers, streaming API, PostgreSQL, Redis) and requires periodic major version upgrades.

GFSC's general direction is to keep control of its own services — and Mastodon is already self-hosted, so the question isn't whether to take it on but whether to keep it as-is.

There are external factors that may affect Mastodon's longer-term home — organise.diy and other federated infrastructure developments in the wider community. Until there's more clarity on those, making major changes to the Mastodon setup would risk having to move it again shortly after.

Short-term position: keep Mastodon running on shaw, fix what's broken, update it to a current version, and write a runbook for it. Don't migrate it as part of this work.

[INPUT NEEDED: is Hometown's local-only posting feature actively used? This affects any future decisions about where Mastodon runs.]

Information Needed¶

The goal is to make existing services safe, well-maintained, and documented — not to replace them with cheaper alternatives. Cost information is still critical though, because capacity decisions (which server Ghost moves to, whether shaw can be downsized) depend on knowing what's actually being paid for.

Costs¶

Note on Mailgun and Ghost: The £12/mo Mailgun account is confirmed for Ghost. For self-hosted Ghost, Mailgun handles email sending — both transactional (member welcome emails, password resets) and newsletter/broadcast sending to subscribers. [INPUT NEEDED: does the Ghost blog actively send newsletters to a subscriber list, or is email only used for transactional messages?] If it's only transactional and volume is low, a free SMTP tier (e.g. Brevo free: 300 emails/day, no monthly limit) could replace Mailgun at no cost. This would only be worth investigating if Ghost is staying on Hetzner long-term and the newsletter feature isn't actively used — it's not a reason to switch platforms.

• Confirm total Hetzner invoice (all boxes)
• Confirm shaw monthly cost and billing arrangement

Shaw — needed before Q4 can be decided¶

• What is the current management arrangement — who does what on shaw, and on what terms?
• Confirm shaw spec from Hetzner dashboard (RAM total especially — affects whether Ghost can move there)
• git.coopcloud.tech repo URL for shaw config

Mastodon¶

• Is Hometown's local-only posting feature actively used? (affects any future decisions, not current work)
• Any known external changes to federated infrastructure (organise.diy etc.)? (context only)

Services — needed before changing anything¶

• What are the Mailman lists? How active? (determines fix vs retire)
• How is HedgeDoc actually used, and by whom?
• Are there active n8n workflows that need preserving?
• Any active Listmonk subscriber lists or campaigns?

Unknown services¶

• What does donna-bot do, and what language/framework is it?
• What is musicwall, and what language/framework is it?

Access — needed before touching anything¶

• Who holds SSH access to the Kamal box?
• Who holds SSH access to shaw?
• Who controls the Cloudflare/Namecheap accounts?
• Who holds the Hetzner account?
• Is there a shared credential store (1Password, Vaultwarden, etc.)?

Monitoring — What Can Be Added Now, For Free¶

The PlaceCal production server likely has unused RAM (estimated ~6.75GB headroom, based on Q1 Rails stack breakdown — not from live monitoring, actual figure TBC). The following can be deployed on it at no extra cost, before any other decisions are made.

Layer 1: Uptime Kuma — 1 hour to deploy, zero ongoing cost¶

Monitors every public URL and sends a Discord ping when something goes down. Generates a public status page at status.gfsc.studio — anyone can check service health without needing an account.

# Single command to deploy on PlaceCal production box:
docker run -d --restart=always -p 3001:3001 \
  -v uptime-kuma:/app/data \
  --name uptime-kuma louislam/uptime-kuma:1

Monitors to configure immediately: - placecal.org (production) - social.gfsc.studio (Mastodon) - lists.gfsc.community (Mailman — currently broken, confirm it) - pad.gfsc.studio (HedgeDoc) - gfsc.community (Ghost blog) - handbook.gfsc.community

Alerts: Discord webhook (no email setup needed). The team gets a ping before users notice.

Status page: Public link. No login required to view. OTP can be added for admin access.

Layer 2: Server health metrics — 3 hours to deploy, zero ongoing cost¶

Node Exporter + cAdvisor on each server → Grafana Cloud free tier.

Grafana Cloud free tier: 10,000 metric series, 50GB logs, 14 days retention. Cost: £0.

What this shows: - CPU, RAM, disk per server — see if anything is running hot - Per-container RAM and restart count — a container restarting repeatedly is a service silently failing - Disk fill rate — catch a full disk before it kills a database

Shared dashboard: Grafana supports sharing dashboards via a public read-only link, or with email + OTP. No Grafana account needed to view.

What monitoring answers without asking anyone¶

• Is anything actually broken right now? (Uptime Kuma answers this immediately)
• How much spare capacity does each server have? (Justifies consolidation decisions)
• Is any container crashlooping silently? (cAdvisor restart counter)
• Is disk space a problem on shaw? (Node Exporter)

Deploy monitoring first. It costs nothing, takes a few hours, and gives real data to inform every other decision.

Recommended Sequence¶

This is a provisional sequence — steps from Week 2 onwards depend on what the information gathering and monitoring reveal.

Step 1 — Information gathering + monitoring (prerequisite for everything else)
  ├── Confirm costs: get actual Hetzner invoice, shaw billing arrangement
  ├── Confirm shaw spec and current management arrangement
  ├── Get SSH access to all boxes
  ├── Deploy Uptime Kuma on PlaceCal production box
  ├── Deploy Node Exporter + cAdvisor on all boxes → Grafana Cloud
  └── Get answers to service usage questions (Mailman, HedgeDoc, n8n, Listmonk)

Step 2 — Decide Q4 direction (needs Step 1 complete)
  └── Review shaw arrangement and monitoring data → choose Option A or B

Step 3 — Ghost migration (early quick win, once server capacity confirmed)
  ├── Confirm which server has headroom from monitoring data
  ├── Provision Ghost as Docker container on that server
  ├── Mirror: export content, import, test
  ├── Switch: lower DNS TTL, cut over, monitor 24–48 hours
  └── Cancel Digital Ocean droplet (~$12/mo saving)
  Note: this also serves as a first run of the mirror/switch method before
  applying it to more complex services

Step 4 — shaw audit and cleanup (parallel with Step 3)
  ├── Confirm status of n8n and Listmonk — preserve anything active, then remove
  ├── Document Mastodon runbook — update to current version if behind
  ├── Decide Mailman: fix or retire based on actual usage data
  └── Decide HedgeDoc: fix or retire based on actual usage data

Step 5 — Review and stabilise
  ├── Confirm all services have runbooks
  ├── Confirm monitoring covers all boxes and services
  ├── Review shaw spec against actual usage — downsize if headroom allows
  └── Confirm credential audit is complete

Related¶

• Current Infrastructure — Full service map with access details
• Infrastructure Vision — Why simplify, and what the end state looks like