0

New? Questions? Want to donate? Check out our new help page!

preview

Myself, @MadPsyentist@lemmy.nz and @terraborra@lemmy.nz have been working on a new help site for LemmyNZ. Instead of various Lemmy posts to explain rules, donations, and FAQs, we now have:

help.lemmy.nz

On this site, we have a Join LemmyNZ page that I hope does a better job of onboarding than a registration page.

I’ve updated various links around LemmyNZ to point to pages on this site instead of the posts that previous had the content.

If you have any feedback, let’s hear it!

Myself, @MadPsyentist@lemmy.nz and @terraborra@lemmy.nz have been working on a new help site for LemmyNZ. Instead of various Lemmy posts to explain rules, donations, and FAQs, we now have:

help.lemmy.nz

On this site, we have a Join LemmyNZ page that I hope does a better job of onboarding than a registration page.

I’ve updated various links around LemmyNZ to point to pages on this site instead of the posts that previous had the content.

If you have any feedback, let’s hear it!

0

LemmyNZ financial update July - September 2026

preview

Welcome back for our next financial update. See the previous one here.

This quarter we moved to a dedicated server at $45USD per month, still with Zappie Host in their Auckland data centre. This dedicated server is significantly more powerful, having:

  • CPU: E3-1270v5
  • Ram: 64GB
  • Disk: 2 x 1TB SSD (in RAID 1 configuration for redundancy, so just 1TB usable)
  • Traffic: 10TB

Compared to the previous shared server:

  • 4 vCPU
  • RAM: 9GB
  • Disk: 160GB
  • Traffic: 1TB

We also stood up piefed.nz running the PieFed software, as an alternative to Lemmy (but they federate, so they generally see the same content, with some differences). In addition there was also threadiverse.nz created as a shared help resource for LemmyNZ and PiefedNZ. Piefed.nz was registered way back in February as I’ve had the idea floating around for a while, so I haven’t included that cost. But I have included the Threadiverse.nz domain registration as that was done in September. Lemmy.nz was paid up for a couple of years last year so no cost for that this year.

Expenses for July - September 2026: $216.82

The expenses are made up of:

  • VPS: $155.26 (two months of hosting, the other month was covered by the previous 6 months prepaid)
  • Image and backup storage: $17.29
  • Domains: $44.27

Expenses are allocated to the period they were paid, not to the period they cover. This is based on the actual amount charged to my card.

Contributions in July - September 2026: $559.82

This is huge! Thanks everyone 🙂

In addition to this, I’ve done a square up on donations reported in the platform vs cash out, and there’s an additional $22.83. The platform removed fees, but also not everyone used NZD so there was actually more paid out than previously reported. This month’s report is based on the cashed out value, which I will try to use going forward.

The main reason for this change is that we got a bunch of donations in other currencies which made it tricky to report on, so I have cashed out and got it all into actual cash in hand I can use for paying for the server.

We received donations totalling $559.82 this quarter, with much of that coming in a flurry after the last financial update showed a shortfall. Thanks to everyone who contributed!

Overall position: $187.97

  • Opening balance: $-177.86
  • Contributions: $559.82
  • Expenses: -$216.82
  • Adjustment for cash out difference: $22.83
  • Total: $187.97
  • Future costs covered by cash on hand: approx 2 months

Numbers are in NZD.

Donations

If you want to contribute to the cost of running LemmyNZ and PieFedNZ, see the page here.

The usual caveat: Please don’t contribute if it will impact you financially. I don’t want to see anyone saying “I only had $5 in my bank account but Lemmy is important so I donated my last $5” like I’ve seen on other instances. If you can’t afford it then don’t feel bad, this round is on me. I don’t mind, truly. But if you can easily chip in a few bucks a month, then it is appreciated 🙂

Pātai

As usual, feel free to ask any questions!

Welcome back for our next financial update. See the previous one here.

This quarter we moved to a dedicated server at $45USD per month, still with Zappie Host in their Auckland data centre. This dedicated server is significantly more powerful, having:

  • CPU: E3-1270v5
  • Ram: 64GB
  • Disk: 2 x 1TB SSD (in RAID 1 configuration for redundancy, so just 1TB usable)
  • Traffic: 10TB

Compared to the previous shared server:

  • 4 vCPU
  • RAM: 9GB
  • Disk: 160GB
  • Traffic: 1TB

We also stood up piefed.nz running the PieFed software, as an alternative to Lemmy (but they federate, so they generally see the same content, with some differences). In addition there was also threadiverse.nz created as a shared help resource for LemmyNZ and PiefedNZ. Piefed.nz was registered way back in February as I’ve had the idea floating around for a while, so I haven’t included that cost. But I have included the Threadiverse.nz domain registration as that was done in September. Lemmy.nz was paid up for a couple of years last year so no cost for that this year.

Expenses for July - September 2026: $216.82

The expenses are made up of:

  • VPS: $155.26 (two months of hosting, the other month was covered by the previous 6 months prepaid)
  • Image and backup storage: $17.29
  • Domains: $44.27

Expenses are allocated to the period they were paid, not to the period they cover. This is based on the actual amount charged to my card.

Contributions in July - September 2026: $559.82

This is huge! Thanks everyone 🙂

In addition to this, I’ve done a square up on donations reported in the platform vs cash out, and there’s an additional $22.83. The platform removed fees, but also not everyone used NZD so there was actually more paid out than previously reported. This month’s report is based on the cashed out value, which I will try to use going forward.

The main reason for this change is that we got a bunch of donations in other currencies which made it tricky to report on, so I have cashed out and got it all into actual cash in hand I can use for paying for the server.

We received donations totalling $559.82 this quarter, with much of that coming in a flurry after the last financial update showed a shortfall. Thanks to everyone who contributed!

Overall position: $187.97

  • Opening balance: $-177.86
  • Contributions: $559.82
  • Expenses: -$216.82
  • Adjustment for cash out difference: $22.83
  • Total: $187.97
  • Future costs covered by cash on hand: approx 2 months

Numbers are in NZD.

Donations

If you want to contribute to the cost of running LemmyNZ and PieFedNZ, see the page here.

The usual caveat: Please don’t contribute if it will impact you financially. I don’t want to see anyone saying “I only had $5 in my bank account but Lemmy is important so I donated my last $5” like I’ve seen on other instances. If you can’t afford it then don’t feel bad, this round is on me. I don’t mind, truly. But if you can easily chip in a few bucks a month, then it is appreciated 🙂

Pātai

As usual, feel free to ask any questions!

0

You can now join the LemmyNZ/PieFedNZ chat room from Discord

preview

Hi all, this is just a quick update to let you know I’ve set up a Discord channel that is bridged to our Matrix chat.

For a long time we’ve had a Matrix chat room, many people joined in the early days but over time that fell off and became just a handful of regulars. Matrix isn’t exactly common, and many users are only using Matrix for Lemmy and they have Discord for their other groups. So I’ve now set up a bridge so you can message to the Discord channel and it will appear in the Matrix room, and vice versa.

Check out our help page on the chatroom, or click these direct links to join:

Join on Matrix by clicking here

Join on Discord by clicking here

Hi all, this is just a quick update to let you know I’ve set up a Discord channel that is bridged to our Matrix chat.

For a long time we’ve had a Matrix chat room, many people joined in the early days but over time that fell off and became just a handful of regulars. Matrix isn’t exactly common, and many users are only using Matrix for Lemmy and they have Discord for their other groups. So I’ve now set up a bridge so you can message to the Discord channel and it will appear in the Matrix room, and vice versa.

Check out our help page on the chatroom, or click these direct links to join:

Join on Matrix by clicking here

Join on Discord by clicking here

0

LemmyNZ will have downtime this Friday from 8:30pm-ish until the job is done, approx 2hrs - we are moving to a new server (it's a good thing!)

preview

TL;DR, moving to a dedicated server. Also some updates later in this post.

Since we moved to a VPS in February, I’ve been closely monitoring the resources of the server. I’ve kept just ahead of it, with our disk usage around 90%, RAM usage around 90%, and using 90% or more of our monthly bandwidth allocation. This was deliberate, we don’t want to pay more than we need to.

I’ve added a little RAM or disk space occasionally, and this time when I opened a ticket to request some more, they said “hey, so we have this deal at the moment”. One thing led to another, and now we’ll have a dedicated server for $45 USD a month, still in the Auckland data centre.

The specs are as follows:

CPU: E3-1270v5
Ram: 64GB
Disk: 2 x 1TB SSD
Traffic: 10TB

It’s much more powerful than what we have currently:

4 vCPU
9GB RAM
160GB SSD
Traffic: 1TB

I have already transferred over everything except Lemmy itself. So all the alternative frontends, our help pages, and the survey software now run off the new server.

Also, currently we have almost entirely moved off of Cloudflare. Currently they are still providing the DNS nameservers but there is no proxying or caching happening anymore, and we seem to be doing ok. I’ll be looking at that last step at some point, but it’s awfully convenient having a Cloudflare API to automatically set up TLS for any new subdomains so I have some investigation to do there. Anubis appears to be able to stop some 90% of bot scraping traffic, which I’m pretty happy with since we don’t have the Cloudflare blocking rules anymore.

So anyway, the plan is about 8:30pm this Friday I’ll take down Lemmy, back up the database, transfer it across and load it into the new server. This can take some time, expect an hour or two of downtime if everything goes well. If you want to follow along, join the LemmyNZ Matrix chat room.

Feel free to ask any questions about anything and I’ll do my best to answer!

TL;DR, moving to a dedicated server. Also some updates later in this post.

Since we moved to a VPS in February, I’ve been closely monitoring the resources of the server. I’ve kept just ahead of it, with our disk usage around 90%, RAM usage around 90%, and using 90% or more of our monthly bandwidth allocation. This was deliberate, we don’t want to pay more than we need to.

I’ve added a little RAM or disk space occasionally, and this time when I opened a ticket to request some more, they said “hey, so we have this deal at the moment”. One thing led to another, and now we’ll have a dedicated server for $45 USD a month, still in the Auckland data centre.

The specs are as follows:

CPU: E3-1270v5
Ram: 64GB
Disk: 2 x 1TB SSD
Traffic: 10TB

It’s much more powerful than what we have currently:

4 vCPU
9GB RAM
160GB SSD
Traffic: 1TB

I have already transferred over everything except Lemmy itself. So all the alternative frontends, our help pages, and the survey software now run off the new server.

Also, currently we have almost entirely moved off of Cloudflare. Currently they are still providing the DNS nameservers but there is no proxying or caching happening anymore, and we seem to be doing ok. I’ll be looking at that last step at some point, but it’s awfully convenient having a Cloudflare API to automatically set up TLS for any new subdomains so I have some investigation to do there. Anubis appears to be able to stop some 90% of bot scraping traffic, which I’m pretty happy with since we don’t have the Cloudflare blocking rules anymore.

So anyway, the plan is about 8:30pm this Friday I’ll take down Lemmy, back up the database, transfer it across and load it into the new server. This can take some time, expect an hour or two of downtime if everything goes well. If you want to follow along, join the LemmyNZ Matrix chat room.

Feel free to ask any questions about anything and I’ll do my best to answer!

0

LemmyNZ financial update April - June 2026

preview

Welcome back for our second financial update. See the first one here.

This quarter we added a little bit of hard drive space and 1GB of RAM but haven’t been charged for it yet. As I paid for the VPS for 6 months in February, there are no further VPS charges this quarter.

Expenses for April-June 2026: $28.33

The expenses are made up of:

  • VPS: $0
  • Image and backup storage: $28.33

Expenses are allocated to the period they were paid, not to the period they cover. This is based on the actual amount charged to my card.

Contributions in April-June 2026: $140

We received five donations totalling $140 this quarter. Donations are based on the amount the person donated, as listed by Buy Me A Coffee (before their fees). At the end of the year I’ll do a match up to try to match actual payout to donated amount to see what this difference is.

Overall position: -$177.86

  • Opening balance: -$289.53
  • Contributions: $140
  • Expenses: -$28.33
  • Total: -$177.86
  • Percent of expenses paid for by contributions (this year): 55%

An update on our VPS

During the quarter we added 10GB of storage space and 1GB more RAM but otherwise the VPS seems to be sized appropriately. Last time I mentioned bandwidth usage and the potential to move away from Cloudflare. As a step towards this, I have disabled Cloudflare image caching, so all images should come via the server. This could also impact the load speed for geolocation reasons so we will see how this goes.

Donations

If you want to contribute to the cost of running Lemmy.nz, see the post here. Or the direct link: buymeacoffee.com/lemmynz

The usual caveat: Please don’t contribute if it will impact you financially. I don’t want to see anyone saying “I only had $5 in my bank account but Lemmy is important so I donated my last $5” like I’ve seen on other instances. If you can’t afford it then don’t feel bad, this round is on me. I don’t mind, truly. But if you can easily chip in a few bucks a month, then it is appreciated 🙂

Pātai

As usual, feel free to ask any questions!

Welcome back for our second financial update. See the first one here.

This quarter we added a little bit of hard drive space and 1GB of RAM but haven’t been charged for it yet. As I paid for the VPS for 6 months in February, there are no further VPS charges this quarter.

Expenses for April-June 2026: $28.33

The expenses are made up of:

  • VPS: $0
  • Image and backup storage: $28.33

Expenses are allocated to the period they were paid, not to the period they cover. This is based on the actual amount charged to my card.

Contributions in April-June 2026: $140

We received five donations totalling $140 this quarter. Donations are based on the amount the person donated, as listed by Buy Me A Coffee (before their fees). At the end of the year I’ll do a match up to try to match actual payout to donated amount to see what this difference is.

Overall position: -$177.86

  • Opening balance: -$289.53
  • Contributions: $140
  • Expenses: -$28.33
  • Total: -$177.86
  • Percent of expenses paid for by contributions (this year): 55%

An update on our VPS

During the quarter we added 10GB of storage space and 1GB more RAM but otherwise the VPS seems to be sized appropriately. Last time I mentioned bandwidth usage and the potential to move away from Cloudflare. As a step towards this, I have disabled Cloudflare image caching, so all images should come via the server. This could also impact the load speed for geolocation reasons so we will see how this goes.

Donations

If you want to contribute to the cost of running Lemmy.nz, see the post here. Or the direct link: buymeacoffee.com/lemmynz

The usual caveat: Please don’t contribute if it will impact you financially. I don’t want to see anyone saying “I only had $5 in my bank account but Lemmy is important so I donated my last $5” like I’ve seen on other instances. If you can’t afford it then don’t feel bad, this round is on me. I don’t mind, truly. But if you can easily chip in a few bucks a month, then it is appreciated 🙂

Pātai

As usual, feel free to ask any questions!

0

The vote is over, and we have a new logo!

preview

A week back I posted a poll for LemmyNZ users to vote for a new logo.

The results are in.

The winner was the globe by @Manticore@lemmy.nz!

digital painted world sphere with NZ filling it top to bottom and some recognisable countries around the edges

Out of the LemmyNZ users, we had 8 votes for the winning entry, 2 votes for Kimmy, and 1 vote to keep the existing logo. I even made some graphs:

Pie

pie graph showing the split (8 for the winner, 2 for the runner up, 1 for keeping the current logo

Bar

bar graph showing the split (8 for the winner, 2 for the runner up, 1 for keeping the current logo

A week back I posted a poll for LemmyNZ users to vote for a new logo.

The results are in.

The winner was the globe by @Manticore@lemmy.nz!

digital painted world sphere with NZ filling it top to bottom and some recognisable countries around the edges

Out of the LemmyNZ users, we had 8 votes for the winning entry, 2 votes for Kimmy, and 1 vote to keep the existing logo. I even made some graphs:

Pie

pie graph showing the split (8 for the winner, 2 for the runner up, 1 for keeping the current logo

Bar

bar graph showing the split (8 for the winner, 2 for the runner up, 1 for keeping the current logo

0

Vote for a new logo for LemmyNZ [LemmyNZ users only]

preview

Hi all,

TL;DR: If you’re a LemmyNZ user, go and vote for your preferred logo here:

A few weeks back there was some discussion in the weekly kōrero posts about replacing out logo. The logo was AI generated, was done using the really early tools available at that time, and was our logo before there were any users other than me. I’ve always hated it, and have wanted to replace it for a long time.

Of the back of that discussion, I started a vote. I underestimated how much people cared, and so cancelled the vote but learnt that people seem to like the banner (Laser kiwi).

I asked for help in this post, and one user played with some ideas for us (various versions in that post) and has come up with a cool logo with NZ the centre stage of the world. For good measure, I have kept the existing logo and the one from the previous vote in the poll.

I have also left the banner out this time. Last time I had them grouped and it wasn’t clear if people liked the banner or the logo. We don’t have a good suggestion to replace the banner so unless people have suggestions we can leave the laser kiwi.

The images in this vote are:

“Revenge for not being on so many world maps. Now its our turn to be the world map” by @Manticore@lemmy.nz

painted world shere with NZ filling it top to bottom and some recognisable countries around the edges

Kimmy the kiwi/lemmy logo hybrid, that I made and came about from the chat in the weekly kōrero:

kiwi/lemming hybrid thing, basically lemmy logo with kiwi beak and coloured brown like a kiwi. NZ flag in background

And the existing one:

Kind of brown lemming thing with a spread of green lasers glowing from the bottom centre

This is a vote for the logo of the LemmyNZ instance, so please only vote if your account is on LemmyNZ.

You can vote here.

Hi all,

TL;DR: If you’re a LemmyNZ user, go and vote for your preferred logo here:

A few weeks back there was some discussion in the weekly kōrero posts about replacing out logo. The logo was AI generated, was done using the really early tools available at that time, and was our logo before there were any users other than me. I’ve always hated it, and have wanted to replace it for a long time.

Of the back of that discussion, I started a vote. I underestimated how much people cared, and so cancelled the vote but learnt that people seem to like the banner (Laser kiwi).

I asked for help in this post, and one user played with some ideas for us (various versions in that post) and has come up with a cool logo with NZ the centre stage of the world. For good measure, I have kept the existing logo and the one from the previous vote in the poll.

I have also left the banner out this time. Last time I had them grouped and it wasn’t clear if people liked the banner or the logo. We don’t have a good suggestion to replace the banner so unless people have suggestions we can leave the laser kiwi.

The images in this vote are:

“Revenge for not being on so many world maps. Now its our turn to be the world map” by @Manticore@lemmy.nz

painted world shere with NZ filling it top to bottom and some recognisable countries around the edges

Kimmy the kiwi/lemmy logo hybrid, that I made and came about from the chat in the weekly kōrero:

kiwi/lemming hybrid thing, basically lemmy logo with kiwi beak and coloured brown like a kiwi. NZ flag in background

And the existing one:

Kind of brown lemming thing with a spread of green lasers glowing from the bottom centre

This is a vote for the logo of the LemmyNZ instance, so please only vote if your account is on LemmyNZ.

You can vote here.

0

Help us get a new LemmyNZ logo!

preview

Way back at the beginning, when genAI was new and LemmyNZ had but one user, our logo was created. It’s terrible, it’s AI generated, and I hate it.

lemmyNZ logo - kind of a 3d lemmyish creature with green rays coming out, with an AI generated vibe

I am keen to get a new one! There’s also an option to get a new banner if that’s what people want, though people have expressed that they like the current banner.

lemmyNZ banner - the famous Laser kiwi flag that was submitted to the flag competition when NZ tried to change it’s flag. It’s black with a silver fern at top left, and a kiwi shape on the bottom right with a green laser coming out of it’s eyes down to just above the bottom left corner of the flag

This post is to ask for people to supply their ideas for a new logo or banner. Hopefully there are some artists out there that can help! After we have some options I will organise a vote.

Reply to this post with your suggestions or your submissions.

Way back at the beginning, when genAI was new and LemmyNZ had but one user, our logo was created. It’s terrible, it’s AI generated, and I hate it.

lemmyNZ logo - kind of a 3d lemmyish creature with green rays coming out, with an AI generated vibe

I am keen to get a new one! There’s also an option to get a new banner if that’s what people want, though people have expressed that they like the current banner.

lemmyNZ banner - the famous Laser kiwi flag that was submitted to the flag competition when NZ tried to change it’s flag. It’s black with a silver fern at top left, and a kiwi shape on the bottom right with a green laser coming out of it’s eyes down to just above the bottom left corner of the flag

This post is to ask for people to supply their ideas for a new logo or banner. Hopefully there are some artists out there that can help! After we have some options I will organise a vote.

Reply to this post with your suggestions or your submissions.

0

We're now running Lemmy 0.19.19

preview

Just a quick note to say I’ve updated LemmyNZ to run 0.19.19 today.

It was only just released, and only resolves security issues, so you shouldn’t see any changes. Release notes here.

Just a quick note to say I’ve updated LemmyNZ to run 0.19.19 today.

It was only just released, and only resolves security issues, so you shouldn’t see any changes. Release notes here.

0

LemmyNZ Logo update vote [Cancelling for now]

preview

UPDATE: While the new logo is marginally winning, I wasn’t expecting this much interest. I also regret grouping the logo and banner instead of asking separately as many people seem to like the existing laser kiwi banner. I’ll try to drum up some more suggestions.

Hi all! We’ve had the crappy logo since LemmyNZ started, and I’ve had on my list to update it. With a bit of discussion in the weekly chats, I now have a proposed replacement.

Please vote on the new logo to see if we will adopt it as the LemmyNZ logo.

Vote here

UPDATE: While the new logo is marginally winning, I wasn’t expecting this much interest. I also regret grouping the logo and banner instead of asking separately as many people seem to like the existing laser kiwi banner. I’ll try to drum up some more suggestions.

Hi all! We’ve had the crappy logo since LemmyNZ started, and I’ve had on my list to update it. With a bit of discussion in the weekly chats, I now have a proposed replacement.

Please vote on the new logo to see if we will adopt it as the LemmyNZ logo.

Vote here

0

Update to Lemmy 0.19.18, issues with image proxy, and your thoughts please

preview

Hi all,

TL;DR at bottom

Tonight I’ve noticed some issues loading images. Any images already proxied work fine, but new images were failing to load at all. Not sure how long it’s been happening for today, but this led to a pretty shit experience browsing Lemmy 🫤

You might not be aware, but most images are run through the proxy at lemmyNZ so that when you browse, you’re only loading images direct from Lemmy (other than a few exceptions, such as imgur, which blocks this and is also commonly used).

The main benefit to this is privacy. Without it, images are loaded direct from the site. This means as you scroll your feed, all those sites know you specifically (or at least your IP address) are seeing their content in your feed on Lemmy. By proxying the images, only LemmyNZ is seen as the one loading the image then it passes it on to you.

Lemmy does support only proxying thumbnails, so other sites are not called unless you click on something, but for some reason Lemmy NEVER deletes the thumbnail, and your storage just fills up and up with no way to clear it.

So tonight I’ve been trying to troubleshoot. I have upgraded Lemmy to the latest version (relase notes for 0.19.17 here, and 0.19.18 here. Only minor changes), updated the image version to the latest version, and restarted everything a few times. As of writing it seems ok, but I do want to know people’s thoughts on this.

Do people like to have the images proxied so sites aren’t tracking them? Or should we prioritise stability, where images are more likely to load consistently and have a better browsing experience if we load them from the site directly?

If the image proxy stays stable going forward, we could still consider an exception for images loaded directly from other Lemmy instances, which could reduce load a bit while still protecting from most tracking.

It’s worth noting if we want to move off Cloudflare, not proxying much will likely help with speed, but we’ll also be loading a lot of stuff off Cloudflare still from the other sites using it. I say it will likely help with speed because images are currently stored in an EU data centre, they load relatively quick because Cloudflare caches more locally. If we leave Cloudflare then images may have a longer delay to load.

TL;DR Image proxy in Lemmy has been unstable tonight so you might have noticed not being able to see any images. Curious on people’s thoughts on continuing to use it:

  1. Keep proxying as much as possible even at the expense of some failing images
  2. Proxy most content but add some exceptions for content loaded direct from other Lemmy instances, and some common big image hosts
  3. Proxy nothing
  4. Other

Thoughts?

Hi all,

TL;DR at bottom

Tonight I’ve noticed some issues loading images. Any images already proxied work fine, but new images were failing to load at all. Not sure how long it’s been happening for today, but this led to a pretty shit experience browsing Lemmy 🫤

You might not be aware, but most images are run through the proxy at lemmyNZ so that when you browse, you’re only loading images direct from Lemmy (other than a few exceptions, such as imgur, which blocks this and is also commonly used).

The main benefit to this is privacy. Without it, images are loaded direct from the site. This means as you scroll your feed, all those sites know you specifically (or at least your IP address) are seeing their content in your feed on Lemmy. By proxying the images, only LemmyNZ is seen as the one loading the image then it passes it on to you.

Lemmy does support only proxying thumbnails, so other sites are not called unless you click on something, but for some reason Lemmy NEVER deletes the thumbnail, and your storage just fills up and up with no way to clear it.

So tonight I’ve been trying to troubleshoot. I have upgraded Lemmy to the latest version (relase notes for 0.19.17 here, and 0.19.18 here. Only minor changes), updated the image version to the latest version, and restarted everything a few times. As of writing it seems ok, but I do want to know people’s thoughts on this.

Do people like to have the images proxied so sites aren’t tracking them? Or should we prioritise stability, where images are more likely to load consistently and have a better browsing experience if we load them from the site directly?

If the image proxy stays stable going forward, we could still consider an exception for images loaded directly from other Lemmy instances, which could reduce load a bit while still protecting from most tracking.

It’s worth noting if we want to move off Cloudflare, not proxying much will likely help with speed, but we’ll also be loading a lot of stuff off Cloudflare still from the other sites using it. I say it will likely help with speed because images are currently stored in an EU data centre, they load relatively quick because Cloudflare caches more locally. If we leave Cloudflare then images may have a longer delay to load.

TL;DR Image proxy in Lemmy has been unstable tonight so you might have noticed not being able to see any images. Curious on people’s thoughts on continuing to use it:

  1. Keep proxying as much as possible even at the expense of some failing images
  2. Proxy most content but add some exceptions for content loaded direct from other Lemmy instances, and some common big image hosts
  3. Proxy nothing
  4. Other

Thoughts?

0

Introducing two new admins: terraborra and MadPsyentist

preview

Hi again everyone. Off the back of this post, I have appointed two new admins to help.

Both have been around for some time and may be familiar faces to you: @terraborra@lemmy.nz and @MadPsyentist@lemmy.nz

I’m not expecting any sudden changes in anything but wanted to let everyone know.

Hi again everyone. Off the back of this post, I have appointed two new admins to help.

Both have been around for some time and may be familiar faces to you: @terraborra@lemmy.nz and @MadPsyentist@lemmy.nz

I’m not expecting any sudden changes in anything but wanted to let everyone know.

0

Expressions of interest to be a helping hand around here

preview

TL;DR looking for more people to volunteer as admins for various technical and non-technical roles

Every since we moved off of the piggy-backed hosting onto our own VPS, a big concern of mine has been that I’m now a single point of failure across many parts of LemmyNZ. Unfortunately I sometimes go near busses, which makes the chances of me being hit by a bus a realistic possibility.

There are a few specific things I’d like more people involved with, which don’t all necessarily require technical skills:

  • Admins to help with having others available to resolve reports and monitor Matrix groups for reported issues (Lemmy has no global-level moderator role, you literally have to be an admin to remove any content posted to non-LemmyNZ communities)
  • Self-hosters familiar with docker compose to help with hands on server management stuff
  • Help with setting up some other content, such as a join.lemmy.nz page to help with user on-boarding and perhaps other helpful instructions for users
  • Help from someone with art skills to replace that shitty green logo we’ve had since day one

If others have suggestions on picking people, I’m happy to take input. To be considered for roles with elevated permission, I think you should meet the following criteria:

  • Must be a LemmyNZ user and have a history as a LemmyNZ user - not a newly joined account
  • Must not be a controversial decision to grant the user this special access. The average LemmyNZ user should not be concerned with the user being appointed.
  • Must be an active user. To fill a gap we need people who are generally around on Lemmy. This doesn’t necessarily mean they have to be posting or commenting, but there should be enough of a trail to gauge a user’s controversial-ness.
  • Must use the chat platform Matrix - you don’t need a history of using it but Lemmy runs on a collaboration between instances, and this collaboration happens across a range of Matrix chats. Therefore we need people willing to collaborate on Matrix and also monitor the admin chats.

If you’re interested in helping and think you fit the criteria, reply below. Also feel free to tag other users if you think they would be a good fit, but no one is obligated to do anything. Please message me if you have concerns about any users in particular.

For all users, note that email addresses are visible to admins who know where to look, which will include any newly appointed admins. If you’re concerned with this, you can change your email address or remove it before other admins are appointed. You can also freely use a relay email service that forwards emails to your main email address, such as SimpleLogin, Firefox Relay, or AnonAddy. Some level of IP logging is also done by Lemmy, you’re free to use a VPN.

TL;DR looking for more people to volunteer as admins for various technical and non-technical roles

Every since we moved off of the piggy-backed hosting onto our own VPS, a big concern of mine has been that I’m now a single point of failure across many parts of LemmyNZ. Unfortunately I sometimes go near busses, which makes the chances of me being hit by a bus a realistic possibility.

There are a few specific things I’d like more people involved with, which don’t all necessarily require technical skills:

  • Admins to help with having others available to resolve reports and monitor Matrix groups for reported issues (Lemmy has no global-level moderator role, you literally have to be an admin to remove any content posted to non-LemmyNZ communities)
  • Self-hosters familiar with docker compose to help with hands on server management stuff
  • Help with setting up some other content, such as a join.lemmy.nz page to help with user on-boarding and perhaps other helpful instructions for users
  • Help from someone with art skills to replace that shitty green logo we’ve had since day one

If others have suggestions on picking people, I’m happy to take input. To be considered for roles with elevated permission, I think you should meet the following criteria:

  • Must be a LemmyNZ user and have a history as a LemmyNZ user - not a newly joined account
  • Must not be a controversial decision to grant the user this special access. The average LemmyNZ user should not be concerned with the user being appointed.
  • Must be an active user. To fill a gap we need people who are generally around on Lemmy. This doesn’t necessarily mean they have to be posting or commenting, but there should be enough of a trail to gauge a user’s controversial-ness.
  • Must use the chat platform Matrix - you don’t need a history of using it but Lemmy runs on a collaboration between instances, and this collaboration happens across a range of Matrix chats. Therefore we need people willing to collaborate on Matrix and also monitor the admin chats.

If you’re interested in helping and think you fit the criteria, reply below. Also feel free to tag other users if you think they would be a good fit, but no one is obligated to do anything. Please message me if you have concerns about any users in particular.

For all users, note that email addresses are visible to admins who know where to look, which will include any newly appointed admins. If you’re concerned with this, you can change your email address or remove it before other admins are appointed. You can also freely use a relay email service that forwards emails to your main email address, such as SimpleLogin, Firefox Relay, or AnonAddy. Some level of IP logging is also done by Lemmy, you’re free to use a VPN.

0

LemmyNZ financial update Jan-Mar 2026 quarter

preview

Hi all, this is (I think) the first Lemmy.nz financial update! In case you missed it, we are no longer piggybacking on someone else’s hosting and now have our own virtual server.

Now that we have to pay our own way, I am going to start doing semi-regular updates about how things are going. I am thinking quarterly updates.

This quarter we have migrated to a new VPS and moved images to object storage, each of which has an ongoing cost. Other potential costs are email sending (which to date I don’t believe we have ever crossed over the free sending threshold for Mailgun) and the domain registration, which is already paid up for a couple of years.

Expenses for January-March 2026: $369.53

The expenses are made up of:

  • VPS: $358.10
  • Image and backup storage: $11.43

Expenses are allocated to the period they were paid, not to the period they cover. The VPS expense covers the first month, plus a further 6 months, which makes this look a lot more expensive than it should. The image and backup storage only includes a partial month for February and does not include March as March will be billed in April based on usage.

Contributions in January-March 2026: $35

We received two donations totalling $35 this quarter.

Overall position: -$289.53

This ending balance is mostly because I have paid for hosting 6 months in advance. I started collecting donations late last year, and we had $45 donated, so we actually started with $45 at the start of the quarter.

  • Opening balance: $45
  • Contributions: $35
  • Expenses: -$369.53
  • Total: -$289.53
  • Percent of expenses paid for by contributions: 21%

An update on our move to the VPS

We have now had over a month on the VPS. The specs we have seem to be appropriate with RAM constantly at 80% use, the CPU peaks at an occasional 90% but averaging around 40%, and the hard drive storage at about 83% which may need a little more space soon, adding another dollar or two a month.

In March we cut it pretty close on bandwidth usage as I am using a new backup tool and didn’t have compression set up right initially, which burnt through a lot of data in the first week. So I have been carefully rationing what is used for offsite transfer to keep us within the 1TB allocation for this month, however, it does feel like we could move away from Cloudflare’s caching and still be within the 1TB in the future. I will let it run all April and see what sort of usage we have.

Donations

If you want to contribute to the cost of running Lemmy.nz, see the post here.

The usual caveat: Please don’t contribute if it will impact you financially. I don’t want to see anyone saying “I only had $5 in my bank account but Lemmy is important so I donated my last $5” like I’ve seen on other instances. If you can’t afford it then don’t feel bad, this round is on me. I don’t mind, truly. But if you can easily chip in a few bucks a month, then it is appreciated 🙂

Pātai

As usual, feel free to ask any questions!

Hi all, this is (I think) the first Lemmy.nz financial update! In case you missed it, we are no longer piggybacking on someone else’s hosting and now have our own virtual server.

Now that we have to pay our own way, I am going to start doing semi-regular updates about how things are going. I am thinking quarterly updates.

This quarter we have migrated to a new VPS and moved images to object storage, each of which has an ongoing cost. Other potential costs are email sending (which to date I don’t believe we have ever crossed over the free sending threshold for Mailgun) and the domain registration, which is already paid up for a couple of years.

Expenses for January-March 2026: $369.53

The expenses are made up of:

  • VPS: $358.10
  • Image and backup storage: $11.43

Expenses are allocated to the period they were paid, not to the period they cover. The VPS expense covers the first month, plus a further 6 months, which makes this look a lot more expensive than it should. The image and backup storage only includes a partial month for February and does not include March as March will be billed in April based on usage.

Contributions in January-March 2026: $35

We received two donations totalling $35 this quarter.

Overall position: -$289.53

This ending balance is mostly because I have paid for hosting 6 months in advance. I started collecting donations late last year, and we had $45 donated, so we actually started with $45 at the start of the quarter.

  • Opening balance: $45
  • Contributions: $35
  • Expenses: -$369.53
  • Total: -$289.53
  • Percent of expenses paid for by contributions: 21%

An update on our move to the VPS

We have now had over a month on the VPS. The specs we have seem to be appropriate with RAM constantly at 80% use, the CPU peaks at an occasional 90% but averaging around 40%, and the hard drive storage at about 83% which may need a little more space soon, adding another dollar or two a month.

In March we cut it pretty close on bandwidth usage as I am using a new backup tool and didn’t have compression set up right initially, which burnt through a lot of data in the first week. So I have been carefully rationing what is used for offsite transfer to keep us within the 1TB allocation for this month, however, it does feel like we could move away from Cloudflare’s caching and still be within the 1TB in the future. I will let it run all April and see what sort of usage we have.

Donations

If you want to contribute to the cost of running Lemmy.nz, see the post here.

The usual caveat: Please don’t contribute if it will impact you financially. I don’t want to see anyone saying “I only had $5 in my bank account but Lemmy is important so I donated my last $5” like I’ve seen on other instances. If you can’t afford it then don’t feel bad, this round is on me. I don’t mind, truly. But if you can easily chip in a few bucks a month, then it is appreciated 🙂

Pātai

As usual, feel free to ask any questions!

0

LemmyNZ users - it's been 2 weeks since we moved to the new VPS, how's it going?

preview

Two weeks ago we moved to a new VPS, less powerful than the previous server but we have to pay now so it is what it is.

We also recently implemented Anubis, which should have cut the amount of processing power wasted on AI scraper bots.

I’m interested in knowing how people have found it.

Is it feeling sluggish or similar to the last server?

Images are now in object storage instead of on the hard drive, do you notice a difference in loading speed?

Is anyone still having trouble with Anubis and getting stuck on the success screen or have those issues all been resolved?

Any other feedback on the new setup?

Two weeks ago we moved to a new VPS, less powerful than the previous server but we have to pay now so it is what it is.

We also recently implemented Anubis, which should have cut the amount of processing power wasted on AI scraper bots.

I’m interested in knowing how people have found it.

Is it feeling sluggish or similar to the last server?

Images are now in object storage instead of on the hard drive, do you notice a difference in loading speed?

Is anyone still having trouble with Anubis and getting stuck on the success screen or have those issues all been resolved?

Any other feedback on the new setup?

0

Lemmy.nz is now using Anubis for certain requests

preview

Earlier in the series:

TL;DR of this post: We now have Anubis on Lemmy.nz, which helps protect against scrapers.

Hi everyone! As some have noticed, the site has been slow these past few days, which was due to a massive amount of traffic presumed to be from AI scraping bots training LLMs. I posted this elsewhere but will put it here again for good measure (7 day history): unique visitors graph showing seven days of graph, no scale but a total of 4.53 million unique visitors, with the last three days being a huge increase

Yeah so that peak of the graph is over 2 million unique requests that day, and the earlier part of the week has about 100k a day. So a 20x increase in traffic.

You may have also seen some weird stuff over this weekend as I played with how to get Anubis inserted into our Lemmy setup, which wasn’t too straightforward with our multitude of frontends and our load balanced Lemmy-UI. You might have seen errors, apps might not have been able to load due to being incorrectly blocked, or you might not have been able to load the site at all. Apologies for this, but I think I have it sorted now.

Running under the assumption that the scrapers are not directly targeting Lemmy but are just scraping everything on the net and we just happen to have a significant amount of content, Anubis has been added only to the front ends (websites you see accessing in a browser), and if you are logged in then the policies are set so Anubis should just pass you straight through without any checks.

The APIs and federation and anything else happening directly with the backend do not go through Anubis. This means apps should not be affected and loading images should not be affected.

I’ve already noticed a big difference in CPU usage. It went from periods of time averaging over 150% (as in, the CPU couldn’t keep up, and you’d see Lemmy fail to load at these times), to the last hour peaking at 47% and mostly hovering around 15-25% CPU usage.

Click for image of last 24 hours CPU usage

CPU usage graph of past day showing high CPU usage, then a significant drop over the last few hours

Adding Anubis is a pretty big change that has a decent chance of breaking something so please let me know if you see anything weird!

I’m also happy to give more technical info if anyone wants to know about how it’s implemented, the architecture of our Lemmy setup and how it fits in, or anything else. Just ask!

Earlier in the series:

TL;DR of this post: We now have Anubis on Lemmy.nz, which helps protect against scrapers.

Hi everyone! As some have noticed, the site has been slow these past few days, which was due to a massive amount of traffic presumed to be from AI scraping bots training LLMs. I posted this elsewhere but will put it here again for good measure (7 day history): unique visitors graph showing seven days of graph, no scale but a total of 4.53 million unique visitors, with the last three days being a huge increase

Yeah so that peak of the graph is over 2 million unique requests that day, and the earlier part of the week has about 100k a day. So a 20x increase in traffic.

You may have also seen some weird stuff over this weekend as I played with how to get Anubis inserted into our Lemmy setup, which wasn’t too straightforward with our multitude of frontends and our load balanced Lemmy-UI. You might have seen errors, apps might not have been able to load due to being incorrectly blocked, or you might not have been able to load the site at all. Apologies for this, but I think I have it sorted now.

Running under the assumption that the scrapers are not directly targeting Lemmy but are just scraping everything on the net and we just happen to have a significant amount of content, Anubis has been added only to the front ends (websites you see accessing in a browser), and if you are logged in then the policies are set so Anubis should just pass you straight through without any checks.

The APIs and federation and anything else happening directly with the backend do not go through Anubis. This means apps should not be affected and loading images should not be affected.

I’ve already noticed a big difference in CPU usage. It went from periods of time averaging over 150% (as in, the CPU couldn’t keep up, and you’d see Lemmy fail to load at these times), to the last hour peaking at 47% and mostly hovering around 15-25% CPU usage.

Click for image of last 24 hours CPU usage

CPU usage graph of past day showing high CPU usage, then a significant drop over the last few hours

Adding Anubis is a pretty big change that has a decent chance of breaking something so please let me know if you see anything weird!

I’m also happy to give more technical info if anyone wants to know about how it’s implemented, the architecture of our Lemmy setup and how it fits in, or anything else. Just ask!

0

Downtime for Lemmy.nz this Friday (again) to move to a new VPS

preview

Earlier in the series:

TL;DR of this post:

  • We will move to a new VPS this weekend
  • Lemmy.nz will be down (properly down) from sometime after 7:30PM Friday until the job is done. This should be completed within a few hours but I reserve the right to use Saturday as well 🙂
  • Initial estimates are that the VPS will be around NZD $50/mo, on top of image storage costs (hard to guess - perhaps $10/mo)

I have now confirmed a plan with a VPS host Zappie, in their Auckland data centre. This is in fact the same place the existing server is colocated.

The specs of the new VPS:

CPU: 4 Cores Ram: 8GB Disk: 150GB Monthly Traffic: 1TB

We may need to adjust things as we go. Initially I will keep our current setup (including Cloudflare), but as we understand better what our needs are I am keen to move off Cloudflare as there was quite a lot of support for that move.

I have copied things across to test and it runs Lemmy well, similar to the current server, though that’s without users, federation, and rogue AI scrapers but Zappie can bump specs pretty quick if needed.

Please let me know if you have any questions, I’m always happy to provide more details 🙂

Earlier in the series:

TL;DR of this post:

  • We will move to a new VPS this weekend
  • Lemmy.nz will be down (properly down) from sometime after 7:30PM Friday until the job is done. This should be completed within a few hours but I reserve the right to use Saturday as well 🙂
  • Initial estimates are that the VPS will be around NZD $50/mo, on top of image storage costs (hard to guess - perhaps $10/mo)

I have now confirmed a plan with a VPS host Zappie, in their Auckland data centre. This is in fact the same place the existing server is colocated.

The specs of the new VPS:

CPU: 4 Cores Ram: 8GB Disk: 150GB Monthly Traffic: 1TB

We may need to adjust things as we go. Initially I will keep our current setup (including Cloudflare), but as we understand better what our needs are I am keen to move off Cloudflare as there was quite a lot of support for that move.

I have copied things across to test and it runs Lemmy well, similar to the current server, though that’s without users, federation, and rogue AI scrapers but Zappie can bump specs pretty quick if needed.

Please let me know if you have any questions, I’m always happy to provide more details 🙂

0

Object storage migration is complete - images should now be working

preview

In case you missed it, images have been down for the night while I did a migration from the hard drive storage to object storage.

The migration is now complete! Please let me know if you find anything odd with images.

In case you missed it, images have been down for the night while I did a migration from the hard drive storage to object storage.

The migration is now complete! Please let me know if you find anything odd with images.

0

Lemmy.nz will have downtime from this Friday 13th Evening through to Saturday when the job is done, to move to object storage

preview

In case you missed it, I have an update in this post. TL;DR we are losing our host and will need to move to a VPS.

It seems pretty clear that once we have to start paying for storage, it will be very expensive to keep everything on hard drive storage, so we will need to move to storing images on object storage.

Unless anyone has any objections, instead of doing everything in one go, I would like to do the migration to object storage for images in advance of the main move to a VPS in order to better assess what we will need for storage once that part has been offloaded. The plan is to use Backblaze B2 storage, because it’s cheap, reliable, and already used for our backups.

This means we will need two patches of downtime, most likely this weekend and next as we need to be off the current host by the end of the month.

This post is to propose an outage from this Friday after 8pm, through to whenever it is finished on Saturday. I have been doing some testing, and it seems it will be a bit flaky if I try to do the migration too fast. My plan is to set it going the night before, run it overnight, and start Lemmy back up in the morning assuming the migration has finished by then. But there is every chance that it just stops while I’m sleeping and needs to be restarted in the morning, taking most of Saturday.

My current plan is to keep Lemmy itself running during this time, meaning there will be no images but federation will continue so we shouldn’t need to catch up after it’s done.

TL;DR

  • Outage Friday after 8pm through to when it’s done on Saturday
  • Most likely Lemmy text content will remain available but no guarantees
  • Moving image storage to Backblaze B2 - please tell me ASAP if there are objections (American company, but only real alternatives are Cloudflare or big tech and American anyway)
  • A further outage will be required the following weekend to move the actual Lemmy service

In case you missed it, I have an update in this post. TL;DR we are losing our host and will need to move to a VPS.

It seems pretty clear that once we have to start paying for storage, it will be very expensive to keep everything on hard drive storage, so we will need to move to storing images on object storage.

Unless anyone has any objections, instead of doing everything in one go, I would like to do the migration to object storage for images in advance of the main move to a VPS in order to better assess what we will need for storage once that part has been offloaded. The plan is to use Backblaze B2 storage, because it’s cheap, reliable, and already used for our backups.

This means we will need two patches of downtime, most likely this weekend and next as we need to be off the current host by the end of the month.

This post is to propose an outage from this Friday after 8pm, through to whenever it is finished on Saturday. I have been doing some testing, and it seems it will be a bit flaky if I try to do the migration too fast. My plan is to set it going the night before, run it overnight, and start Lemmy back up in the morning assuming the migration has finished by then. But there is every chance that it just stops while I’m sleeping and needs to be restarted in the morning, taking most of Saturday.

My current plan is to keep Lemmy itself running during this time, meaning there will be no images but federation will continue so we shouldn’t need to catch up after it’s done.

TL;DR

  • Outage Friday after 8pm through to when it’s done on Saturday
  • Most likely Lemmy text content will remain available but no guarantees
  • Moving image storage to Backblaze B2 - please tell me ASAP if there are objections (American company, but only real alternatives are Cloudflare or big tech and American anyway)
  • A further outage will be required the following weekend to move the actual Lemmy service
0

The state of Lemmy.nz - some updates and some questions for Lemmy.nz users

preview

As many of you will know, we have been borrowing server hosting, which kept costs very minimal. Many of you donated to the host (thanks!), but a while back they stopped taking donations as they wound down operations including transferring mastodon.nz to new people and gave me notice that we would eventually need to move to other hosting.

That was quite some time back, and now I’ve been notified that the time has finally come. At some point in this month we will swap over to a new host. It will also involve significantly increased costs to run the server (up from almost nothing), and most likely we will move to a VPS off of the current shared but dedicated hardware to more appropriate specs, and move to using object storage for images. All this will likely require some down time, which I will try to give notice of. I’m currently planning for this to be quite significant (perhaps done over the course of a whole weekend), since this would involve migrating all images to object storage which must be done with pict-rs shut down (others have reported 8+ hours for this), as well as transferring all other data to a new VPS, then a slow period of time as it catches up on federation. The second day would be a backup in case something went wrong.

When this move happens, I’ll also start to do regular financial updates. In the past people have been interested in running costs, but they have been minimal out of pocket due to the server hosting being free to me. With the increased costs it makes sense to start regular updates about what it costs to run the instance, similar to what other instances do. There is a very wide range in what this could cost, and it’s not clear right now what the lowest spec server is that we could get away with. But I am hoping we can keep this under $50 a month. People have asked previously about donations, a little while back I set up a page so you could donate directly to Lemmy.nz, here is the post about it (with the emphasis on not donating if it will materially impact you): lemmy.nz/post/21494039

Before moving forward, I’d like some input from Lemmy.nz users on some specific questions:

Currently we are hosted in NZ, how important is it to continue this? (with cost being the main barrier)

We will try to host in NZ if we can, but if cost becomes an issue, is it a big deal to host in Australia or elsewhere? We will want to try to stay close to NZ so NZ based users have low latency

Do people object to Cloudflare?

Currently we use Cloudflare, with benefits around loading faster for people in other countries (due to the CDN) and the ability to more easily handle AI scrapers.

Cloudflare puts us in the power of a large company and reduces our independence, but it also reduces bandwidth usage and have tools to more easily fight AI scrapers that cripple the server. Cloudflare offers captcha services that help this.

Remember, users on other instances that don’t proxy images will also be loading images directly from our servers, which Cloudflare will reduce the latency of due to their CDN.

If people object to Cloudflare, I am willing to attempt to avoid using it, adding Anubus and managing IP blocklists, but it will be more work. It will also be a slightly higher cost, as we will need a higher traffic allocation since Cloudflare currently caches a lot of image traffic.

Pātai?

If you have any questions, let me know!

As many of you will know, we have been borrowing server hosting, which kept costs very minimal. Many of you donated to the host (thanks!), but a while back they stopped taking donations as they wound down operations including transferring mastodon.nz to new people and gave me notice that we would eventually need to move to other hosting.

That was quite some time back, and now I’ve been notified that the time has finally come. At some point in this month we will swap over to a new host. It will also involve significantly increased costs to run the server (up from almost nothing), and most likely we will move to a VPS off of the current shared but dedicated hardware to more appropriate specs, and move to using object storage for images. All this will likely require some down time, which I will try to give notice of. I’m currently planning for this to be quite significant (perhaps done over the course of a whole weekend), since this would involve migrating all images to object storage which must be done with pict-rs shut down (others have reported 8+ hours for this), as well as transferring all other data to a new VPS, then a slow period of time as it catches up on federation. The second day would be a backup in case something went wrong.

When this move happens, I’ll also start to do regular financial updates. In the past people have been interested in running costs, but they have been minimal out of pocket due to the server hosting being free to me. With the increased costs it makes sense to start regular updates about what it costs to run the instance, similar to what other instances do. There is a very wide range in what this could cost, and it’s not clear right now what the lowest spec server is that we could get away with. But I am hoping we can keep this under $50 a month. People have asked previously about donations, a little while back I set up a page so you could donate directly to Lemmy.nz, here is the post about it (with the emphasis on not donating if it will materially impact you): lemmy.nz/post/21494039

Before moving forward, I’d like some input from Lemmy.nz users on some specific questions:

Currently we are hosted in NZ, how important is it to continue this? (with cost being the main barrier)

We will try to host in NZ if we can, but if cost becomes an issue, is it a big deal to host in Australia or elsewhere? We will want to try to stay close to NZ so NZ based users have low latency

Do people object to Cloudflare?

Currently we use Cloudflare, with benefits around loading faster for people in other countries (due to the CDN) and the ability to more easily handle AI scrapers.

Cloudflare puts us in the power of a large company and reduces our independence, but it also reduces bandwidth usage and have tools to more easily fight AI scrapers that cripple the server. Cloudflare offers captcha services that help this.

Remember, users on other instances that don’t proxy images will also be loading images directly from our servers, which Cloudflare will reduce the latency of due to their CDN.

If people object to Cloudflare, I am willing to attempt to avoid using it, adding Anubus and managing IP blocklists, but it will be more work. It will also be a slightly higher cost, as we will need a higher traffic allocation since Cloudflare currently caches a lot of image traffic.

Pātai?

If you have any questions, let me know!

0

We are now running Lemmy 0.19.15

preview

Earlier today I updated Lemmy.nz to Lemmy 0.19.15

Previously we werre running 0.19.13 so there are two versions updated.

Release notes for 0.19.14.

Release notes for 0.19.15.

Only minor changes. The main things seem to be:

  • Recently some malicious users started to use an exploit where they would post rule violating content and then delete the account. This would prevent admins and mods from viewing the user profile to find other posts, and would also prevent federation of ban actions. The new release fixes these problems.
  • Pages are now capped to 100, as in you can only go 100 pages of content clicking “Next” on the website, and the equivalent number of posts on apps. This is to reduce the risk of DDOS attacks.

The 0.19.15 release notes also mention Lemmy version 1.0 is coming soon and include a list of new features.

Earlier today I updated Lemmy.nz to Lemmy 0.19.15

Previously we werre running 0.19.13 so there are two versions updated.

Release notes for 0.19.14.

Release notes for 0.19.15.

Only minor changes. The main things seem to be:

  • Recently some malicious users started to use an exploit where they would post rule violating content and then delete the account. This would prevent admins and mods from viewing the user profile to find other posts, and would also prevent federation of ban actions. The new release fixes these problems.
  • Pages are now capped to 100, as in you can only go 100 pages of content clicking “Next” on the website, and the equivalent number of posts on apps. This is to reduce the risk of DDOS attacks.

The 0.19.15 release notes also mention Lemmy version 1.0 is coming soon and include a list of new features.

0

Issue with LW

preview

Hello Dave,

It seems there is an issue with lemmy.world. Votes and comments are delayed, for an example this post: lemmy.nz/post/30741653.

I think this has been happening since yesterday, and I only realized it today.

Hello Dave,

It seems there is an issue with lemmy.world. Votes and comments are delayed, for an example this post: lemmy.nz/post/30741653.

I think this has been happening since yesterday, and I only realized it today.

0

Down time 15/10/2025

preview

Sorry about the down time, Lemmy.nz was just down for 30 mins.

I wish I had more info but I’m not sure what the issue was!

It seems our instance started getting hit hard about 5am, then we were at 100% CPU for the last 30 mins:

graph showing CPU usage over past 24 hours, with it hovering around 10% until 5am NZ time where it increases to 70% or more, then to the last 30 mins where it’s a solid 100%

The number of hits has shot up too: screen shot of cloudflare dashboard showing around that same time of 5am the number of requests skyrocketed from 2000 an hour to over 100,000 per hour

In case you’re thinking well of course an NZ site has many more hits in daylight hours, here’s the week: screen shot of cloudflare dashboard showing a daily 20,000 hits going to 600,000 today

I’m guessing AI scrapers?

Sorry about the down time, Lemmy.nz was just down for 30 mins.

I wish I had more info but I’m not sure what the issue was!

It seems our instance started getting hit hard about 5am, then we were at 100% CPU for the last 30 mins:

graph showing CPU usage over past 24 hours, with it hovering around 10% until 5am NZ time where it increases to 70% or more, then to the last 30 mins where it’s a solid 100%

The number of hits has shot up too: screen shot of cloudflare dashboard showing around that same time of 5am the number of requests skyrocketed from 2000 an hour to over 100,000 per hour

In case you’re thinking well of course an NZ site has many more hits in daylight hours, here’s the week: screen shot of cloudflare dashboard showing a daily 20,000 hits going to 600,000 today

I’m guessing AI scrapers?

0

[survey for Lemmy.nz users] Should we add !mapswithoutnz@lemmy.nz community?

preview

Reposting this due to a grand total of zero votes from Lemmy.nz users…

It’s been noticed that there is no Maps without NZ community on Lemmy. There’s a short conversation here that suggests it’s added on Lemmy.nz

Please vote in the linked survey. Lemmy.nz users only please 🙂 ^___^

Survey here: survey.lemmy.nz/index.php/892864

Reposting this due to a grand total of zero votes from Lemmy.nz users…

It’s been noticed that there is no Maps without NZ community on Lemmy. There’s a short conversation here that suggests it’s added on Lemmy.nz

Please vote in the linked survey. Lemmy.nz users only please 🙂 ^___^

Survey here: survey.lemmy.nz/index.php/892864

0

[survey for Lemmy.nz users] Should we add !mapswithoutnz@lemmy.nz community?

preview

It’s been noticed that there is no Maps without NZ community on Lemmy. There’s a short conversation here that suggests it’s added on Lemmy.nz

Please vote in the linked survey. Lemmy.nz users only please 🙂

It’s been noticed that there is no Maps without NZ community on Lemmy. There’s a short conversation here that suggests it’s added on Lemmy.nz

Please vote in the linked survey. Lemmy.nz users only please 🙂

0

Sorry about the downtime

preview

We just had about 2 hours of downtime, sorry about that. It was an issue at the host end.

We borrow hosting, and the host had to make a trip into the data centre to get things going again.

With a length of downtime like this there will be a period of time while federation catches up, so expect some amount of delay with getting content from other servers.

We just had about 2 hours of downtime, sorry about that. It was an issue at the host end.

We borrow hosting, and the host had to make a trip into the data centre to get things going again.

With a length of downtime like this there will be a period of time while federation catches up, so expect some amount of delay with getting content from other servers.

0

We are now running Lemmy 0.19.12

preview

Apologies for the bit of downtime. I’ve just updated Lemmy.nz to 0.19.12 as well as updating pict-rs which handles images.

We have upgraded two Lemmy versions (previously we were on 0.19.10, now we are on 0.19.12).

The first is 0.19.11. The release notes are here.

For 0.19.12, the release notes are here.

Some changes:

  • The Lemmy developers added a donation dialogue for donating to them.
  • When registration applications are approved (or denied), you will now get an email (yay!)
  • Various accessibility changes and a new theme originally used on the instance Rblind.com

There are also heaps of minor changes mentioned in the release notes linked above.

Please let me know if you find anything not working right!

Apologies for the bit of downtime. I’ve just updated Lemmy.nz to 0.19.12 as well as updating pict-rs which handles images.

We have upgraded two Lemmy versions (previously we were on 0.19.10, now we are on 0.19.12).

The first is 0.19.11. The release notes are here.

For 0.19.12, the release notes are here.

Some changes:

  • The Lemmy developers added a donation dialogue for donating to them.
  • When registration applications are approved (or denied), you will now get an email (yay!)
  • Various accessibility changes and a new theme originally used on the instance Rblind.com

There are also heaps of minor changes mentioned in the release notes linked above.

Please let me know if you find anything not working right!

0

Donations

preview

We have a new page for this now! See: help.lemmy.nz/about/donate.html

Hi all!

TL;DR We now collect donations directly. You can donate to LemmyNZ’s server costs here: [removed in case we change it, please see new link above]

LemmyNZ is hosted in a data centre in Auckland, and costs around $60NZD per month to run.

You can now donate to LemmyNZ here: [removed in case we change it, please see new link above]

If making the donation will materially affect you, PLEASE DON’T DONATE! Lemmy.nz is not going to die if you don’t donate. Please only donate if you have plenty of money, but if you can donate on a regular basis without it materially affecting you, then please consider a regular donation.

Also, it’s only a “donation” in the loosest form of the word. It’s not a tax deductible donation as we aren’t registered as a charity or approved donee organisation. It is simply a contribution to the costs of running LemmyNZ.

We have a new page for this now! See: help.lemmy.nz/about/donate.html

Hi all!

TL;DR We now collect donations directly. You can donate to LemmyNZ’s server costs here: [removed in case we change it, please see new link above]

LemmyNZ is hosted in a data centre in Auckland, and costs around $60NZD per month to run.

You can now donate to LemmyNZ here: [removed in case we change it, please see new link above]

If making the donation will materially affect you, PLEASE DON’T DONATE! Lemmy.nz is not going to die if you don’t donate. Please only donate if you have plenty of money, but if you can donate on a regular basis without it materially affecting you, then please consider a regular donation.

Also, it’s only a “donation” in the loosest form of the word. It’s not a tax deductible donation as we aren’t registered as a charity or approved donee organisation. It is simply a contribution to the costs of running LemmyNZ.

0

We are now running Lemmy 0.19.10

preview

Apologies for the bit of downtime. I’ve just updated Lemmy.nz to 0.19.10 as well as aligning our nginx config with the latest config from the Lemmy repository. We had been running an original one from when the instance started, with some custom changes. Now we should benefit from changes over the last couple of years.

Details of the latest release are here: join-lemmy.org/…/2025-03-19_-_Lemmy_Release_v0.19…

Some highlights:

  • Youtube thumbnails should be fixed
  • DMs are now removed when banning user with content removal (i.e. deletes scammer “Nicole the Fediverse chick” DMs)
  • Resolving some language setting issues

I was hoping to solve a video streaming issue but the update to the nginx config doesn’t seem to have helped so I need to play with it some more. So there may be more brief outages tonight but I will try things in my test instance first.

Apologies for the bit of downtime. I’ve just updated Lemmy.nz to 0.19.10 as well as aligning our nginx config with the latest config from the Lemmy repository. We had been running an original one from when the instance started, with some custom changes. Now we should benefit from changes over the last couple of years.

Details of the latest release are here: join-lemmy.org/…/2025-03-19_-_Lemmy_Release_v0.19…

Some highlights:

  • Youtube thumbnails should be fixed
  • DMs are now removed when banning user with content removal (i.e. deletes scammer “Nicole the Fediverse chick” DMs)
  • Resolving some language setting issues

I was hoping to solve a video streaming issue but the update to the nginx config doesn’t seem to have helped so I need to play with it some more. So there may be more brief outages tonight but I will try things in my test instance first.

0

We are now running Lemmy 0.19.9

preview

Hi everyone, I finally did it! I got us upgraded to the latest version.

The release notes are here.

It’s a minor release but the main things seem to be a fix for languages for new users, a security fix, and various other fixes as listed in the link.

Apologies for interrupting any evening browsing, I saw my chance and took it.

Questions always welcome 🙂

Hi everyone, I finally did it! I got us upgraded to the latest version.

The release notes are here.

It’s a minor release but the main things seem to be a fix for languages for new users, a security fix, and various other fixes as listed in the link.

Apologies for interrupting any evening browsing, I saw my chance and took it.

Questions always welcome 🙂

0

Use quotations so it's possible to follow replies

preview

It would be nice if everyone would use tags or brief quotations, so I can see who is replying to who.

Modern forums are confusing my old brain because I’m used to the old BB code and old style of forum navigation, but a lot of things have changed in Lemmy, and also with proprietary forums used by large companies.

Sometimes I forget how to use features. I’m no better than a boomer who needs constant reminders on how to hold a mouse.

I think some people either don’t care or they simply forget that you can highlight text with the mouse and then click “reply” to make a quotation. Maybe they are using a phone and it’s too difficult to do this.

It would be nice if everyone would use tags or brief quotations, so I can see who is replying to who.

Modern forums are confusing my old brain because I’m used to the old BB code and old style of forum navigation, but a lot of things have changed in Lemmy, and also with proprietary forums used by large companies.

Sometimes I forget how to use features. I’m no better than a boomer who needs constant reminders on how to hold a mouse.

I think some people either don’t care or they simply forget that you can highlight text with the mouse and then click “reply” to make a quotation. Maybe they are using a phone and it’s too difficult to do this.

0

We are now running Lemmy 0.19.8

preview

Hi all, welcome to Lemmy 0.19.8!

We had about 10 mins of downtime starting from 1:17 this afternoon as the database migrations ran. Apollogies if this interrupted your lunchtime scrolling, this was needed because the Controversial scores needed to be re-calculated as detailed below.

We have skipped a few versions, we were on 0.19.5, and so we have .6, .7, and .8 updates. A brief summary follows, with links to the Lemmy announcements.

Lemmy 0.19.6 changes:

  • This version introduces parallel sending, which should solve our issues with Lemmy.world and remove the need for the extra VPS in Finland. Unfortunately Lemmy.world needs to update for this to work, so it will be a while longer.
  • Changed sameSite cookie from Strict to Lax - I think this should help the issue where you are logged out from the browser a lot (especially in Firefox)
  • There’s a change in how “Controversial” is calculated, to use exponents instead of multiplication, to match how Reddit does it. Unfortunately the Lemmy documentation page seems not to list Controversial ranking among the other explanations, but there is some discussion here.
  • Removed the “Trending” box from the home page. I kinda liked how it worked for us to find our local communities but oh well.
  • This was a reasonably big release but mostly backend stuff - see link above for more info.

Lemmy 0.19.7 and Lemmy 0.19.8 are small bugfix releases, some of which fix issues introduced by 0.19.6 so we didn’t see them. I don’t see anything of interest to an average user but if you’re interested take a look at the announcement posts linked.

Let me know if you see any issues, or have any questions.

Hi all, welcome to Lemmy 0.19.8!

We had about 10 mins of downtime starting from 1:17 this afternoon as the database migrations ran. Apollogies if this interrupted your lunchtime scrolling, this was needed because the Controversial scores needed to be re-calculated as detailed below.

We have skipped a few versions, we were on 0.19.5, and so we have .6, .7, and .8 updates. A brief summary follows, with links to the Lemmy announcements.

Lemmy 0.19.6 changes:

  • This version introduces parallel sending, which should solve our issues with Lemmy.world and remove the need for the extra VPS in Finland. Unfortunately Lemmy.world needs to update for this to work, so it will be a while longer.
  • Changed sameSite cookie from Strict to Lax - I think this should help the issue where you are logged out from the browser a lot (especially in Firefox)
  • There’s a change in how “Controversial” is calculated, to use exponents instead of multiplication, to match how Reddit does it. Unfortunately the Lemmy documentation page seems not to list Controversial ranking among the other explanations, but there is some discussion here.
  • Removed the “Trending” box from the home page. I kinda liked how it worked for us to find our local communities but oh well.
  • This was a reasonably big release but mostly backend stuff - see link above for more info.

Lemmy 0.19.7 and Lemmy 0.19.8 are small bugfix releases, some of which fix issues introduced by 0.19.6 so we didn’t see them. I don’t see anything of interest to an average user but if you’re interested take a look at the announcement posts linked.

Let me know if you see any issues, or have any questions.

0

Update on Lemmy updates

preview

TL;DR: Lemmy upgrade some time early in the new year. I also recently enabled image proxying, let me know if you noticed.

I thought I’d make a quick post just to explain the plan for updating Lemmy.nz to the latest Lemmy version and another recent change to image proxying.

Lemmy upgrade

Some of you make have noticed that Lemmy 0.19.8 just came out, and yet we are still on Lemmy 0.19.5.

There have been various bugs in 0.19.6 and 0.19.7 that while not critical, there was also no reason to rush the upgrade and I decided to wait for 0.19.8 which had a fix for some thumbnails not working that was broken in I think 0.19.6.

My plan is to give a little time to see if others have issues, and then upgrade to 0.19.8. Most likely this will be early in the new year. I don’t believe there should be any significant downtime so I will probably just do it when I am ready. Before I do, I’ll do a test run and see how long it takes, then decide then.

As always, I’ll post on the Lemmy.nz Matrix chat before I do it, and if I am having any issues.

Image proxying

You may have noticed I have enabled imaged proxying some 3 or 4 weeks back. This means all images are loaded from the Lemmy.nz server instead of whichever instance is hosting it. Personally I have found there to be significantly less instances of an image not loading. I have also noticed some images load a bit slower than before, which I think is if the image is not already cached then Lemmy.nz has to download it, then you have to download it from Lemmy.nz, so this tends to happen more on smaller communities.

However, overall I feel like the experience is improved with image proxying turned on (even if it has doubled CPU usage of the server).

As always, happy to answer any questions and hear your feedback 🙂

TL;DR: Lemmy upgrade some time early in the new year. I also recently enabled image proxying, let me know if you noticed.

I thought I’d make a quick post just to explain the plan for updating Lemmy.nz to the latest Lemmy version and another recent change to image proxying.

Lemmy upgrade

Some of you make have noticed that Lemmy 0.19.8 just came out, and yet we are still on Lemmy 0.19.5.

There have been various bugs in 0.19.6 and 0.19.7 that while not critical, there was also no reason to rush the upgrade and I decided to wait for 0.19.8 which had a fix for some thumbnails not working that was broken in I think 0.19.6.

My plan is to give a little time to see if others have issues, and then upgrade to 0.19.8. Most likely this will be early in the new year. I don’t believe there should be any significant downtime so I will probably just do it when I am ready. Before I do, I’ll do a test run and see how long it takes, then decide then.

As always, I’ll post on the Lemmy.nz Matrix chat before I do it, and if I am having any issues.

Image proxying

You may have noticed I have enabled imaged proxying some 3 or 4 weeks back. This means all images are loaded from the Lemmy.nz server instead of whichever instance is hosting it. Personally I have found there to be significantly less instances of an image not loading. I have also noticed some images load a bit slower than before, which I think is if the image is not already cached then Lemmy.nz has to download it, then you have to download it from Lemmy.nz, so this tends to happen more on smaller communities.

However, overall I feel like the experience is improved with image proxying turned on (even if it has doubled CPU usage of the server).

As always, happy to answer any questions and hear your feedback 🙂

0

Can't set pronouns in display name

preview

Xi wants to experiment with using xi/drag person-independent pronouns on this account. So far xi likes using the pronouns, but xi’s only written 4 comments and already someone accused drag of talking about Chinese politics when xi mentioned dragself. Xi knows xi’s pronouns are confusing, so xi would like to put xi’s pronouns in xi’s displayname to clear things up.

But the character limit for display names on lemmy.nz is only 22 characters, and xi’s handle already takes up 16 characters. Xi would need 24 characters to put xi’s pronouns in xi’s display name. Xi knows some instances also have a separate field for pronouns, and the pronouns get displayed next to the name without taking up displayname characters. Xi would like a place, or at least enough room, for xi’s pronouns, please.

Xi wants to experiment with using xi/drag person-independent pronouns on this account. So far xi likes using the pronouns, but xi’s only written 4 comments and already someone accused drag of talking about Chinese politics when xi mentioned dragself. Xi knows xi’s pronouns are confusing, so xi would like to put xi’s pronouns in xi’s displayname to clear things up.

But the character limit for display names on lemmy.nz is only 22 characters, and xi’s handle already takes up 16 characters. Xi would need 24 characters to put xi’s pronouns in xi’s display name. Xi knows some instances also have a separate field for pronouns, and the pronouns get displayed next to the name without taking up displayname characters. Xi would like a place, or at least enough room, for xi’s pronouns, please.

0

"All" button not working from lemmy.nz homepage, Android.

preview

I use bromine as my browser on Android and when I navigate to the lemmy.nz homepage in bromite the “All” button does not work. Works fine on lemmy.world, issue with latest update?

I use bromine as my browser on Android and when I navigate to the lemmy.nz homepage in bromite the “All” button does not work. Works fine on lemmy.world, issue with latest update?

0

We are now running Lemmy 0.19.5

preview

Welcome to the latest version of Lemmy!

Version 0.19.5 is a minor release with some bug fixes for 0.19.4. We were never running 0.19.4 so the bug fixes are less important than that new features in 0.19.4.

Release notes for 0.19.4 are here.

Release notes for 0.19.5 are here.

The upgrade took a bit longer than expected, perhaps 40 minutes of downtime. Partly this is because of an issue with postgres memory that happened before I started the actual upgrade.

Probably the two biggest new features for users are the ability to hide posts, and the ability to see any images you’ve uploaded (since lemmy started tracking).

There is also a new setting to change how upvotes are displayed, and we have a new option to have communities that don’t federate. Maybe this community is a good target for that? Let me know your thoughts.

Let me know if you have any questions or see any strangeness!

Welcome to the latest version of Lemmy!

Version 0.19.5 is a minor release with some bug fixes for 0.19.4. We were never running 0.19.4 so the bug fixes are less important than that new features in 0.19.4.

Release notes for 0.19.4 are here.

Release notes for 0.19.5 are here.

The upgrade took a bit longer than expected, perhaps 40 minutes of downtime. Partly this is because of an issue with postgres memory that happened before I started the actual upgrade.

Probably the two biggest new features for users are the ability to hide posts, and the ability to see any images you’ve uploaded (since lemmy started tracking).

There is also a new setting to change how upvotes are displayed, and we have a new option to have communities that don’t federate. Maybe this community is a good target for that? Let me know your thoughts.

Let me know if you have any questions or see any strangeness!

0

Update to 0.19.5 planned for this weekend

preview

Lemmy 0.19.5 has been released with some bug fixes, and we haven’t upgraded to 0.19.4 yet, so I’m planning on doing the upgrade (to 0.19.5) this weekend.

Relase notes here: 0.19.4 / 0.19.5

No specific time, but in my test run it was less than 10 mins. This assumes that nothing goes wrong 🙂

I’ll do it over the weekend when we normally have lower traffic.

As always, I’ll post updates in the Matrix chat.

If anyone knows of any reason why we should hold off on the update, that would be good to know too!

Lemmy 0.19.5 has been released with some bug fixes, and we haven’t upgraded to 0.19.4 yet, so I’m planning on doing the upgrade (to 0.19.5) this weekend.

Relase notes here: 0.19.4 / 0.19.5

No specific time, but in my test run it was less than 10 mins. This assumes that nothing goes wrong 🙂

I’ll do it over the weekend when we normally have lower traffic.

As always, I’ll post updates in the Matrix chat.

If anyone knows of any reason why we should hold off on the update, that would be good to know too!

0

Alternate frontends

preview

We have a new page for this now! See: help.lemmy.nz/lemmynz/webapps.html

I’ve been neck deep in Lemmy stuff recently and realised we don’t have a post that explains the different frontend options that are available on Lemmy.nz.

If you don’t like the Lemmy.nz website, you can test out one of the other website versions to see if one is to your liking (or use a Lemmy app, of which there are now many).

The default website is called lemmy-ui, and is accessed at lemmy.nz.

In addition, we have the following alternate websites. You use the same login details, it’s the same site, it’s a bit like using a different app to connect.

I don’t actively monitor these, but they do have automatic updates. If one isn’t working, please let me know!

If you have any questions, feel free to reply here.

We have a new page for this now! See: help.lemmy.nz/lemmynz/webapps.html

I’ve been neck deep in Lemmy stuff recently and realised we don’t have a post that explains the different frontend options that are available on Lemmy.nz.

If you don’t like the Lemmy.nz website, you can test out one of the other website versions to see if one is to your liking (or use a Lemmy app, of which there are now many).

The default website is called lemmy-ui, and is accessed at lemmy.nz.

In addition, we have the following alternate websites. You use the same login details, it’s the same site, it’s a bit like using a different app to connect.

I don’t actively monitor these, but they do have automatic updates. If one isn’t working, please let me know!

If you have any questions, feel free to reply here.

0

Down time last night 17/5/24-18/5/24

preview

Hi everyone,

Sorry, site seems to have been down from about 8pm last night through to about 7:30 this morning.

I spotted it last night and thought it was back up, but it seems some component wasn’t so it was loading the pages but you couldn’t do any actions.

It also means federation is behind because instances wouldn’t have been able to contact us, and so that will be slowly catching back up. It may take a while because after a lot of downtime the instances will stop trying for a while.

Always happy to answer questions 🙂

Hi everyone,

Sorry, site seems to have been down from about 8pm last night through to about 7:30 this morning.

I spotted it last night and thought it was back up, but it seems some component wasn’t so it was loading the pages but you couldn’t do any actions.

It also means federation is behind because instances wouldn’t have been able to contact us, and so that will be slowly catching back up. It may take a while because after a lot of downtime the instances will stop trying for a while.

Always happy to answer questions 🙂

0

Update on federation issues with Lemmy.world

preview

A while back I made a post about Lemmy.world federation and a lot has changed since then so I thought I’d do an update post.

TL;DR Lemmy.world posts, comments, votes coming to lemmy.nz have been delayed by a gradually increasing amount over recent months, peaking at about 4 days behind, but we should be back on track soon.

Background

Check the above post for some background, but TL;DR our server is in Auckland, NZ and Lemmy.world’s server is in Helsinki, Finland. We are about as far apart as we can get from each other. Because Lemmy can currently only send one action at a time (Post, comment, vote), we can only accept about 4 or 5 actions per second as it takes around 1/5 of a second to make the round trip. Lemmy.world is now creating more than this on average, which means we have been falling behind more and more.

Pre-fetcher

After that post, things continued to get worse. While hanging out on Lemmy Matrix rooms discussing the problem, someone (I’m not actually sure of their Lemmy account) offered to set up a pre-fetcher. Roughly how it works is it monitors lemmy.world for new posts and comments, then sends a request to lemmy.nz to see that content. Lemmy.nz then requests it from lemmy.world because it doesn’t exist.

This helps because when lemmy.world sends lemmy.nz a post or comment, lemmy.nz then needs to make other requests. e.g. it might not know about the user, so it needs to request the user from their home instance, it may need to generate a thumbnail, etc. By pre-fetching the posts, this means when lemmy.world sends it’s normal outbound federation lemmy.nz already has it so can move straight on to the next one. We can’t prefetch everything, notably votes, so for a long time we have had lemmy.world posts show with zero votes until the federation activities start coming through. Unfortunately comments from lemmy.world users on lemmy.nz posts can’t be pre-fetched so they were still taking a long time to come through.

Prior to this pre-fetcher being turned on, Lemmy.nz was doing a lot worse than aussie.zone (as seen in the above post). After turning it on, within not too long we were in better shape than aussie.zone, but unfortunately we were both still getting worse.

This is the state we have been in until yesterday. Gradually things were getting worse and worse until we were at about 4 days behind lemmy.world, so if a lemmy.world user posted on one of our posts then it took 4 days to show up (you might have noticed this if you got a notification of a reply to your post or comment that then said it was from days ago).

Batcher

The same user who created the pre-fetcher was also working on a batching process. The basic idea was that instead of lemmy.world sending each item halfway across the world, instead you add an extra server that is hosted close to lemmy.world. Lemmy.world sends their federation items to that server, then that server collects them up into one batch, which gets sent to some software running on the lemmy.nz server. That software then unbundles them into separate pieces again then feeds them into lemmy.

The idea here is that you greatly reduce the lag. Lemmy.world gets a very quick response from the extra server and so can send the next activity almost straight away. The software that passes it to lemmy.nz is on the same server as lemmy.nz so communication is very quick. And collecting up the items into a batch for the trip across the world saves a lot of time in back and forths, so we can keep our goal of receiving things in the correct order while also not having to send one at a time. Receiving in the correct order is important, for example, if you accedentally downvoted then quickly changed it to an upvote, you wouldn’t want another instance to receive the upvote first and then the downvote as it would show you downvoted instead of upvoted.

This batcher I have set up (with a lot of help!) over the weekend, and turned on yesterday morning once testing had been completed and I could get lemmy.world to redirect their lemmy.nz traffic to this new server (this was done through a change in something called the hosts file, long story short it tells your server “ignore what anyone says, lemmy.nz is actually over here”).

In the last 24 hours or so we have got from 1.5 million activities behind to about 970k activities behind. This puts us at about 2.3 days behind now, a huge improvement!

Pictures

Graph of activities behind lemmy.world

Here we have lemmy.nz in yellow and aussie.zone in green. The hump is from a large number of activities generated on the lemmy.world side, they didn’t need federating but the way this is measured means they show up until it’s worked out that they aren’t needed. You can see a sharp fall after the batcher was turned on yesterday. Graph showing aussie.zone gradually increasing from 700k 30 days ago reaching 2.6 million activities behind lemmy.world, with lemmy.nz starting at 600k behind, reaching 1.5 million, then dropping sharply over the last day to about 970k behind

Graph of time behind lemmy.world

Same colours, lemmy.nz in yellow underneath and aussie.zone in green on top. This one shows how long the delay is, or more accurately it looks at the last activity that was received from lemmy.world and checks what time that activity actually happened. So if the last activity was a comment from 4 days ago, it shows 4 days here. Graph showing a similar shape to last one, starting at around 1.7 days behing for aussie.zone and growing to over 6 days behind. Lemmy.nz starts at around 1.3 days behind, grows to about 4 days behind at the peak about 24 hours ago, then starts dropping sharply down to about 2.4 days behind currently

Conclusion

So that’s a breakdown of everything that has happened the last few months, hopefully this new batching process will bring us back in line with lemmy.world. If you see anything weird happening, please let me know!

Also this is a shout out to all the people who made this happen, and who are building all sorts of tools that we use. We have a selection of different front-end websites you can access lemmy through, an automod, prefetcher, batcher, and all sorts of help from others! I definitely couldn’t do this stuff without help 😆

And as always, if you have any questions or want more detail on any of this, feel free to ask!

Edit:

We are now up to date! Yay!

graph as above but now showing sharp drop in activities behind in recent days right down to 0 graph as above but now showing sharp drop in time behind in recent days right down to 0

A while back I made a post about Lemmy.world federation and a lot has changed since then so I thought I’d do an update post.

TL;DR Lemmy.world posts, comments, votes coming to lemmy.nz have been delayed by a gradually increasing amount over recent months, peaking at about 4 days behind, but we should be back on track soon.

Background

Check the above post for some background, but TL;DR our server is in Auckland, NZ and Lemmy.world’s server is in Helsinki, Finland. We are about as far apart as we can get from each other. Because Lemmy can currently only send one action at a time (Post, comment, vote), we can only accept about 4 or 5 actions per second as it takes around 1/5 of a second to make the round trip. Lemmy.world is now creating more than this on average, which means we have been falling behind more and more.

Pre-fetcher

After that post, things continued to get worse. While hanging out on Lemmy Matrix rooms discussing the problem, someone (I’m not actually sure of their Lemmy account) offered to set up a pre-fetcher. Roughly how it works is it monitors lemmy.world for new posts and comments, then sends a request to lemmy.nz to see that content. Lemmy.nz then requests it from lemmy.world because it doesn’t exist.

This helps because when lemmy.world sends lemmy.nz a post or comment, lemmy.nz then needs to make other requests. e.g. it might not know about the user, so it needs to request the user from their home instance, it may need to generate a thumbnail, etc. By pre-fetching the posts, this means when lemmy.world sends it’s normal outbound federation lemmy.nz already has it so can move straight on to the next one. We can’t prefetch everything, notably votes, so for a long time we have had lemmy.world posts show with zero votes until the federation activities start coming through. Unfortunately comments from lemmy.world users on lemmy.nz posts can’t be pre-fetched so they were still taking a long time to come through.

Prior to this pre-fetcher being turned on, Lemmy.nz was doing a lot worse than aussie.zone (as seen in the above post). After turning it on, within not too long we were in better shape than aussie.zone, but unfortunately we were both still getting worse.

This is the state we have been in until yesterday. Gradually things were getting worse and worse until we were at about 4 days behind lemmy.world, so if a lemmy.world user posted on one of our posts then it took 4 days to show up (you might have noticed this if you got a notification of a reply to your post or comment that then said it was from days ago).

Batcher

The same user who created the pre-fetcher was also working on a batching process. The basic idea was that instead of lemmy.world sending each item halfway across the world, instead you add an extra server that is hosted close to lemmy.world. Lemmy.world sends their federation items to that server, then that server collects them up into one batch, which gets sent to some software running on the lemmy.nz server. That software then unbundles them into separate pieces again then feeds them into lemmy.

The idea here is that you greatly reduce the lag. Lemmy.world gets a very quick response from the extra server and so can send the next activity almost straight away. The software that passes it to lemmy.nz is on the same server as lemmy.nz so communication is very quick. And collecting up the items into a batch for the trip across the world saves a lot of time in back and forths, so we can keep our goal of receiving things in the correct order while also not having to send one at a time. Receiving in the correct order is important, for example, if you accedentally downvoted then quickly changed it to an upvote, you wouldn’t want another instance to receive the upvote first and then the downvote as it would show you downvoted instead of upvoted.

This batcher I have set up (with a lot of help!) over the weekend, and turned on yesterday morning once testing had been completed and I could get lemmy.world to redirect their lemmy.nz traffic to this new server (this was done through a change in something called the hosts file, long story short it tells your server “ignore what anyone says, lemmy.nz is actually over here”).

In the last 24 hours or so we have got from 1.5 million activities behind to about 970k activities behind. This puts us at about 2.3 days behind now, a huge improvement!

Pictures

Graph of activities behind lemmy.world

Here we have lemmy.nz in yellow and aussie.zone in green. The hump is from a large number of activities generated on the lemmy.world side, they didn’t need federating but the way this is measured means they show up until it’s worked out that they aren’t needed. You can see a sharp fall after the batcher was turned on yesterday. Graph showing aussie.zone gradually increasing from 700k 30 days ago reaching 2.6 million activities behind lemmy.world, with lemmy.nz starting at 600k behind, reaching 1.5 million, then dropping sharply over the last day to about 970k behind

Graph of time behind lemmy.world

Same colours, lemmy.nz in yellow underneath and aussie.zone in green on top. This one shows how long the delay is, or more accurately it looks at the last activity that was received from lemmy.world and checks what time that activity actually happened. So if the last activity was a comment from 4 days ago, it shows 4 days here. Graph showing a similar shape to last one, starting at around 1.7 days behing for aussie.zone and growing to over 6 days behind. Lemmy.nz starts at around 1.3 days behind, grows to about 4 days behind at the peak about 24 hours ago, then starts dropping sharply down to about 2.4 days behind currently

Conclusion

So that’s a breakdown of everything that has happened the last few months, hopefully this new batching process will bring us back in line with lemmy.world. If you see anything weird happening, please let me know!

Also this is a shout out to all the people who made this happen, and who are building all sorts of tools that we use. We have a selection of different front-end websites you can access lemmy through, an automod, prefetcher, batcher, and all sorts of help from others! I definitely couldn’t do this stuff without help 😆

And as always, if you have any questions or want more detail on any of this, feel free to ask!

Edit:

We are now up to date! Yay!

graph as above but now showing sharp drop in activities behind in recent days right down to 0 graph as above but now showing sharp drop in time behind in recent days right down to 0

0

Planned picrts (image) upgrade tonight

preview

TL;DR possible thumbnail and local image outage tonight

Tonight I plan to update the image engine behind lemmy, called pictrs, to the latest stable version and to also perform a migration to a different database (sled to postgres).

In my testing there was minimal outage, but it did take quite some hours to do the migration (12+, though my test server is a lot slower than the production one).

Once the migration is complete, I intend to turn on a thumbnail cache cleaner that will significantly reduce the size of the thumbnail cache on the server. Lemmy currently saves many thumbnails locally, and never deletes them. This means we have over 260GB of images that get backed up each night for no reason. Lemmy makes it hard to tell the difference between cached images and user uploaded ones, but the cleaning tool will go through each post in the database, find the thumbnail associated with it, and delete it if it’s more than a month old.

Any questions, let me know!

Also, we are still having issues with lemmy.world. They are testing out new tools to block kbin traffic if they start to see huge amounts of activity coming from them, which will hopefully mitigate this to some extent. There is a tool running to help get posts and comments to us in real time, but it won’t work for votes which means you’ll see lots of lemmy.world posts and comments with zero votes, and the votes will be delayed in arriving by up to 1.5 days.

Feel free to ask questions about that too!

TL;DR possible thumbnail and local image outage tonight

Tonight I plan to update the image engine behind lemmy, called pictrs, to the latest stable version and to also perform a migration to a different database (sled to postgres).

In my testing there was minimal outage, but it did take quite some hours to do the migration (12+, though my test server is a lot slower than the production one).

Once the migration is complete, I intend to turn on a thumbnail cache cleaner that will significantly reduce the size of the thumbnail cache on the server. Lemmy currently saves many thumbnails locally, and never deletes them. This means we have over 260GB of images that get backed up each night for no reason. Lemmy makes it hard to tell the difference between cached images and user uploaded ones, but the cleaning tool will go through each post in the database, find the thumbnail associated with it, and delete it if it’s more than a month old.

Any questions, let me know!

Also, we are still having issues with lemmy.world. They are testing out new tools to block kbin traffic if they start to see huge amounts of activity coming from them, which will hopefully mitigate this to some extent. There is a tool running to help get posts and comments to us in real time, but it won’t work for votes which means you’ll see lots of lemmy.world posts and comments with zero votes, and the votes will be delayed in arriving by up to 1.5 days.

Feel free to ask questions about that too!

0

[meta] Ongoing issues with Lemmy.world inbound federation

preview

I though I’d better make a post about recent lemmy.world federation issues, since not everyone sees the daily posts.

TL;DR:

  • Lemmy.world inbound federation is about 18 hours behind, and slowly catching up.
  • Outbound is fine: If you post or comment on something in a lemmy.world community, it will be almost instantly federated to other users, but if they reply to you then it will take 18 hours before you see it.
  • It was caused by an assumed Kbin bug in combination with a lemmy bug
  • All other instances seem fine for inbound and outbound federation

On Thursday last week Lemmy.world began to receive hundreds of thousands of “activities” (actions - comment, post, upvote, etc) from Kbin to their communities. These actions then federate from Lemmy.world out to any instance with users subscribed to those communities.

Unfortunately, this also helped uncover a big issue with how Lemmy handles inbound federation. Actions are sent one at a time, and the next can’t be sent until the last is done. This means the number of activities your server can receive depends on the latency between the sending and receiving server. In our case, the fact we are hosted in Auckland and Lemmy.world is hosted in Finland means that we can receive about 4 or 5 inbound activities from them per second.

When the Kbin issue happened, suddenly Lemmy.world had hundreds of thousands of activities to send to us in addition to the normal inflow of content. We got way behind in terms of Lemmy.world content coming to us. At the peak, we were around 28 hours behind, about 550k activities behind. We are now sitting at around 19 hours behind, and will likely remain that way until this afternoon, when it should improve some more (it’s now peak time on the other side of the world).

Here is a graph of the progress. Aussie.zone has a similar problem to us, but is closer to Europe where lemmy.world is hosted so the latency issue is less pronounced. We are one of the worst affected instances due to how it’s hard to get further away than where we are now.

graph of lemmy.nz and aussie.zone showing how many activities each are behind, over the past 3 or 4 days. Huge jump, peaking at 550k, then slowly coming down, lemmy.nz about halfway down (250k), and aussie.zone finally reaching the bottom, to be only a few hundred behind (seconds or minutes of delay instead of hours)

As you can see, with Aussie.zone being a bit closer, they were hit hard but not as hard as us, and have now managed to recover. We are still probably days away from being back in near real time federation with lemmy.world, but each day does seem to improve quite a bit.

Other than picking up the server and moving it to Europe, there doesn’t seem to be a lot we can do except wait and hope the lemmy bug is fixed before lemmy.world grows so large that users are creating content faster than we can receive it.

There have been some suggestions for short term band-aids. One is that we could set up a proxy in Europe, and filter out certain traffic. For example, we could prevent downvotes being sent to reduce the amount of traffic coming our way (since everything is sent one at a time, as per the ActivityPub protocol). I’m not super keen on doing anything that gets us out of sync with the others, though.

We could of course move hosting to Europe, but that gets a lot more expensive (since we’d have to pay for it instead of being free-loaders) and also means the site would feel more sluggish, since then that latency would be for users on every page load instead of the behind the scenes federation.

For now we are slowly getting back in sync with lemmy.world, and as long as there aren’t more Kbin bugs then we should be ok for now. Unfortunately doing things like blocking Kbin.social doesn’t seem to help, as lemmy.world still tries to send us the content from Kbin.social users. i.e. it’s rejected by our server, not prevented from being sent, so it still takes a slot.

Any questions, suggestions, or concerns, reply to this post!

I though I’d better make a post about recent lemmy.world federation issues, since not everyone sees the daily posts.

TL;DR:

  • Lemmy.world inbound federation is about 18 hours behind, and slowly catching up.
  • Outbound is fine: If you post or comment on something in a lemmy.world community, it will be almost instantly federated to other users, but if they reply to you then it will take 18 hours before you see it.
  • It was caused by an assumed Kbin bug in combination with a lemmy bug
  • All other instances seem fine for inbound and outbound federation

On Thursday last week Lemmy.world began to receive hundreds of thousands of “activities” (actions - comment, post, upvote, etc) from Kbin to their communities. These actions then federate from Lemmy.world out to any instance with users subscribed to those communities.

Unfortunately, this also helped uncover a big issue with how Lemmy handles inbound federation. Actions are sent one at a time, and the next can’t be sent until the last is done. This means the number of activities your server can receive depends on the latency between the sending and receiving server. In our case, the fact we are hosted in Auckland and Lemmy.world is hosted in Finland means that we can receive about 4 or 5 inbound activities from them per second.

When the Kbin issue happened, suddenly Lemmy.world had hundreds of thousands of activities to send to us in addition to the normal inflow of content. We got way behind in terms of Lemmy.world content coming to us. At the peak, we were around 28 hours behind, about 550k activities behind. We are now sitting at around 19 hours behind, and will likely remain that way until this afternoon, when it should improve some more (it’s now peak time on the other side of the world).

Here is a graph of the progress. Aussie.zone has a similar problem to us, but is closer to Europe where lemmy.world is hosted so the latency issue is less pronounced. We are one of the worst affected instances due to how it’s hard to get further away than where we are now.

graph of lemmy.nz and aussie.zone showing how many activities each are behind, over the past 3 or 4 days. Huge jump, peaking at 550k, then slowly coming down, lemmy.nz about halfway down (250k), and aussie.zone finally reaching the bottom, to be only a few hundred behind (seconds or minutes of delay instead of hours)

As you can see, with Aussie.zone being a bit closer, they were hit hard but not as hard as us, and have now managed to recover. We are still probably days away from being back in near real time federation with lemmy.world, but each day does seem to improve quite a bit.

Other than picking up the server and moving it to Europe, there doesn’t seem to be a lot we can do except wait and hope the lemmy bug is fixed before lemmy.world grows so large that users are creating content faster than we can receive it.

There have been some suggestions for short term band-aids. One is that we could set up a proxy in Europe, and filter out certain traffic. For example, we could prevent downvotes being sent to reduce the amount of traffic coming our way (since everything is sent one at a time, as per the ActivityPub protocol). I’m not super keen on doing anything that gets us out of sync with the others, though.

We could of course move hosting to Europe, but that gets a lot more expensive (since we’d have to pay for it instead of being free-loaders) and also means the site would feel more sluggish, since then that latency would be for users on every page load instead of the behind the scenes federation.

For now we are slowly getting back in sync with lemmy.world, and as long as there aren’t more Kbin bugs then we should be ok for now. Unfortunately doing things like blocking Kbin.social doesn’t seem to help, as lemmy.world still tries to send us the content from Kbin.social users. i.e. it’s rejected by our server, not prevented from being sent, so it still takes a slot.

Any questions, suggestions, or concerns, reply to this post!

0

Results of Lemmy.nz Community Criteria Survey

preview

Here’s the results of what I’ve called the Community Criteria Survey.

Note that you did not have to answer any questions, so there are different numbers of responses for different questions.

First up, the demographic questions. I’m keen to do a fuller census survey like Lemmy.ca does, and will probably do this annually around the anniversary of the site in June.

Which instance(s) do you have an account on that you use to interact with Lemmy.nz communities?

Note this was multiselect, so people could choose multiple.

graph showing 16 responses for lemmy.nz, 4 for lemmy.world, 1 for lemmy.ml, 1 for beehaw.org, 1 for mastodon.nz, 3 other

Where do you live?

graph showing 6 responses for Auckland, 7 for Wellington, 3 for Christchurch, 1 for Hamilton, 1 for USA, 3 other

How old are you?

It seems the majority of our users are aged 30-50.

graph showing 2 responses for 20-29, 6 for 30-39, 9 for 40-49, 1 for 50-59, 2 no answer

Next up are the questions about communities.

Do you think communities on Lemmy.nz should be focused on people who live in NZ, or should we add communities with a more general audience? For example, !politics@lemmy.nz discusses NZ politics, with a relatively narrow audience. We could add communities with a more general audience, such as !rocketlab@lemmy.nz, which could have interested users from all over the world, and discussion that is not NZ-centric. Which of the following best describes your feelings?

Mixed responses here, it seems we will be doing surveys before we add communities as the biggest share thought a case by case basis was appropriate, and the remainder were split on whether we should only have NZ targeted communities or are happy with more general communities.

graph showing 5 prefer NZ focused, 5 happy with general audience, 10 responses for It depends - would prefer case by case basis

Specifically looking at the case of a Rocket Lab community, and considering all factors such as: Rocket Lab was founded in NZ, and launches from Māhia Peninsula in NZ Rocket Lab is a corporate entity A Rocket Lab community may not have a lot of activity outside of launches A Rocket Lab community would have a general audience, being the first Lemmy.nz community not specifically focused on people living in NZ or with close ties to NZ Posts on communities on Lemmy.nz will show in the “Local” feed Would you support the creation of a Rocket Lab community !rocketlab@lemmy.nz?

graph showing 10 responses supporting adding the community, 2 responses showing they are opposed to adding the community, 8 indifferent

I wasn’t sure of the best community for this, but as this one sort of acts as our meta community I thought this was the best place for it.

Here’s the results of what I’ve called the Community Criteria Survey.

Note that you did not have to answer any questions, so there are different numbers of responses for different questions.

First up, the demographic questions. I’m keen to do a fuller census survey like Lemmy.ca does, and will probably do this annually around the anniversary of the site in June.

Which instance(s) do you have an account on that you use to interact with Lemmy.nz communities?

Note this was multiselect, so people could choose multiple.

graph showing 16 responses for lemmy.nz, 4 for lemmy.world, 1 for lemmy.ml, 1 for beehaw.org, 1 for mastodon.nz, 3 other

Where do you live?

graph showing 6 responses for Auckland, 7 for Wellington, 3 for Christchurch, 1 for Hamilton, 1 for USA, 3 other

How old are you?

It seems the majority of our users are aged 30-50.

graph showing 2 responses for 20-29, 6 for 30-39, 9 for 40-49, 1 for 50-59, 2 no answer

Next up are the questions about communities.

Do you think communities on Lemmy.nz should be focused on people who live in NZ, or should we add communities with a more general audience? For example, !politics@lemmy.nz discusses NZ politics, with a relatively narrow audience. We could add communities with a more general audience, such as !rocketlab@lemmy.nz, which could have interested users from all over the world, and discussion that is not NZ-centric. Which of the following best describes your feelings?

Mixed responses here, it seems we will be doing surveys before we add communities as the biggest share thought a case by case basis was appropriate, and the remainder were split on whether we should only have NZ targeted communities or are happy with more general communities.

graph showing 5 prefer NZ focused, 5 happy with general audience, 10 responses for It depends - would prefer case by case basis

Specifically looking at the case of a Rocket Lab community, and considering all factors such as: Rocket Lab was founded in NZ, and launches from Māhia Peninsula in NZ Rocket Lab is a corporate entity A Rocket Lab community may not have a lot of activity outside of launches A Rocket Lab community would have a general audience, being the first Lemmy.nz community not specifically focused on people living in NZ or with close ties to NZ Posts on communities on Lemmy.nz will show in the “Local” feed Would you support the creation of a Rocket Lab community !rocketlab@lemmy.nz?

graph showing 10 responses supporting adding the community, 2 responses showing they are opposed to adding the community, 8 indifferent

I wasn’t sure of the best community for this, but as this one sort of acts as our meta community I thought this was the best place for it.

0

New posts dont seem to be federating from kbin

preview

I am trying to find this thread from kbin’s mma community but it doesnt show up. kbin.social/…/Official-UFC-299-Sean-O-Malley-vs-M…

lemmy.nz/c/mma@kbin.social?dataType=Post&sort=New

I am trying to find this thread from kbin’s mma community but it doesnt show up. kbin.social/…/Official-UFC-299-Sean-O-Malley-vs-M…

lemmy.nz/c/mma@kbin.social?dataType=Post&sort=New