preview
Here is a thread to celebrate those who support Lemmy.zip to keep the (virtual) lights on and the data flowing!
If you’d like to donate and get your name on this list, head over to OpenCollective and select one of the options.
There is absolutely no pressure to donate though. Anything you can give is spent on keeping Lemmy.zip going.
Lemmy.zip Donators (Past or Present)
- EchoCranium
- Cranjis McBasketball
- Thoxy
- Anonymous Donor
- YumYum_the_Great
- Stratoscaster
- EzTerry
- Bobicus
- Levsgetso
- Lumun
- Adanisi
- Piotr Szczurkowski
- TRSturbo
- RadDevon
- Nomadin
- Devorlon
- root
- InfrasoundXP
- Zamotic
- Zarkony
- Simbico
- Blaze
- Swizzlestick
- Ulu-Mulu-no-die
- The Green Wizard
- Jai Singh
- Regedit
- First_Thunder
- Sidebro
Lemmy.zip Supporters
- David Love
- Brian Lindner
- Tamlyn
- AgnosticMammal
- Anonymous Donor
- Zorcron
- mynamesnotrick
- Angelo Arriaga
- MagicShel
- sonicsam41
- MechanicalAnt
- sidestory
- 2bitproduces
- other_cat
- Kelly Weddle
- Adrian Tasistro-Hart
- Piwix
- Gressen
Lemmy.zip Friends
- Anonpony1
- Anonymous Donor
- Grimm
- Firestorm Druid
- Paul Warnes
- SB
Lemmy.zip Heroes
- Locuester
- Jasper Smits
- ScrotusMaximus
- beneeney
Thank you again to everyone who is (or has been) supporting Lemmy.zip.
Perks!
Donators get some special perks - including the ability to choose custom emojis for everyone to use, and to get their own me.lemmy.zip profile - just like this one.
In the future, we’re looking at other ways to reward those who donate. If there is something you’d like to see as a reward, let the admin team know.
If you’d like to stay anonymous please email me at hello@lemmy.zip or send me a message on here :)
Here is a thread to celebrate those who support Lemmy.zip to keep the (virtual) lights on and the data flowing!
If you’d like to donate and get your name on this list, head over to OpenCollective and select one of the options.
There is absolutely no pressure to donate though. Anything you can give is spent on keeping Lemmy.zip going.
Lemmy.zip Donators (Past or Present)
- EchoCranium
- Cranjis McBasketball
- Thoxy
- Anonymous Donor
- YumYum_the_Great
- Stratoscaster
- EzTerry
- Bobicus
- Levsgetso
- Lumun
- Adanisi
- Piotr Szczurkowski
- TRSturbo
- RadDevon
- Nomadin
- Devorlon
- root
- InfrasoundXP
- Zamotic
- Zarkony
- Simbico
- Blaze
- Swizzlestick
- Ulu-Mulu-no-die
- The Green Wizard
- Jai Singh
- Regedit
- First_Thunder
- Sidebro
Lemmy.zip Supporters
- David Love
- Brian Lindner
- Tamlyn
- AgnosticMammal
- Anonymous Donor
- Zorcron
- mynamesnotrick
- Angelo Arriaga
- MagicShel
- sonicsam41
- MechanicalAnt
- sidestory
- 2bitproduces
- other_cat
- Kelly Weddle
- Adrian Tasistro-Hart
- Piwix
- Gressen
Lemmy.zip Friends
- Anonpony1
- Anonymous Donor
- Grimm
- Firestorm Druid
- Paul Warnes
- SB
Lemmy.zip Heroes
- Locuester
- Jasper Smits
- ScrotusMaximus
- beneeney
Thank you again to everyone who is (or has been) supporting Lemmy.zip.
Perks!
Donators get some special perks - including the ability to choose custom emojis for everyone to use, and to get their own me.lemmy.zip profile - just like this one.
In the future, we’re looking at other ways to reward those who donate. If there is something you’d like to see as a reward, let the admin team know.
If you’d like to stay anonymous please email me at hello@lemmy.zip or send me a message on here :)

(please excuse the crappy mobile screenshot)
This was obviously unexpected. At this point my suspicion was a cron job had stuck in some unexpected way and caused the cpu to just shit the bed, but checking docker showed another error when attempting to initialise container security filters. Specifically, libseccomp reported an effective API level of 1 despite the installed library meeting the version requirements for the requested functionality.
This behaviour was inconsistent with the expected capabilities of the server. Further investigation here led to the discovery of unauthorised processes and persistence mechanisms.
(Although interference from malicious software is a possible explanation, the investigation did not establish a direct causal relationship between the malware and the seccomp failures.)
At this point I took the piefed and infrastructure stacks we use offline, and started trying to figure out what was going on.
Based on the hidden artifacts and persistence mechanisms, I was able to tie this to perfctl, which is a stealthy, persistent Linux malware family designed primarily for cryptomining (XMRIG Monero miners).
This then led to looking for the attack vector. It is known for targetting exposed portainer agents (usually port 9001) and sure enough we had an exposed portainer agent.
We (over 12 months ago) used portainer to manage our servers without needing to ssh in to them. It should have been removed from all our servers a long time ago, however on the piefed server the old portainer agent container was stopped but not deleted. At some point during the maintenance this has been restarted by mistake, and it looks like it was brought up in its default state. Regardless of this, the fact this could ever happen is totally my fault and all blame lies at my feet for not tidying up properly.
So now we knew the attack vector and that the attack took place sometime on the 24th leading into the 25th (based on the graphs first showing the cpu maxing out, it was the early hours of the 25th).
At this point both INeedMana and myself were able to start a system restore process. Luckily Hetzner provides a really really handy backup system, and we reimaged the server with a backup from the 22nd, and were able to reapply our encrypted offsite database backup to the server. We also redid the updates, and obviously made sure perfctl was not present on the backup from the 22nd, which it was not.
While we can’t see any evidence that anything touched the piefed database that shouldn’t have, we have changed every single key on the server and in our infrastructure, and we invalidated everyone’s passwords by setting them to 
And UK users going directly to NSFW content will see this instead:
—
I have also recently added a minimum character length to the Lemmy.zip registrations for the answer. You’ve pretty much all filled one in if you’re on Lemmy.zip, but what you don’t see is the amount of people who type stuff such as “lemme in” or “aldkasdkjas” or stuff like that. So, now users have to enter 50 characters to not be auto-rejected. We still read every single one before approving though, we just turn those away that can’t be bothered filling it in.
—
I have also added an improved ban message when a banned user tries to log in. Normal Lemmy just pops up a little message saying can’t log in, now the user gets a full on warning:
Hopefully you never see that screen, but at least you know it exists now :)
—
I did also fix .avif images not opening in-line, although this has since been backported by the devs into 0.19.21 so I hope to anything almighty I stop getting complaints about avif images (just need some outdated apps to finally support them!).
—
I also dropped some very specific local performance improvements into lemmy, including a very detailed and overly complex (for me) bug between postgres and lemmy that was affecting schwim (one our admins) and I’m sure others out there.
I will try and explain it to you, while also explaining back to myself.
When you scroll to page 2 of a feed like All/New, lemmy says “give me the next 25 posts after the last one I showed.” The way it phrased that to the database was one big combined comparison, roughly “find posts that come after (this pinned-status, this score, this timestamp, this ID)” all bundled into a single tuple comparison.
It is an attempt to at least explain how the process works. Hopefully it helps?!



















































































































































































(yes we have some work to do on caching)

































































(Spikes line up with the scrape activity mostly)




















Thanks for any help! :)