ruud

joined 2 years ago
MODERATOR OF
[–] ruud@lemmy.world 22 points 1 week ago

Luckily we don't own the server, we just rent it :-)

 

Woo-hoo today's my cake-day on Lemmy.world.. actually I'm the first here but many will follow the next days. Happy cake day to you all!

 

Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.

Most of these 'attacks' are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN. Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.

For the other attacks, we are using them to investigate and implement measures like rate limiting etc.

0
submitted 2 years ago* (last edited 2 years ago) by ruud@lemmy.world to c/lemmyworld@lemmy.world
 

Looks like it works.

Edit still see some performance issues. Needs more troubleshooting

Update: Registrations re-opened We encountered a bug where people could not log in, see https://github.com/LemmyNet/lemmy/issues/3422#issuecomment-1616112264 . As a workaround we opened registrations.

Thanks

First of all, I would like to thank the Lemmy.world team and the 2 admins of other servers @stanford@discuss.as200950.com and @sunaurus@lemm.ee for their help! We did some thorough troubleshooting to get this working!

The upgrade

The upgrade itself isn't too hard. Create a backup, and then change the image names in the docker-compose.yml and restart.

But, like the first 2 tries, after a few minutes the site started getting slow until it stopped responding. Then the troubleshooting started.

The solutions

What I had noticed previously, is that the lemmy container could reach around 1500% CPU usage, above that the site got slow. Which is weird, because the server has 64 threads, so 6400% should be the max. So we tried what @sunaurus@lemm.ee had suggested before: we created extra lemmy containers to spread the load. (And extra lemmy-ui containers). And used nginx to load balance between them.

Et voilà. That seems to work.

Also, as suggested by him, we start the lemmy containers with the scheduler disabled, and have 1 extra lemmy running with the scheduler enabled, unused for other stuff.

There will be room for improvement, and probably new bugs, but we're very happy lemmy.world is now at 0.18.1-rc. This fixes a lot of bugs.

 

We'll give the upgrade new try tomorrow. I've had some good input from admins of other instances, which are also gonna help troubleshoot during/after the upgrade.

Also there are newer RC versions with fixed issues.

Be aware that might we need to rollback again, posts posted between the upgrade and the rollback will be lost.

We see a huge rise in new user signups (duh.. it's July 1st) which also stresses the server. Let's hope the improvements in 0.18.1 will also help with that.

0
submitted 2 years ago* (last edited 2 years ago) by ruud@lemmy.world to c/lemmyworld@lemmy.world
 

(I'm creating a starting guide post here. Have patience, it will take some time...)

Disclaimer: I am new to Lemmy like most of you. Still finding my way. If you see something that isn't right, let me know. Also additions, please comment!

Welcome!

Welcome to Lemmy (on whichever server you're reading this)

About Lemmy

Lemmy is a federated platform for news aggregagtion / discussion. It's being developed by the Lemmy devs: https://github.com/LemmyNet

About Federation

What does this federation mean?

It means Lemmy is using a protocol (Activitypub) which makes it possible for all Lemmy servers to interact.

  • You can search and view communities on remote servers from here
  • You can create posts in remote communities
  • You can respond to remote posts
  • You will be notified (if you wish) of comments on your remote posts
  • You can follow Lemmy users/communities on other platforms that also use Activitypub (like Mastodon, Calckey etc) (There's currently a known issue with that, see here

Please note that a server only starts indexing a server/community once it has been interacted with by a user of this server.

A great image describing this, made by @ulu_mulu@lemmy.world : https://imgur.com/a/uyoYySY

About Lemmy.world

Lemmy.world is one of the many servers hosting the Lemmy software. It was started on June 1st, 2023 by @ruud@lemmy.world , who is also running https://mastodon.world/, https://calckey.world/ and others.

A list of Lemmy servers and their statistics can be found at FediDB

Quick start guide

Account

You can use your account you created to log in to the server on which you created it. Not on other servers. Content is federated to other servers, users/accounts are not.

Searching

In the top menu, you'll see the search icon. There, you can search for posts, communities etc.

You can just enter a search-word and it will find the Post-titles, post-content, communities etc containing that word that the server knows of. So any content any user of this server ever interacted with.

You can also search for a community by it's link, e.g. [!Netherlands@lemmy.nl](/c/Netherlands@lemmy.nl). Even if the server hasn't ever seen that community, it will look it up remotely. Sometimes it takes some time for it to fetch the info (and displays 'No results' meanwhile..) so just be patient and search a second time after a few seconds.

Creating communities

First, make sure the community doesn't already exist. Use search (see above). Also try https://browse.feddit.de/ to see if there are remote communities on other Lemmy instances that aren't known to Lemmy.world yet.

If you're sure it doesn't exist yet, go to the homepage and click 'Create a Community'.

It will open up the following page:

Here you can fill out:

  • Name: should be all lowercase letters. This will be the /c/
  • Display name: As to be expected, this will be the displayed name.
  • You can upload an icon and banner image. Looks pretty.
  • The sidebar should contain things like description, rules, links etc. You can use Markdown (yey!)
  • If the community will contain mainly NSFW content, check the NSFW mark. NSFW is allowed as long as it doesn't break the rules
  • If you only want moderators to be able to post, check that checkbox.
  • Select any language you want people to be able to post in. Apparently you shouldn't de-select 'Undetermined'. I was told some apps use 'Undetermined' as default language so don't work if you don't have it selected

Reading

I think the reading is obvious. Just click the post and you can read it. SOmetimes when there are many comments, they will partly be collapsed.

Posting

When viewing a community, you can create a new post in it. First of all make sure to check the community's rules, probably stated in the sidebar.

In the Create Post page these are the fields:

  • URL: Here you can paste a link which will be shown at the top of the post. Also the thumbnail of the post will link there. Alternatively you can upload an image using the image icon to the right of the field. That image will also be displayed as thumbnail for the post.
  • Title: The title of the post.
  • Body: Here you can type your post. You can use Markdown if you want.
  • Community: select the community where you want this post created, defaults to the community you were in when you clicked 'create post'
  • NSFW: Select this if you post any NSFW material, this blurs the thumbnail and displays 'NSFW' behind the post title.
  • Language: Specify in which language your post is.

Also see the Lemmy documentation on formatting etc.

Commenting

Moderating / Reporting

Client apps

There are some apps available or in testing. See this post for a list!

Issues

When you find any issue, please report so here: https://lemmy.world/post/15786 if you think it's server related (or not sure).

Report any issues or improvement requests for the Lemmy software itself here: https://github.com/LemmyNet

Known issues

Known issues can be found in the beforementioned post, one of the most annoying ones is the fact that post/reply in a somewhat larger community can take up to 10 seconds. It seems like that's related to the number of subscribers of the community.

I'll be looking into that one, and hope the devs are too.