We're now running 0.19.13

Lots of little bug fixes: https://join-lemmy.org/news/2025-09-10_-_Lemmy_Release_0.19.13

Hello everyone!

It’s time for another long-overdue update on how Fedecan and our various sites are doing. It’s been just over two years since the great Reddit migration, and in that time we’ve made some solid progress:

• Registered Fedecan.ca as a non-profit
• Moved from OVH to our own hardware in a Vancouver datacenter
• Started collaborating with the team behind sh.itjust.works
• Launched Pixelfed.ca
• Launched Piefed.ca

Finances

Here’s a look at our bank balance since we began accepting donations:

We’re currently sitting at around $2,900, with a monthly burn of about $200, which gives us roughly a year of runway. We have some additional annual costs (like domain renewals and non-profit registration), but overall we run very lean.

Fedecan still owes:

• TruckBC: $1,980
• Shadow (me): $525

These were out-of-pocket hosting and non-profit registration costs from 2023/2024. It’d be great to get those covered, but we want to keep at least a year of operating expenses in reserve.

If you're a regular user and value what we're doing, please consider donating! We have multiple ways to donate, you can find the comparison and donation links on our website: https://fedecan.ca/en/donate

Sh.itjust.works

Nothing major to report here - we’ve all been a bit busy lately, but collaboration is continuing slowly behind the scenes.

Fediverse Growth

We're seeing a healthy volume of posts and communities on lemmy.ca, surging with each Reddit drama:

Infrastructure

Our server is a Dell R7515 with an EPYC 7763, 1 TB ram and 4x 7.68tb nvme data disks, which is hosted in a datacenter in Vancouver, BC.

I spun up victoriametrics + victorialogs a few weeks ago and have been ingesting all of our data, giving us the ability to put together some nice grafana dashboards.

Everything is running great on the infrastructure side of things. Our server is barely working up a sweat and we shouldn't have to worry about scaling for a long time.

Lemmy.ca still comprises almost all of our traffic:

Lemmy.ca

Our over provisioned stack is performing well, handling the occasional lemmy / lemmy-ui dropout:

Similarly the DB is mostly running out of ram:

Our object storage is slowly climbing as expected, but we've got several years of capacity to figure out a long term solution:

I’m also doing some limited analytics on our web logs. As expected, lemmy.world makes up the majority of our federation traffic:

One interesting thing to see from the user-agent data is the breakdown of traffic by the different mobile clients:

The “dart” UA is just a common web library, Thunder reports as this and I suspect other clients do too. If you’re a client developer, please set your user-agent!

Out of the alternative web clients we support, tesseract is the most popular although the overall traffic volume is still low:

We only store 7 days of logs but I’m hoping to get these pulled out into metrics soon, since it would be interesting to track which clients / interfaces people use over time.

Pixelfed.ca

Not much to say on this one, due to using local storage it currently runs on a single VM without redundancy.

Piefed.ca

Piefed runs on a pair of VMs with its own database and object storage backends.

Cloudflare

If you want to compare against previous data posts, here’s our same cloudflare graphs for lemmy.ca

As always, feel free to reach out if you have any questions or ideas. Thanks for being a part of the Fediverse!

https://join-lemmy.org/news/2025-06-13_-_Lemmy_Release_0.19.12

Enjoy!

Good morning everyone. Just a quick heads up that I've banned a good chunk of IP space in China due to abusive traffic.

I've tried to restrict this where possible to datacenter blocks from Huawei, Tencent, and Alibaba, but China Telecom / Mobile were also heavy sources of suspicious traffic. I doubt we have many (if any) users in China, but if you are affected please let me know.

This has been ongoing for a while and I ignored it initially since the traffic levels were low, but it wasn't anymore.

The ban has very visibly cut our traffic levels:

cross-posted from: https://sh.itjust.works/post/35952239

Hey all,

Today, I’m writing to share our intention to join Fedecan, a Canadian not-for-profit organization committed to supporting the Fediverse. We believe this partnership will foster collaboration among like-minded individuals who share a common goal: to create a safe, welcoming space where people can connect freely.

Together we will help deliver a fast, reliable, and consistent experience for all and continue to be free from corporate influence and guided by the values of openness, community, and trust.

Who is Fedecan

Fedecan is a registered non-profit organization based in Canada, with the goal to provide a safe and welcoming space for people to connect with each other on the Fediverse. More information can be found here.

You may also know them by their Canadian lemmy instance lemmy.ca.

What does this mean for sh.itjust.works?

From the outside everything will stay much the same, especially with the admin team. Internally, we will collaborate on tasks related to non-profit compliance, policies, banking and common infrastructure elements such as backup/disaster recovery infrastructure. We will continue to operate separate equipment and instances in geographically isolated locations under different names. We will work towards aligning on safety and security practices in order to ensure that data is secure.

From an organizational structure, sh.itjust.works will fall under the Fedecan umbrella and will share common bylaws, policies, methodologies on best practices, security and legal considerations.

Where do donations go?

Donations to sh.itjust.works will continue to support our mission and objectives exclusively. However, users will also have the option to donate directly to Fedecan, which will allocate funds amongst its projects including sh.itjust.works. Operational costs related to running the non-profit will be shared among projects and these expenses will be transparently disclosed in annual reports published on the Fedecan website.

Option to withdrawal

sh.itjust.works will have the right to withdraw from the Fedecan umbrella should our mutual goals no longer be aligned. In such a case, a predefined provision and action plan will be in place to ensure a smooth transition back to independence.

Why Now?

As the Fediverse continues to grow, we believe it's important to collaborate more closely with others who share our values. Joining Fedecan allows us to do just that, strengthening our operation through a non-profit while staying true to our mission.

Timeline / Next Steps

Over the coming weeks, we'll begin the process of integrating with Fedecan on the organization side. You won't notice many changes but we'll keep you informed throughout the process.

Looking Ahead

We’re excited about this next chapter and the opportunities it brings. By joining forces, we’re reinforcing our commitment to the Fediverse and to the principles that brought us all together in the first place—openness, community, transparency, and trust.

Our core mission remains unchanged. We're still independent in spirit and practice and we remain committed to being a space that's not driven by profit, but by people.

I invite your questions or concerns on this thread or on our https://matrix.to/#/#sh.itjust.works:matrix.org)

Stay connected,

--The sh.itjust.works and Fedecan Team

We're now running 0.19.11, changelog here - https://join-lemmy.org/news/2025-04-08_-_Lemmy_Release_v0.19.11

For anyone that noticed the 30 seconds of downtime a few minutes ago, that was to upgrade us to lemmy 0.19.10.

Changes are listed here - https://join-lemmy.org/news/2025-03-19_-_Lemmy_Release_v0.19.10_and_Developer_AMA

This is not the version with breaking API changes, there should be no impact to any clients.

Enjoy!

Sorry everyone I know how much you love the attention she gives you, but I've implemented some quick and dirty filtering for private messaging.

We now have the ability to automatically mark PM's as deleted or read, depending on content inside of them. If we accidentally filter something you legitimately wanted (ie, not Nicole) please let me know.

If any other instances would like to implement this, here's the code. Note that you'll need to set your hostname at the top here for some reason I haven't exactly identified.

SET lemmy.protocol_and_hostname = 'https://lemmy.ca/';

CREATE TABLE private_message_filters (
    id SERIAL PRIMARY KEY,
    phrase TEXT NOT NULL,
    behavior VARCHAR(10) NOT NULL CHECK (behavior IN ('delete', 'mark_read'))
);

CREATE OR REPLACE FUNCTION filter_private_messages()
RETURNS trigger AS $$
DECLARE
    banned_phrase_record private_message_filters%ROWTYPE;
BEGIN
    FOR banned_phrase_record IN 
        SELECT * FROM private_message_filters
    LOOP
        IF LOWER(TRIM(NEW.content)) ILIKE '%' || LOWER(TRIM(banned_phrase_record.phrase)) || '%' THEN
            IF banned_phrase_record.behavior = 'delete' THEN
                NEW.deleted := true;
                RETURN NEW;
            ELSIF banned_phrase_record.behavior = 'mark_read' THEN
                NEW.read := true;
                RETURN NEW;
            END IF;
        END IF;
    END LOOP;
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trg_filter_private_messages
AFTER INSERT ON private_message
FOR EACH ROW
EXECUTE FUNCTION filter_private_messages();

To add filter words:

insert into private_message_filters (behavior, phrase) values ('delete', 'spamtestdelete');
insert into private_message_filters (behavior, phrase) values ('mark_read', 'spamtestread');

If you want to quickly disable / enable filtering while testing:

ALTER TABLE private_message DISABLE TRIGGER trg_filter_private_messages;
ALTER TABLE private_message ENABLE TRIGGER trg_filter_private_messages;

I'll leave it up to you to figure out what phrases to filter on. MAKE SURE YOU TEST. If there's an error, private messaging could break completely. You should not get an error message from the UI while sending a message with a banned word.

Sorry for the downtime! Unfortunately our secondary firewall took over for some reason, and haproxy failed to properly come up.

I'll be scheduling a maintenance window in the next few days to do some further digging, so I can make sure this is fully resolved.

Sorry about the brief downtime there!

Wasn't related to the new users or high load, but was a combination of two problems:

1. Our fw02 took over for some reason TBD
2. Opnsense isn't keeping haproxy configs in sync and fw02 had a broken config

Still digging into the root cause, but shouldn't be any more issues =)