There are ongoing issues between incoming and outgoing activities between Ani.Social and Lemmy.World. Posts, comments, and votes from both instances are not in sync with each other.

The exact cause is to be determined. The instance may experience downtime and interruptions until further notice.

In the meantime, we suggest using an account on an instance that is federated with both Ani.Social and Lemmy.World.

Thanks to @wjs018 for informing me about this issue. Thanks also to MrKaplan from the Lemmy.World team for continuously helping us resolve this issue.

    • Neshura@bookwormstory.socialEnglish
      0·
      2 months ago

      Wouldn’t be the first time lemmy.world had federation issues, I’m honestly surprised how quickly Lemmy ran into scaling problems of this nature.

  • wjs018@ani.socialEnglish
    0·
    2 months ago

    The outbound federation to lemmy.world has been resolved at this point (thanks hitagi!), but the inbound federation from lemmy.world continues to be an issue. I don’t really have any insight into solutions for that as it seems it might be due to physical constraints similar to what has been plaguing the AU/NZ servers.

    Just a heads up to moderators of ani.social communities though. Because the world version of any community here is going to be a couple hours delayed (currently ~8 hours), it might make sense to have an account on world that you appoint as a mod to your community. This lets you take moderator actions on things that are in the world version of your community even before it federates over instead of having to wait hours for the spam/whatever to federate over before you can remove it.

    Tagging @MentalEdge@sopuli.xyz to let you know since you probably have some of the more active communities on the instance.

    • Rottcodd@ani.socialEnglish
      0·
      2 months ago

      Ah… I get it.

      I couldn’t understand why I keep duplicating Kadath’s posts. I was positive I was checking sufficiently to be sure chapters hadnt been posted yet, but then I’d come back later and find that I’d doubled one of her posts.

      Because her account is on .world.

      I expect there’ll turn put to be some duplicates in the latest batch too.

      So I guess when I see that something I really expect to have already been posted hasn’t been (like just a bit ago with Shiretto Sugee Koto Iteru Gal), I need to just let it go, since the odds are that it has already been posted and it’s just that .world hasn’t let us know yet…

      • wjs018@ani.socialEnglish
        0·
        2 months ago

        Yeah, it’s a weird situation. I am not removing duplicates that haven’t federated over yet. Until posts federate to ani.social, no other, third-party instances see them. The situation is growing worse too. The current delay is up to ~12 hours behind.

  • wjs018@ani.socialEnglish
    0·
    2 months ago

    The federation delays seem to be getting better over the past day or so. So, whatever you are doing seems to be working. It seems like the PR to engineer out a lot of the latency sensitivity of Lemmy federation is still a WIP, so this might be something that comes and goes as the network topology and routing changes day to day. Almost like federation weather.

  • wjs018@ani.socialEnglish
    0·
    2 months ago

    Alright, I have been doing some poking around the grafana dashboard and noticed that about 20k activities/hour (~ 6 per second) seems to be the limit that ani.social can process coming in from lemmy.world. Whenever the activity peaks on world go over that (generally EU afternoon/NA morning), we start to lag a bit. Then, after the peak has subsided, we catch up.

    All this really seems like it is putting a pretty hard limit on how big the fediverse could actually grow without federation becoming completely impossible. I was reading up on efforts that reddthat has undertaken to improve federation from world (since they are in AUS). Their EU-based proxy seems to have worked well, but even with batching like this, federation is always going to be a lot of bandwidth and message passing between servers that just might not scale past a certain point. Anyway, I am off topic.

    In any case, the lag seems like it will be coming and going with a bit of regularity, kind of like fediverse tides.

    • MentalEdge@ani.socialEnglish
      0·
      2 months ago

      The latency limit is caused by the activity queue that was introduced in v19.

      Servers can only talk as fast as round time allows, because Lemmy instances now keep track that each event actually does get federated, and in the right order.

      That last point means each event only gets sent once acknowledgement of the last one is received, creating a hard limit for how many events can be communicated, depending on ping. A mere two per second with a latency of 500ms.

      This serial process will obviously need to be parallelized. But that’s difficult.

  • Toes♀@ani.socialEnglish
    0·
    2 months ago

    Random thought, what would happen if there was an ani.social cache server at the same datacenter as Lemmy.world?

  • SatouKazuma@ani.socialEnglish
    0·
    2 months ago

    I was wondering what was happening, so I ended up just creating an account here. Especially since I probably spend most of my time either here or on programming.dev.