I'd want a demo that shows a much better example of conflicts occuring.

The local-first/realtime space is getting busy these days. Replicache/Reflect is well worth checking out for the beautifully simple data model / coding model. I'm very happy with it.

One plus point compared to CRDTs is that you get to decide how you want to deal with conflicts yourself, with simple, sequential code. I'm not up to date with the latest in CRDT-land but I believe it can be complicated to add application specific conflict resolution if the built-in rules don't fit your needs.

I agree wholeheartedly — we took the same approach for PowerSync with a server reconciliation architecture [1] over CRDTs, which is also mentioned elsewhere in the comments here. For applications that have a central server, I think the simplicity of this kind of architecture is very appealing.

Props to Aaron for bringing awareness to and evangelizing server reconciliation.

[1] https://www.gabrielgambetta.com/client-side-prediction-serve...

A JavaScript framework to build offline-first but collaborative webapp - https://news.ycombinator.com/item?id=33269440 - Oct 2022 (2 comments)

Linear clone, with realtime sync and instant UI - built with Replicache - https://news.ycombinator.com/item?id=31331660 - May 2022 (3 comments)

Replicache: Easy Offline-First for Existing Applications - https://news.ycombinator.com/item?id=22173500 - Jan 2020 (22 comments)

Anyway, I want to comment on this:

>For example, any kind of arithmetic just works:

Of course, it is extremely trivial to set those up as idempotent operations.

>List operations just work:

Nope ... or at least, I'd have to take a closer look at your implementation, consider:

Some sort of array/list that looks like: [1, 2, 3, 4, 5]

* User A deletes the range from 2-4.

* User B deletes the range from 3-5.

* User C inserts (uh-oh) something between 3 and 4.

All updates reach the server at the same time. What is the solution to that? Timestamps? Fallback to LLW? This breaks the transactional model.

Pick any of those updates as "the winner", what do the other users see? How do you communicate this to them?

If your answer is like "we just send them the whole array again", this breaks the transactional model (x2).

>List operations just work:

Fair point. Will change. What I really meant here was "(Many) list operations just work" (just like above I said "all kinds of things just work".

> All updates reach the server at the same time. What is the solution to that? Timestamps? Fallback to LLW?

There are a number of issues you might be pointing out, and I'm not sure which one you mean.

In general, you can't use list indexes as identifiers for edit/delete in Reflect because they aren't stable. That's why in this example I commented to use the item itself (if atomic) or more likely a stable ID: https://i.imgur.com/IKzmf0q.png

There is also the potential issue of User C's inserts getting deleted. This seems like a problem at first, but in a realtime collaborative context, nothing can fix this. User C's insert can land just before User A or B's delete. In this case, sync would be correct to delete user C's insert, no matter the protocol. But User C might still be sad. From her perspective, she just wrote something and it got deleted. This is the nature of two humans working in the same place at the same time and potentially having different intentions. For these problems, undo and presence indicators go a long way to avoid these problems through human/social mechanisms.

Exactly.

>For these problems, undo and presence indicators go a long way to avoid these problems through human/social mechanisms.

x1,000 to this. This guy builds. That's exactly my takeaway as well from years working on this; provide a solid (by that I mean clearly defined) deterministic algo, and solve the rest w/ UX.

That said this sort of reconciliation can be extremely painful if people are working separately offline for any length of time as arbitrarily resolving many conflicts doesn’t necessarily result in coherent results. But as I understand it that’s also an issue for other ways of solving the issue as well.

Another missing piece is signalling failure. For example two people trying to fill the last slot of a limited resource. Having to infer from the state that comes back whether your operation succeeded or not is not a fun way to write multiplayer code.

For the former you want (as mentioned) good presence info and undo. For the latter you (sometimes) want explicitly flagged conflicts and manual resolution, like we have in git etc.

With the Replicache/Reflect model, you could actually handle both approaches. Might not be easy but the model can support it.

  - client A appends “a” to list L
  - client B appends “b” to list L
  - client C appends “c” to list L

  - client A thinks L is [“a”]
  - client B thinks L is [“b”]
  - client C thinks L is [“c”]
  - server sends state event to clients saying that L is [“c”,”b”,”a”]

Reflect has first-class support for this kind of situation by allowing mutators to run special code server-side. I talk about this a bit here:

https://rocicorp.dev/blog/ready-player-two#server-authority

You can use the same mechanism to have a branch in the code that places the piece that only runs server-side and chooses the winner.

>If your answer is like "we just send them the whole array again", this breaks the transactional model (x2).

You also did that, whoops.

a) [1, 5]

or

b) [1, , 5]

?

Is the solution to that,

a) [1]

or

b) [1, ]

?

>Notably, the server has to sequence actions somehow [...]

Yes, that's my whole point, you have to fallback to something like LLW, and then it's over for the transactional model :P.

Edit: Btw, I'm not trying to be the snarky, pessimistic dude. This kind of models are really interesting and I love working with them. I am just trying to illustrate how things could go awry without much complexity involved.

If they can't see each other working on the data together they could be surprised but they could eventually fix the data to the desired state.

If they don't check the result or can't check the result, the state won't be OK. However it's not what probably happens in interactive apps like collaborative editing or games.

1. You wrote "For example, schema validation and migrations just sort of fall out of the design for free." - very curious to read about what you've found to work well for migrations! I feel like there's a lot of nice stuff you can build here (tracking schema version as part of the doc, pushing migration functions into clients, and then clients can live-update) but I never got the chance to build that.

2. Do you have a recommendation for use-cases that involve substantial shared text editing in a TCR system? I'd usually default to Yjs and Tiptap/Prosemirror here (and am watching Automerge Prosemirror with interest). The best idea I've come up with is running two data stores in parallel: a CRDT doc that is a flat key/value identifying a set of text docs keyed by UUID, and a TCR doc representing the data, which occasionally mentions CRDT text UUIDs.

2. This is what I want to work on next. I am similarly intrigued by the Prosemirror model. It seems like a good match.

https://x.com/replicache/status/1714684061589877235?s=20

ElectricSQL is a distributed database. It's not going to run anywhere near 60fps with tons of users, because it's not running in memory on the server like Reflect/PartyKit/Liveblocks do. OTOH it will allow you to filter/query interact with much more data at the same time than Reflect/PartyKit/Liveblocks do – Reflect has a limit currently of 50MB per room and it's not practical to have dozens of rooms open at one time to get around this.

Systems in the Reflect room/document based model are well suited for applications where you primarily interact with one "document" at a time and you want that to be as realtime as possible. Figma is the canonical example. You want the entire document to move together at 60 FPS, completely fluidly. It's not going to be possible to do this well in the ElectricSQL model IMO. Spreadsheets, presentations and documents are other examples. Systems in the ElectricSQL model are more suited for applications where you are interacting with lots of documents at the same time. Think CRMs, bug trackers, etc.

Of course real applications are messy. Even document editors always have a dashboard where you can at least see all your documents at once. And CRMs of course have a detail view that looks like a document editor. Both types of systems are going to track toward the other to support the needs of real applications.

HTH!

For the GA, we plan to implement a scheme that will allow the room to support up to ~100 concurrent active users (actually doing things) and thousands just watching.

I totally feel hgs3's confusion when reading the discussion here or that website. I always confuse the use cases - "are they now talking about computer games, or are they talking about the broader scope of any kind of software in which multiple users can act concurrently within a shared environment?".

I think this confusion would be much less severe if I hadn't been an avid gamer for decades and thus had a very specific idea of the meaning of the term "multiplayer".

On the other hand all web software is "multiuser", which doesn't tell you anything.

This ends up being a lot of work to maintain and easy to break as the application gets bigger, and it also defeats some of the benefits of the CRDT in the first place (now the server has to mediate everything and you can't have peer-to-peer sync).

Also if there are side-effects of any actions which require auth which aren't undoable, then that gets more complicated.

If you already have a server in the middle, it's a lot simpler to just use a protocol that allows the server to reject messages in the first place.

E.g., someone connects peer to peer to you and claims some privilege etc., you could use the server to verify their claim.

Also, I don't think CRDT necessarily implies peer to peer, as you can use a central server for message passing, but keep the CRDT model for resolving the current state on both server and client.

But as aboodman says in a sibling comment, if there’s a server as an authority, it can simply reject messages from unauthorized clients.

Right now that window is pretty small because we haven't turned on persistence yet.

PartyKit is extremely unopinionated. It's essentially lightweight javascript server, that launches fast and autoscales (I don't say this as as a bad thing, it's a useful primitive). Most people seem to run yjs in PartyKit, but you can also run automerge or even Replicache – my company's other project.

Reflect is entirely focused on providing the best possible multiplayer experience. We make a lot of choices up and down the stack to tightly integrate everything so that multiplayer just works and you can focus on building your app.

That's the plan anyway :).

For an in-depth explanation, see this excellent GDC talk on its implementation in Mortal Kombat and Injustice 2: https://youtu.be/7jb0FOcImdg

That is an algorithm for peer-to-peer games, where each peer waits for the input from all other players before advancing the game simulation. The deterministic part is because players share inputs (not simulation results) AND the simulation is deterministic for the same inputs. The lockstep part is because all clients advance at a coordinated pace. The Age of Empires series use this approach, and that’s why units don’t move immediately when you click. Starcraft uses this too, but it has some tricks to smooth the gameplay experience. Both are peer-to-peer with no single “server”.

In the case of Reflect, we have a server-authorative simulation (not peer-to-peer). Clients send their inputs to the server, but they do not wait for the result, they instead predict the result locally without confirmation from the server. The server also rolls back time and then replays inputs to compensate for individual client latency. And the client corrects/reconciles their local simulation once the server sends a simulation result that included one of their inputs.

The keywords for this algorithm are:

- Server-authoritative

- Predicted

- Lag compensated (simulation rollback / server rewind / input replay)

- Prediction reconciliation (local simulation rollback / local input replay / misprediction smoothing)

I’m not sure if Reflect has it, but client-side interpolation is also a common feature in FPS games, where upon receiving a world update, the client will tween entities to their new positions/rotations over a fixed interval (such as 0.1s). This allows you to send updates to clients at only 10Hz but have entities move smoothly (without needing the client to locally simulate the physics of every entity).

There is only one authority, the server, so determinism isn’t super important. However, it is nice to have for client predictions to reduce the occurrences of mispredictions. Mispredictions occur when the server state did not advance the way the client predicted. These can happen when:

(1) Another client’s input changed the world state in an important way,

(2) The simulation is not deterministic (e.g. the random number generation is not synced).

In FPS games, (1) is impossible to eliminate. These mispredictions are often smoothed using interpolation. (2) should be minimised, but may actually be desirable. For example, Counter Strike does not sync the RNG for bullet spread randomness, to prevent “nospread” cheat programs from predicting where each bullet will go and instantly adjusting the player’s aim so the bullet lines up perfectly.

https://www.gabrielgambetta.com/client-side-prediction-live-...

https://developer.valvesoftware.com/wiki/Latency_Compensatin...

https://developer.valvesoftware.com/wiki/Source_Multiplayer_...

Deterministic lockstep means that the speed of the simulation is dependent on the slowest member as everyone waits to get all the inputs before advancing a frame. You’ll get literally pauses and jank if someone is on a poor connection. That’s not the case here.

The difference with this model of networking is what is synced in the lockstep case the simulation being deterministic means you only need to send inputs to keep the game in sync so you vastly reduce network traffic. In this case it’s not necessarily deterministic and the server sends back the new state to the clients after determining what it is as it’s the authority on what that state is. Clients can predict ahead by making the change locally but must correct mispredictions. In the deterministic case there can be no mispredictions and it’s a pain in the butt to ensure that your game is actually deterministic and resolving sync issues is a real pain.

Factorio sounds like it uses a deterministic model that doesn’t wait to run in lockstep and instead reconciles errors that causes. The typical way of doing that is to rollback to a prior state where the simulation diverged due to missing inputs and redo the following ticks. That’s probably prohibitively expensive given the scale of Factorio versus something simpler like a fighting game so I assume there is a hybrid model and some extra info is sent by an authoritative source that can be checked locally.

https://i.imgur.com/q9FnN4R.png

The whole idea of DL as I understand it is to take advantage of determinism to reduce bandwidth. But Reflect doesn't do this.

I think this is a feature for developers. It's nice to be able to have the server just do whatever it wants for whatever reason it wants. It also means you can use normal JS, and don't need to try and enforce determinism somehow.

There are a lot of different names for these protocols floating around. Server Reconciliation is the one I found that best describes what Reflect does: https://www.gabrielgambetta.com/client-side-prediction-serve...

But in the end I felt like since we're going to continue to evolve this and have made specific choices, it made sense to give our version its own name so that we can refer to it.

Happy to answer any questions as a customer :)

Going to have to check this out.

Firstly is rebasing these operations in this style of system takes a bigger hit on performance than I expected in my experience. Sure, managing counters and doing integer math can consistently run at 120FPS but more complex operations can really bog down the system when a lot of users are interacting with it once.

The second challenge is that when operations fail (e.g. for authorization) undoing that operation isn't always straightforward. You either pop the operation and replay ops from a earlier snapshot (which necessitates storing all ops in order), writing an inverse operation manually, or having the server send the entire state back to rebase your unconfirmed ops on which can also end being expensive.

In general, the challenges of this approach are true for all operation-based CRDTs too. State-based approaches are a good deal simpler just tend to over-send data over the network.

We are targeting 100 concurrent users at 120 FPS. Not completely there, but it is certainly achievable, and we will scale back framerate when we need to.

Maybe this is a bit out of scope of what the goal is for this project but just thought I would bring it up.

I've done a lot of thinking (but not coding) around "simply replicating state" being the right answer to interconnected apps.

The degenerate case is to model everything as a git transaction log, but be able to run it "faster" than really using git. It's very intriguing to hear your discussion of "branching, sequentialization, etc..." because that seems like the right way to go. ( edit: and implicit _physics_ vs. state transfer in some cases... it's cheaper to send nuke@[x,y,z] instead of the literal calculations and changes to state that single action may imply ... physics in games maybe being gravity and ballistics, but 'business-physics' might be update/select/modify, etc. )

There's a few interesting optimizations/tradeoffs in some of the "real" multiplayer networking libraries: https://torque-3d.readthedocs.io/en/latest/script/network.ht...

...eg: Unguaranteed, Guaranteed, Guaranteed Order, Most Recent State, Guaranteed Quickest (and each client has it's own "camera" / perspective against the "scene", where +180 might be "need it now b/c I'm looking at it", but -180 might be "meh, need close-to-the-latest-state before I turn around and look at it").

I'm rooting for you b/c "multiplayer + physics" seems like the right answer (eg: the ghost of couchdb), but no one seems to have (yet) cracked the semantic code as to how to describe "as fast as local, but synchronized with $THESE tradeoffs and $EXCEPTIONS".

I'm curious what trade-offs you looked at regarding sending state vs mutations from the server back to clients. As the server is authoritative, I imagine it could also send mutations to the clients in the order in which the server has determined parallel operations need to be resolved (and adjust mutations as needed to resolve conflicts or permissions issues). Clients would then need to rebase any pending local mutations, so that would involve more logic vs the data sent for mutations most likely being smaller than the "full" state of certain objects being updated?

Has anyone done the cost + risk assessment of building a for-profit product on top of this? Would love to know, as I am working on a web IDE with collaboration. There’s also the matter of obscuring client data from 3rd and even 1st party.

But I get your argument too. Naming is hard.

I don’t think it trivializes the goal or the work being done, it just distinguishes it from parallel non-coordinated use.

Also, players are not just for games; the Bard had some thoughts on that. ;)

Iw as wondering what kind of checkingpoint or snapshotting there might be. Such that one doesn't have to "replay-the-world" if something goes awry (localstack persistence did/l (does?) this, storing state by just recording/replaying API calls). It sort of seems like having a globally accepted state is enough & makes sense, but it still seems like work to rollback & do again, still seems to require snapshotting.

Something like this feels like a 100% fit for having immutable data structures. Where you can snapshot the world very very quickly/at low cost.

I implemented the same solution with Redux actions and Cloudflare Durable Workers. https://emojis.cadell.dev

https://supabase.com/docs/guides/realtime

- Supabase Broadcast: one client sends every other client a one-time message

- Supabase Presence: one client tells all other clients that its own state has changed. This is similar to above, except that server keeps track of all the last-state for each client so that new members of the channel can get up to date.

- Supabase Realtime: server tells all clients when server state changes (so you will not get optimistic client-side changes this way)

Supabase’s real-time stuff lets various parts of your system send messages about changes to each other. But when it comes to realtime editing, that’s the easy part!

Reflect is a much higher level abstraction that handles the hard parts:

- maintaining a cache on the device (this alone is annoying hard to get right)

- queuing updates to send to the server reliably

- handling conflicts between different users while preserving the intent of each user, by rebasing histories locally, and choosing the winner on the server

The other side of it is to make sure that real-time collab is something users actually want and use. There’s a lot of froth at the moment about real-time but IMO and IME it has a much smaller set of useful cases than most people actually think. But where it is useful, it is a step up for collaboration.

For example with Google Docs I find myself and our broader team mostly solo write but collaboratively review and edit. Where we collaboratively write its carefully structured so people aren’t stomping on each other and often done with someone acting as a mediator.

Edit: I have no idea why this was downvoted, it's true

I think that’s the point of comparing this to CRDTs - to show that there may be a better tool for the job if you’ve been considering CRDT because it’s seen some recent popular exposure

:)

For this particular problem, it just comes down to CPU time. At the bottom of the stack ws.send() is a syscall and it takes significant time. You can only do so many of these calls per second. We measured it at about 8k/sec a year ago.

With a conservative limit of 2k calls to ws.send() (so that we can stay about about 50% utilization and only use half of the time for calling send), this implies 6 clients at 60 FPS using a naive approach (6 * 6 * 60 = ~2k).

It's not really a DO problem, it's just that doing n^2 messages can't really work in any platform.

Reflect is based on Server Reconciliation: https://www.gabrielgambetta.com/client-side-prediction-serve....

The critical difference is that Reflect doesn't need to know the operations up front. That is what allows developers to provide their own operations, or as we call them "mutators".

You'd have to split up the image into chunks of about 1KB each, this would be fairly easy to do by pixels. It would be a fun project to try.

See https://hello.reflect.net/rooms#data-model for more information.

Curious why you think no-one has done TCR before? Also, any insights on why you're bullish on the "multiplayer web"?

Will be cool to see example docs / repos as they come out!

TCR is just a small generalization of what the game industry has been doing for awhile. So in that sense, we are not at all the first.

But I'm honestly not sure why it's not been done on the web yet. To me, it's a really elegant approach. It is harder to implement in a general way (like as a library) because you need to run code on the server. And as others have pointed out, it's harder to make fast.

Perhaps multiplayer is just new enough to the web that there hasn't been time for these approaches to evolve.

I mean, React was also inspired by game techniques that at that time were decades old.

- If you can have an authoritative server, CRDTs come with unnecessary restrictions and overhead. Reflect presumably gains a lot of efficiency by loosening those constraints.

- Conversely, if you want clients to collaborate without a central server, you can't use a service like Reflect, since there's no server to run your conflict resolution logic.

But most apps people build today do in fact have a central server. And by leveraging that you can get some really nice benefits.

Anyway, congrats on the launch! Reflect looks great. I'm really excited to see the building blocks emerge for local-first software.

Reflect could be built using Erlang. Erlang could not be used to replace what it does.

Specifically does BEAM build in a data structure for preserving user intent while multiple users type/delete/apply styling in the same text field?

https://github.com/untu/comedy

However to the below point, Bolt ons are never as good as baked ins. This is especially true for concurrency.