~

Roadmap

Improve Network Performance

Summary

An ordinary Urbit ship needs to be able to host a large chatroom, in addition to other scaling considerations. The performance of Urbit's networking is the bottleneck limiting this kind of scaling at the moment.

A number of incremental improvements need to be made to the implementation of Ames, Urbit's networking protocol, including fixes for flaky connections and more efficient use of timers. A second protocol called Fine will also be added for scalable content distribution. A multi-part project called "subscription reform" will allow apps to use this protocol effectively.

Projects

Completed

Consolidate Packet Re-Send Timers

Ames sets a lot of timers to remind it to send packets again if they don't get acknowledged fast enough. Reducing the number of timers lowers disk write usage, improves quiescence (which should eventually let hosting providers use less RAM and thereby lower costs), and should improve overall performance on publisher ships.

Completed

Drop Pokes to Dead Agents

Adds an interface between Ames and Gall that lets Gall tell Ames to drop an incoming %plea packet in case the plea targets an agent that isn't running. The sender will retry every two minutes indefinitely, so if the agent starts running, the %plea will eventually go through, with minimal state on the receiving ship.

Completed

HTTP Mutable URL Caching

Urbit should be able to serve websites at custom URLs efficiently to support serving websites to the old web from a single ship without needing to configure caching reverse proxies such as nginx or varnish.

Completed

HTTP Scrying: Arvo

This makes the scry namespace available over HTTP enables reads of Urbit data from the clearweb.

Completed

Informal Ping

Introduces an informal (stateless) ping mechanism, moving route maintenance out of arvo and dramatically reducing the load on galaxies, without requiring operator configuration.

Completed

Refactor Ames Vane

The Ames vane could be shorter, easier to read, more performant, and easier to prove correct.

Completed

Basic Remote Scry Protocol

The "Fine" remote scry protocol will form the foundation of scalable content distribution in Urbit, by allowing many subscriber ships to read data efficiently from a publisher ship without incurring excessive load on the publisher.

Completed

Solid-State Publications Prototype

For scalability, basic solid-state publications are being prototyped. This is the next step after basic remote scry toward scalable data publishing on Urbit. These prototypes live in userspace; once we've proven the model with some real-world examples, we can build kernel support into Gall.

Completed

The way ships ping their sponsors doesn't always work well behind home routers or other NATs, causing flaky connections.

Completed

Tune %clog Backpressure Time and Space Constants

Tuning the constants in Ames's `%clog` system should reduce network flakiness, reduce the number of Arvo events, and improve publication bandwidth, especially until chat uses remote scry and solid-state subscriptions.

Completed

Typed Paths

Typed paths should improve performance and developer experience, and it could unblock a typed interface to publications.

Future

Commit Before Compute

duration

1-3 Months

owner

TBD

Current Vere needs to run Nock on an event before it can write it to disk. This places a lower bound on event latency, defined as time between receiving an event and performing its effects, at `D + N`, i.e. disk write latency (`D`) plus nock execution time (`N`). Commit-before-compute has amortized latency `max(D, N)`, which is usually significantly better.

Future

Generalized Deferral Mechanism

duration

TBD

owner

TBD

Arvo's Behn vane (kernel module) currently serves two purposes: setting timers, and deferring tasks to later events. Deferral could be split out into a separate feature, which could aid both in refactoring Behn to be easier to verify and optimize.

Future

Shared Memory IPC

duration

2-3 Months

owner

TBD

Vere's two processes, Urth (the I/O process) and Mars (the Nock worker process), communicate using a custom noun-based interprocess communication (IPC) protocol. This currently uses the Unix stdin and stdout, but using shared memory instead would make IPC significantly faster, reducing event processing latency and improving overall data throughput.

Future

Timer Improvements

duration

TBD

owner

TBD

Urbit's timer system could be better in several ways.

Future

Typed Interface to Solid-State Publications

duration

4-8 Months

A typed interface to solid-state publications should improve developer experience and performance (by avoiding runtime typechecks and type coercions).