How it works

1. Data comes in as events

An import fetches data from a source. For now these are commands we run by hand. The sources are EveryElection for councils and wards, LGSF's scrapes of council websites for councillors, and Democracy Club's results for elections. The import compares each record with what we already hold and writes one event for each field that changed. Each event sets one value on one thing.

Here is what EveryElection's entry for Stroud's Thrupp ward became:

Run the import again and nothing is written, because nothing changed. If EveryElection renames the ward, one division_name_observed event is written, and nothing else. The same goes for corrections and matches made by people on this site: each one is an event too, recorded with who made it.

2. Aggregates and projections

Two words do most of the work here, and they mean different things.

An aggregate is the thing an event is about: one ward, one person, one councillor's seat. It isn't a table. It's an identity: every event names its aggregate's type and id, and all the events with that id, in order, are that thing's history. The Thrupp events above all name the same division aggregate.

A projection is a table built from events, for reading: an ordinary Django model whose name ends in Projection. Each event type has a small function, a handler, that applies it: division_name_observed sets the name column on one ward's row. This happens at the moment the event is saved, in the same transaction. Pages never replay events; they query projections with indexes, like any Django site.

Most aggregates have one projection, so the two are easy to confuse. They aren't the same: a seat's events build two projections, the public record and the bot's snapshot (section 3), and disagreements are a projection with no aggregate of their own, worked out from people's and seats' events. Here is every projection, with its live counts:

So the code has two sides. The write side decides what happened and records it as events; it is the only code that writes to the store. The read side turns events into projections and shows them; it never writes an event. Handlers are the bridge, and they only ever cross one way. This is enforced, in Python and in Postgres: anything else that tries to write a projection, or change a stored event, is refused.

Rebuilding is something we choose to do, say after changing a rule. Replaying all 0 events takes about half a minute.

3. When the website and a person disagree

Two kinds of writer change the same records. The scraper reads council websites again and again. People fix what the website gets wrong. The risk is an edit war: a person fixes a name, the next scrape puts the old one back, and the fix is lost without anyone noticing.

The answer is that the scraper never looks at the public record. It keeps its own bot snapshot of what the website said last time, and only writes an event when the website changes. Here's how that plays out:

So the website can't quietly undo a person's work, and a real change on the website is never lost either: it is shown to a person to decide. The same rule covers matching. A link made by "Match automatically" never replaces one a person chose.

The disagreement itself isn't stored as an event. It's worked out from the events every time the tables are built, so a better rule can be applied to the whole history. The person's decision is an event, because nothing else could tell us what they chose. You can see both records side by side under "Stored records" at the foot of any person's page.

4. Try it: one person, event by event

A made-up councillor with fifteen events: a first scrape, an automatic ward match, two corrections, and the council's website changing things a person had already corrected. Move the slider to replay the log up to any point; the record is rebuilt from the start each time, as the real tables are. Switch between the public record and the bot's snapshot to see why the scraper compares against its own copy.

This runs in your browser on its own small copy of the rules. It doesn't read the site's data.

5. In the code

Nothing here is a framework: four small pieces, each in a file you can open. Events are defined up front, written in one place, and applied by plain functions.

flowchart LR
    S["Source
EveryElection, LGSF, YNR,
or a person on the site"] --> F
    subgraph write["Write side: events"]
        F["event_factory.py
compares with what we hold"] -->|"one event per changed field"| A["append_event()"]
        A --> E[("events_event
append-only")]
    end
    subgraph read["Read side: projections"]
        H["handlers.py
HANDLER_MAP"] -->|"one setattr"| P[("Projections
PersonProjection, …")]
        P --> V["Pages
ordinary queries"]
    end
    A -->|"same transaction"| H
  

Event types are defined, not made up

Every event type is named in events/event_types.py, grouped by aggregate: these are the person aggregate's. The name says which field it sets; the payload only says which record, and the value.

class PersonEventType(StrEnum):
    CREATED = "person_created"
    NAME_UPDATED = "person_name_updated"
    YNR_LINKED = "person_ynr_linked"
    CONFLICT_RESOLVED = "person_conflict_resolved"
    MERGED = "persons_merged"
    NOTE_ADDED = "person_note_added"

Stored, a person_name_updated event looks like this. Every event has the same columns:

event_type      = "person_name_updated"
aggregate_type  = "person"
aggregate_id    = "fdb7e3d7-…"           # which person
actor_type      = "bot"                  # or "human", "system", "gdpr"
actor_id        = "lgsf/MAN"             # which scraper, or which user
payload         = {"person_id": "fdb7e3d7-…", "value": "Amna Saad Omar Abdullatif"}
sequence_number = 41208                  # Postgres decides the order

One way in

Imports and forms never touch a projection. They call a function in an app's event_factory.py, which decides what changed and calls append_event() (events/append.py) for each change:

append_event(
    aggregate_type="person",
    aggregate_id=person_id,
    event_type=PersonEventType.NAME_UPDATED,
    actor_type="human",
    actor_id=request.user.get_username(),
    payload={"person_id": person_id, "value": "Amna Abdullatif"},
)

A projection is a normal Django model

This is the whole PersonProjection (people/models.py). Each value has a few columns beside it saying who set it, with which event, and when. That's what lets every page say where a fact came from.

class PersonProjection(models.Model):
    person_id = models.CharField(max_length=200, unique=True)

    name = models.CharField(max_length=300, blank=True, default="")
    name_actor_type = models.CharField(max_length=20, default="")
    name_event_id = models.UUIDField(null=True, blank=True)
    name_updated_at = models.DateTimeField(null=True, blank=True)
    name_is_human_set = models.BooleanField(default=False)

    ynr_person_id = models.CharField(max_length=50, blank=True, db_index=True)
    ynr_person_actor_type = models.CharField(max_length=20, blank=True)
    ynr_person_event_id = models.UUIDField(null=True, blank=True)
    ynr_person_updated_at = models.DateTimeField(null=True, blank=True)

    has_conflict = models.BooleanField(default=False)
    merged_into_person_id = models.CharField(max_length=200, blank=True, default="")

Handlers connect the two

The model doesn't know about events. Each app's handlers.py has one small function per event type, and a HANDLER_MAP saying which function applies which type. This is the only place an event turns into a column:

def handle_person_ynr_linked(event):
    p = event.payload
    people = PersonProjection.objects.filter(person_id=p["person_id"])
    if event.actor_type == "system":
        # A matching rule never replaces a person's decision.
        people = people.exclude(ynr_person_actor_type="human")
    people.update(
        ynr_person_id=p["value"],
        ynr_person_actor_type=event.actor_type,
        ynr_person_event_id=event.id,
        ynr_person_updated_at=event.occurred_at,
    )


HANDLER_MAP = {
    PersonEventType.CREATED: handle_person_created,
    PersonEventType.NAME_UPDATED: handle_person_name_updated,
    PersonEventType.YNR_LINKED: handle_person_ynr_linked,
    # …one line per event type
}

So adding a new field means four things: an event type, a handler, a column on the model, and the factory function that writes it. The repetition is on purpose: you can read any handler and see exactly what it changes.

Why bother?

Because every value can say where it came from. Each person's page has a "How we know this" table, and Behind the scenes lists every event. The tables can always be thrown away and rebuilt; the events are the record.