It’s taken me a while to wrap my head around how security might work in a completely
decentralized, peer-to-peer application. I’ll dig into the big questions around local-first
authentication and authorization and explain how I’ve approached solving these problems in a new
library, <a href='https://github.com/local-first-web/auth'>@localfirst/auth</a>.

Establishing trust in the strange world of peer-to-peer collaboration

Alice and Bob in wonderland

CRDTs (conflict-free replicated datatypes) are exciting. They’re widely regarded as a key enabling
technology for creating local-first software — or for building any kind of decentralized multiplayer
system, for that matter.
The idea emerged from academia ^^TK details^^
When I first became interested and was digging around to try to understand how they work, I took one
look at the academic literature and decided it was over my head and I didn’t really need to know the
details.
Some time later, I was making a <a href="https://github.com/local-first-web/auth">thing</a> and once I got it
working, I realized that I had created a CRDT without intending to.
I still don’t know for sure what a semilattice is. At some point I intend to educate myself, but for
now I think I have enough of a practical understanding of CRDTs to equip you, the reader, with
reasonably good intuitions about how they work.
<h3 id="what-problem-are-crdts-intended-to-solve">What problem are CRDTs intended to solve?</h3>
Think about the difference between Microsoft Word running on your computer and Google Docs running
on a server somewhere.
The selling point of Google Docs is that any number of people can work on a document at the same
time, and you don’t have to worry about reconciling conflicting versions of a document.
<aside>

Of course, there’s now a cloud-based version of Microsoft Word; and it’s also kind of true that
Google Docs can now work offline.
</aside>

With Word, you can have your documents stored on your own hard drive on your own machine, which has
its advantages: You can work on your stuff when you’re offline, for example. And more generally, you
have ownership and control over your stuff. ^^(link to article)^^
On the other hand, working together on a document can be a nightmare if everyone has their own
copies on their own computers.
The promise of local-first software is that we might get the best of both worlds: Software that
works without a server and lets us have our stuff on our own devices, like Word, but where our
devices talk directly to each other and make collaboration as easy and seamless as it is with Google
Docs.
The big thing that makes that complicated to pull off is the problem of resolving conflicts
between different versions of a document.
The reason you don’t have to worry about conflicts with Google Docs is that there’s only one version
of the document — the one on Google’s servers — and you have to be online to edit it. But any
software that lets you work while offline has to deal with the prospect of conflicting changes — and
they often deal with it by panicking and throwing the conflicts back at the user.
^^TK illustration of conflict error messages^^
The idea behind CRDTs, as the name suggests, is that you don’t have to worry about these conflicts,
because they’re resolved automagically.
So how does this work? The “magic” of CRDTs boils down to this:
<ol>
<li>Keep track of every change everyone makes</li>
<li>Define rules for mechanically deciding what to do about changes that are conflict with each
other</li>
</ol>
<h3 id="living-in-the-present-considered-harmful">Living in the present considered harmful</h3>
The typical database system is organized primarily around the way things are right now. The way
things were in the past, and the fact that things might change at any minute, are
current state of your data. All
the information about how we got to that state might be recorded for a while, in the form of logs.
But this history of changes — the sequence of individual edits, additions, and deletions that
were made along the way — gets no respect. It’s an afterthought, treated as a transient and
ultimately disposable artifact.
The first thing I’d like to convince you of is that this approach gets it backwards: That log of
changes? That’s the good stuff. That’s the real first-class data, and we should treat it with
respect. The current state of the database? We don’t even need to store it, in principle. As long as
we’ve preserved the full history, we can recalculate the latest state at any time by starting with
nothing and reapplying all of the changes in order.
Suppose Alice’s copy of a contact database says that I live in Barcelona, but Bob’s copy says I live
in Washington DC. Which version is correct? Without the logs or some other form of metadata,
it’s impossible to know.
<table>
<thead>
<tr>
<th>First</th>
<th>Last</th>
<th>City</th>
</tr>
</thead>
<tbody><tr>
<td>Herb</td>
<td>Caudill</td>
<td>Washington DC</td>
</tr>
</tbody></table>
<table>
<thead>
<tr>
<th>First</th>
<th>Last</th>
<th>City</th>
</tr>
</thead>
<tbody><tr>
<td>Herb</td>
<td>Caudill</td>
<td>Barcelona</td>
</tr>
</tbody></table>
But if this database, rather than reifying the current state of things, records the actual events that brought us to this state, it becomes trivial to sort it out. Maybe Alice’s database has one entry like this:
<pre><code class="language-txt">1999-09-01 | update Herb Caudill | city: Washington DC
</code></pre>
Whereas Bob’s database has two entries:
<pre><code class="language-txt">1999-09-01 | update Herb Caudill | city: Washington DC
2016-02-01 | update Herb Caudill | city: Barcelona
</code></pre>
Now it’s obvious what’s going on and what needs to happen — Alice is simply missing the most recent change, and once she has it, Alice and Bob will be synced up and up to date.
Martin Kleppmann calls this “turning the database inside out” — giving the logs primacy and treating
the current state as a derived artifact, rather than the other way around.
This turns out to be a good idea for lots of reasons. For example, you have straightforward access
to your state at any point in time, not just the present. If you’re a programmer and you’ve used
React for a while, you might have had your mind blown a few years back by Dan Abramov’s “<a href="https://www.youtube.com/watch?v=xsSnOQynTHs">time
travel</a>“ demo for Redux. Rich Hickey’s seminal talk
“<a href="https://www.youtube.com/watch?v=Cym4TZwTCNU">Deconstructing the Database</a>“ explores lots of other
advantages.
The advantage we’re interested in right now is how it simplifies conflict resolution, and there the
key insight is that conflicting states are hard to resolve automatically, whereas conflicting
actions can be resolved in a principled way.
<h3 id="time-in-a-world-without-clocks">Time in a world without clocks</h3>
In the example above, you’ll notice that the timestamps on the changes were essential to our ability
to sort things out. Without them, Alice and Bob wouldn’t know the order the changes came in —
whether I lived in DC and then moved to Barcelona, or vice versa. And without knowing the correct
order of the changes, they wouldn’t be able to clear up the current value of the city field in
my contact record.
There’s a problem, though: It’s not a good idea for distributed systems to rely too much on
timestamps to resolve conflicts, especially when the stakes are high. Without an authoritative
server, you don’t have an impartial shared clock; and you can’t trust self-reported timestamps. A
device’s clock might be off. Or, less innocently, someone might be deliberately manipulating them in
order to make sure their version of things wins out.
So we need a way to keep our log of changes in the correct order, without relying on clock time.
There are a couple of ways to do this. I’m going to focus on the one I like best, which is hash
chaining. This might ring a bell if you’re familiar with distributed source control systems like
Git, or with the way the blockchain works in cryptocurrencies like Bitcoin.
<aside>

A hash can be thought of as a fingerprint for data. A hash function takes a chunk of data of any
size, and returns a random-looking short code called a digest that uniquely1
identifies that chunk of data. It’s impossible2 to engineer a collision - that is, come
up with two inputs that give the same fingerprint. Whether the data is one password or the collected
works of Shakespeare, the hash is always the same length. If you change the source data, even by a
single letter, you get a completely different fingerprint. The hash is deterministic — given
the same input, you always get the same output.
1For all practical purposes. 2For all practical purposes.
For example:
SHA512 hash of “Herb Caudill”: <code>1fbbc7bb4bdb50b9ce97b1025f6530c302...</code>
SHA512 hash of “Hurb Caudill”: <code>9f4a55639d69b6f3479f8440b2d2de2e79...</code>
</aside>

A hash chain is a sequence in which each entry includes the hash of the entry that precedes
it. This creates a self-verifying data structure — the relative order of the entries cannot be
manipulated, and neither can the contents of the entries.
^^TK illustration^^
In Git, these entries are called commits. On the Bitcoin blockchain, they represent
transactions. In both cases, storing changes on a hash-linked chain ensures that entries can
neither be reordered nor modified without invalidating the rest of the chain.
<h3 id="everything-all-at-once">Everything all at once</h3>
We still haven’t gotten to how we deal with conflicts. But what is a conflict, anyway?
If Alice changes my city to Barcelona, and Bob sees that change, and then changes my city it to
Istanbul, that’s not a conflict. These changes happened in sequential order — Alice’s first, then
Bob’s — and the most recent change wins.
But what if Alice is offline when she changes my city to Barcelona and Bob changes it to
Istanbul? That’s a conflict, because it’s not clear what the software is supposed to do.
Why is there no obviously right answer? Because there’s no obviously right way to put the
changes in sequential order. We can’t trust timestamps, so we can’t say that Alice’s change came
after Bob’s, and we can’t say the opposite either.
<aside>

In everyday speech, when two things happen at the same absolute time, we say they happened
simultaneously.
In computer science, when two things happen at the same relative time, we say they
happened concurrently.
</aside>

If we only have relative time, we have to think of these changes as happening “at the same time”,
or concurrently.
Of course, not all concurrent changes are in conflict. For example, if Alice edits my city and Bob
concurrently edits my phone number, there’s no conflict. If Alice edits my city and Bob concurrently
edits someone else’s city, there’s no conflict. In these cases, we can put Alice’s change before
Bob’s, or Bob’s before Alice’s, and either way we get the same result.
So in the case of our contact database, the only time you have a conflict is when the same field
on the same person’s record is changed concurrently by two different people. In practice,
this happens pretty rarely; but it’s always a possibility, and we need to be able to deal with it.

Conflict free something something

CRDTs demystified

As an American living in Spain, every once and a while I’m mildly surprised to be reminded that
Spain still has a king (and a queen, and two little teenage princesses) whose lineage — and thus his
authority — can be traced back to the dark ages.
<figure>

<img src="/images/posts/20220115-is-this-a-server/felipe.jpg" alt="">
Felipe Juan Pablo Alfonso de Todos los Santos de Borbón y Grecia, King Felipe VI of Spain 🙄
</figure>

Back when some humans first got the notion of replacing monarchs with more democratic forms of
governments, most other humans thought this was a preposterous suggestion.
For starters, it took a while for the idea to sink in that kings and queens were actually just
people like the rest of us.
But even once that was conceded, there were still legitimate questions to be answered. Monarchs
played a number of roles, both practical and symbolic, that we would need to fulfill some other way.
Who would make the laws? Who would enforce them? Who would lead us into battle? Who would be the
decider of last resort in contentious court cases? What would unify us as a nation? Who would
represent us to the world?
We’ve found pretty good answers to all of these questions. To get to where we are, we had to give a
lot of thought to the various jobs that royalty carried out, go back to first principles, and come up with
new ways of fulfilling these functions: new roles, new
<h3 id="this-is-a-metaphor">This is a metaphor</h3>
Servers are just computers. If we want to build collaborative software that doesn’t rely on
centralized servers , we’ll need to understand the utility that they currently provide, and figure
out different ways of fulfilling those needs.
Let’s start by enumerating the things
<h2 id="what-are-servers-are-good-for">What are servers are good for?</h2>
Servers do a lot, but the value they add may not be what you think it is.
I think maybe in the popular imagination, we need servers because they’re bigger and more powerful
than consumer-grade computers. That’s certainly the case for some applications. But as CPUs and GPUs
get faster and faster and SSDs get bigger and bigger and everything gets cheaper and cheaper, there
are fewer and fewer applications that are too big or too complex to run on users’ own devices.
So for the moment let’s disregard use cases that involve vast amounts of data or server farms’ worth
of computation. Instead, think of something like a community group’s project management system, or a
medium-sized company’s financial records, or a sales team’s database of leads and deals and
customers.
Even with these modest applications, we depend on servers to make collaboration happen. I’ll divide
the essential services they provide into five big categories: consistency, discovery,
persistence, authentication, and authorization.
Consistency A server provides a single central point of contact for collaborative applications
to read and write data. The data in the cloud acts as our single source of truth. There’s no
possibility of two people making conflicting changes.
Discovery There are a couple of things that turn an ordinary computer in to a server. One is
that it has a stable IP address. This makes it easy for two users to find each other in the vastness
of the internet. I connect to a URL from wherever I am in the world, you connect to the same URL
from wherever you are, and just like that we’re connected.
Persistence Another thing that makes a computer a server is that it’s always on and always
online. Typically there are teams of skilled people who have that as their only job. This means that
we can collaborate asynchronously: I can make some changes, log off and go to sleep, and when
you come online you’ll see my change
Authentication A server can securely store sensitive information — passwords, for example — and
then use that to challenge me to prove who I am. If I can’t prove that I know my password, it can
refuse to connect with me, and thereby prevent me from connecting to others. So the server acts as
trusted third party, essentially vouching for my identity to my collaborator
Authorization If there’s something I’m not allowed to see, a server just won’t let me have it;
and if there’s something I’m not allowed to do, a server simply won’t allow me to do i
<h3 id="how-do-we-do-these-things-without-servers">How do we do these things without servers?</h3>
So if we want to build local-first software, we first need to figure out how to provide these
services without a server.
<ol>
<li>Consistency Without a server, if we each have our own local copy of the data, how do we stay
in sync? What if our edits are in conflict with each other?
<div class='spoiler'>

A: An emerging class of technologies called conflict-free replicated data types (CRDTs)
allows software on users’ own devices to sync up directly with each other, automatically merging
changes on both sides, and resolving any conflicts in a predictable way.
 </div>
</li>
<li>Discovery Without a server, how can my laptop connect directly to your phone? Neither my
laptop nor your phone has a stable IP address, so how do we find each other in order to
communicate?
<div class='spoiler'>

A: While we’re waiting for true peer-to-peer networking technologies to become mature and
widespread, we can use relays — tiny, generic servers that can be deployed to the cloud at
little or no cost — that allow two devices anywhere on the internet to instantly connect to each
other.
 </div>
</li>
<li>Persistence Without a server, how do we communicate asynchronously? Our laptops and tablets
and phones are often offline or powered down. We can’t really collaborate effectively if we
always have to be online at the same time.
<div class='spoiler'>

TK
 </div>
</li>
<li>Authentication Without a server acting as a trusted third party, who vouches for me so I can
prove who I am to my collaborators? They don’t know my passwords (I hope!), so how can they be
sure that I’m not an impostor?
<div class='spoiler'>

TK
 </div>
</li>
<li>Authorization Without a server to keep track of the rules, how do we know who’s allowed to
do what? And if everybody has a full copy of the data, how can we keep some people from seeing
sensitive information?
<div class='spoiler'>

TK
 </div></li>
</ol>
To summarize:
<table>
<thead>
<tr>
<th>A server is…</th>
<th align="left">because…</th>
<th align="left">as a result it can provide…</th>
<th align="left">which means we can…</th>
<th align="left">In a peer-to-peer world, we might instead use…</th>
</tr>
</thead>
<tbody><tr>
<td>available</td>
<td align="left">it’s always plugged in, powered on, and online</td>
<td align="left">persistence</td>
<td align="left">collaborate asynchronously</td>
<td align="left">cloud clients</td>
</tr>
<tr>
<td>findable</td>
<td align="left">it has a public IP address</td>
<td align="left">discovery</td>
<td align="left">find each other</td>
<td align="left">relays</td>
</tr>
<tr>
<td>trusted</td>
<td align="left">it is under the control of a known entity</td>
<td align="left">authorization</td>
<td align="left">enforce read and write permissions</td>
<td align="left">signature chains, lockboxes, and attention</td>
</tr>
<tr>
<td>a third party</td>
<td align="left">it is not a peer</td>
<td align="left">authentication</td>
<td align="left">be sure of each other’s identity</td>
<td align="left">signature chains and signature challenges</td>
</tr>
<tr>
<td>authoritative</td>
<td align="left">we agree that it is the only source of truth</td>
<td align="left">consistency</td>
<td align="left">edit shared data without conflicts</td>
<td align="left">CRDTs</td>
</tr>
</tbody></table>

Before we can move past the server-centric model for collaboration software, we need to understand all the things we’re currently depending on servers for.

Five functions that servers carry out

Deconstructing the server

Imagine you were trying to reach a mountain peak, in heavy fog, with no maps or trails. How would
you proceed?
One smart heuristic might be simply “head uphill, until you can’t get any higher”.
If you think about it for a moment, though, you might realize that there’s a problem with this
approach. Maybe if the mountain you’re climbing looks like Mt. Fuji, you’re fine - but what if, like
most mountains, it includes a number of smaller peaks at different altitudes?
<img src="/images/posts/20220110-reasons-to-be-excited-local-first/mountain.gif" alt="">
If you only go up and never go down, you’ll very likely find yourself in what mathematicians call a
<a href="TK">local maximum</a>.
And if others are also using your same mountain-climbing heuristics, you might quickly find yourself
in the ridiculous situation of competing with a bunch of other people for this same false summit.
Here’s the thing, though - even if you can see higher peaks from where you are, it’s not altogether
irrational to keep heading for the nearer, lower peak. After all, you’re almost there. If everyone
else is there too, then the trail is well-worn so you won’t get lost. Going downhill just feels
wrong - especially while others are making progress uphill.
<h3 id="this-is-a-metaphor">This is a metaphor</h3>
I think that we in software industry have ended up in a local maximum. When I say the “software
industry”, I’m thinking specifically of those of us who make collaboration software: multiplayer
apps that help people work together over the internet.
In this metaphor the false summit, the local maximum, is cloud software, also referred to as
“web-based software” and “software as a service”. My hope with this article to persuade you that
there’s a higher peak out there - a better approach to creating collaboration software, which goes
by the name of local-first software. And I’d like to not only persuade you that this other peak
exists, but that we should be excited about it, that it’s worth the additional effort to get there.
<hr>
<h3 id="from-client-server-and-local-only-software-to-cloud-software">From client-server and local-only software, to cloud software</h3>
Twenty or thirty years ago, if I had an important document to write — say, a term paper or a
proposal - the only real choice was to fire up WordPerfect or Microsoft Word or MacWrite on my
computer and started typing.
But what if this wasn’t a solo project? What if I was working on a team? Maybe different people are
writing different sections. Maybe other people are helping with proofreading or editing.
Well, if you’re old enough to remember, you’ll know how this went: Before you know it everyone is
emailing each other files and you have 27 different conflicting versions, with filenames that only
grow longer and longer. One person might ask everyone to hold off on making any further changes
while they’re working on the document; but inevitably at some point you would still end up
laboriously re-applying edits from one document to another.
<img src="/images/posts/20220110-reasons-to-be-excited-local-first/filesharing.gif" alt="">
Around this time, if you wanted multiplayer software, you generally had to host it yourself; so this
was mostly the province of bigger companies that had the kind of infrastructure to do this.
Accounting software, CRM tools to help sales teams work together, that kind of thing. Sprawling
“enterprise resource planning” systems intended to do everything for everyone, provided by giants
like SAP and Oracle. For the most part these were delivered as “client-server” software - the
company would host a server, and each user would install the client application, which you could
only use within the company’s local network.
<hr>
Today, instead I can use web software like Google Docs. This makes it super easy for me and my
teammates to work together on a document, without emailing files or worrying who has the latest
version.
<blockquote>
TK graph of SaaS replacing installed software?
</blockquote>
The story of software over the last couple of decades is largely the story of installed software
giving way to software in the cloud. You have Google Workspace and Microsoft 365. You have project
management tools like Basecamp, Asana, Monday.com, Trello, and Jira. You have QuickBooks and
FreshBooks. For shared notes, there’s Notion, Coda, and Roam. Designers have tools like Canva and
Figma. Sales teams have Salesforce, FreshSales, and Pipedrive.
All of these products have roughly the same architecture: Your data lives on the provider’s server,
not in files on your computer. You have an account and you log in with a username and password. The
technical architecture involves a database layer and at least a couple of codebases — one for the
server and one for the client, which is typically delivered as a web application, and perhaps also
as native apps for platforms like iOS, Android, MacOS, or Windows.
<h2 id="why-we-like-cloud-software">Why we like cloud software</h2>
I’m going to be talking about an alternative way of building software, but to be clear, the
cloud-based model has a lot going for it. I depend heavily on the software products I just
mentioned myself. Not only that, I’m a cloud software provider myself. My small company, DevResults,
makes web-based monitoring and evaluation software for international aid projects.
What’s good about this approach?
<h3 id="users-get-easy-collaboration">Users get easy collaboration</h3>
For the user, the big advantage of cloud software is having a single source of truth: Your data
lives in only one place, and that makes collaboration easier.
You never have to worry about upgrading your software — it’s continuously upgraded so you’re
always using the latest version.
You can access it from anywhere, and it’s backed up for you — you don’t have to worry about
losing data if something happens to your device.
<h3 id="providers-get-control-over-customers-and-their-data">Providers get control over customers and their data</h3>
The advantages for the software developer are even greater. It’s very convenient having everything
on your servers; it gives you a lot of control. It’s much easier to debug, because you have
access to all the data and there are fewer unknowns regarding the environment.
Once a customer starts to depend on your software, they’re kind of locked in. It’s very
difficult for them to leave, because you’ve got their data. If they don’t pay their bills, you can
shut off their access.
<h2 id="what-weve-traded-off">What we’ve traded off</h2>
So there’s a lot to like here. But everything is about trade-offs, and I think it’s worth thinking
about what we traded off when we moved everything to the cloud.
<h3 id="users-have-given-up-the-ability-to-work-offline">Users have given up the ability to work offline</h3>
For the user, the tradeoffs all have to do with the fact that our data, which used to live on
our own computer, is now inconveniently located hundreds or possibly thousands of miles away, on
someone else’s computer. For starters, this means that we have to have a good, working networking
connection in order to do anything at all. If we’re offline, or having network problems, we have
no access to our stuff.
<h3 id="and-responsiveness">…and responsiveness</h3>
And our own network connection is just the beginning: Every leg of the journey between you and your
data represents a possible point of failure. And even if everything is working perfectly and every
step takes place at the maximum possible speed, which it won’t, you’ll still experience some
latency with every interaction. You’re ultimately limited by the speed of light, which starts to
matter at planetary scale.
<h3 id="and-ownership-of-our-data">…and ownership of our data</h3>
So the “hundreds or possibly thousands of miles away” part is inconvenient, but the “someone else’s
computer” part is what’s really troublesome. “Possession is nine tenths of the law”, and in a very
real sense, my document on Google Docs is not mine, it’s Google’s. My project on Asana is not
mine. It lives on someone else’s hardware, not mine, and if they want to, they can keep it from me.
There’s always the threat of being locked out of my stuff because my credit card expired, or
because my product was discontinued, or the company went out of business, or just because someone
goofed up.
<h3 id="-and-our-privacy">… and our privacy</h3>
Putting my stuff on someone else’s computers also means that I have less control over who sees my
stuff.
Once a provider starts to accumulate lots and lots of people’s personal data on its servers, those
servers become an attractive target for attackers.
You have no guarantee that your cloud software company’s employees aren’t <a href="http://vice.com/en/article/xwnva7/snapchat-employees-abused-data-access-spy-on-users-snaplion">rifling through your
things</a>.
In the U.S., the fourth amendment to the constitution specifically protects my “papers and effects”
from “unreasonable searches and seizures”; and so if law enforcement wanted access to my phone or my
laptop, they would need a warrant, and I would have all the legal protections of due process. But my
stuff stored in the cloud is <a href="https://epic.org/ecpa/">not protected</a>: Not only can the government
require providers to hand it over with just a subpoena, but then they can issue a gag order that
prevents my providers from even letting me know about the search and seizure that’s taking
place.
Now maybe you feel like you have nothing to hide, and you’re blasé about the prospect of law
enforcement going through your stuff. If you’re like me, your “papers and effects” are pretty boring
and the idea that they would be of interest to Interpol or the FBI is kind of hilarious.
But if that’s the way you feel, I think it’s worth thinking about the very big picture, and
reflecting on what a rare luxury it is to live in a time in history, and a place in the world, where
you generally are not worried that people in power will abuse that power. The culture and the
institutions that have made that possible aren’t universal facts of life — they’re human constructs,
and they’re fragile, and they’re not available to everyone. If you’re a government whistleblower,
you can’t be blasé. If you’re an activist in Hong Kong, you don’t have that luxury.
<h3 id="as-a-society">As a society</h3>
And I’ll say one last thing about what we as consumers and as a society have given up.
When we’re talking about power, and abuses of power, at least when it comes to the government, in a
democracy, we have the rule of law and the protections of due process. If I’m accused of something,
I can get a lawyer, I can ask to see the evidence against me, and so on.
Something that’s starting to worry me more and more is that we depend so much on just a few tech
companies, that they end up with the kind of power over our lives that normally we reserve for
democratically elected governments. So much is at stake if you’re booted off Twitter or Google or
Amazon that these companies have what amounts to internal judicial systems to deal with various
kinds of misbehavior. But unlike the real thing, in these shadow judiciaries you have no rights.
Big tech doesn’t have to prove anything to you, or even tell you what you’re accused of.
So I think as a society, “we the people” should be worried about the quasi-governmental power that
we’re conceding, little by little, to unelected, unaccountable private companies.
<h3 id="developers-have-taken-on-lots-of-extra-complexity">Developers have taken on lots of extra complexity</h3>
So the shift to cloud software has resulted in us users giving up a lot of control to software
providers. But for software developers, cloud software has been a mixed bag as well — especially for
small startups, and for individual developers with a good idea.
Matthew Weidner makes this point vividly in <a href="http://youtube.com/watch?v=Exr0iY_D-vw">this talk</a>. He
asks us to imagine that we have an idea for a better recipe-sharing app.
The client application itself — the software that the user interacts with — isn’t complicated: You
could code it up in a weekend.
But to turn that into a multi-user cloud-based app?
For starters, you need servers, and those cost money.
And you need a database, and a whole new codebase on the server. Before you know it, most of your
code is just about getting data from the client to the server to the database and back, and adding
the smallest new feature requires tons of soul-crushing boilerplate code.
And you need to think about user accounts, and passwords, and resetting passwords, and securing
passwords.
Suddenly this is not a weekend project, it’s a startup in search of funding — all because of the
added cost and complexity that comes with creating a cloud-based app.
<h3 id="where-we-are-today">Where we are today</h3>
In sum: Cloud software has made it much easier for us to collaborate, and that’s a wonderful thing.
But that has come at a cost.
We’ve decided that we’re OK with paying $10 per month here and $25 per month there, every month of
our lives without fail until we’re dead and maybe afterwards as well.
We’ve given up our ability work offline.
We’ve given up much of our expectation of privacy in our digital lives.
We’ve given unaccountable companies the power to decide whether or not we deserve to have access to
our own stuff.
And we’ve raised the barrier of entry for new software products, further entrenching the power of
the incumbent software companies.
<h2 id="is-there-a-better-way">Is there a better way?</h2>
Maybe the value we’re getting from this software is worth the cost we’ve paid — I’m not saying it’s
not!
But I think it’s worth asking: Is this state of affairs inevitable? Or is it possible to imagine
another way, that would give us the benefits of collaboration without giving up ownership of our
digital lives?
The local-first model starts with the question: To collaborate, couldn’t our devices just talk to
each other directly?
You and I want to work together, and we both have computers. So what do we need a third party’s
computers for?
After all, today’s laptops are far more powerful than the beefiest servers from just a few years
ago. Even the phones in our pockets would vastly outperform the servers that the SaaS industry
started out with.
Going back to the word processing example we started out with: With good ol’ Microsoft Word, I
didn’t have to give up my privacy, or the ability to work offline, or the confidence that I’d still
have access to my document next year. The only reason I switched to Google Docs was because it made
collaboration easier.
But what if my Microsoft Word could talk to your Microsoft Word directly, allowing you and I
to work together on a document, Google Docs-style? We would both have the full document stored on
our devices, and the software would take care of keeping our copies synced up.
That’s the vision behind local-first software: Applications that are primarily local to your
device — giving back the control and agency you should have over your data — but that still enable
collaboration, via peer-to-peer communication with the local applications on other people’s devices.
I think that’s a really exciting vision of the future — one that would get us back everything we
traded away when we switch to cloud applications, without making us give up the ability to
collaborate smoothly. The best of both worlds.
<table>
<thead>
<tr>
<th></th>
<th align="center">💿 traditional apps</th>
<th align="center">🌧 cloud apps</th>
<th align="center">👍 local-first apps</th>
</tr>
</thead>
<tbody><tr>
<td>⌚ fast: my data is on my device</td>
<td align="center">✅</td>
<td align="center">⛔</td>
<td align="center">✅</td>
</tr>
<tr>
<td>🚙 robust: network optional</td>
<td align="center">✅</td>
<td align="center">⛔</td>
<td align="center">✅</td>
</tr>
<tr>
<td>🕵️‍♀️ private: only I decide who sees it</td>
<td align="center">✅</td>
<td align="center">⛔</td>
<td align="center">✅</td>
</tr>
<tr>
<td>📦 mine: I own it and can take it with me</td>
<td align="center">✅</td>
<td align="center">⛔</td>
<td align="center">✅</td>
</tr>
<tr>
<td>👪 multiplayer: I can work with others in real time</td>
<td align="center">⛔</td>
<td align="center">✅</td>
<td align="center">✅</td>
</tr>
<tr>
<td>📱 multidevice: move from laptop to phone to tablet</td>
<td align="center">⛔</td>
<td align="center">✅</td>
<td align="center">✅</td>
</tr>
<tr>
<td>🌩 backed up: no worries about data loss</td>
<td align="center">⛔</td>
<td align="center">✅</td>
<td align="center">✅</td>
</tr>
</tbody></table>
But if you look around, you’ll see that very little software these days works that way. Why is that?
If this is such an attractive vision, why are we still building web apps? Why did Microsoft go to
the trouble of <a href="https://www.microsoft.com/en-ww/microsoft-365/free-office-online-for-the-web">rebuilding Word from the ground up as a web
application</a>, rather
than just adding collaboration capabilities to the existing program?
The short answer is that as an industry, we know how to build collaborative apps using a central
server, and we really haven’t figured out any other way to enable collaboration.
So if we want to build local-first applications without central servers, we need to understand
the jobs those servers are currently doing for us — and figure out a way to fulfill those needs
without them.

Cloud-based software has made collaboration easier, but at a cost to both developers and users.
Could a new model for software give us the best of both worlds?

Collaboration without giving up control.

Seven reasons to be excited about local-first software

Cryptography is a fundamental piece of the digital infrastructure we all rely on, and yet most of
us are pretty fuzzy on how it actually works. I’ve written this for programmers — as well as
curious civilians — who would like a clearer idea of what the pieces are and how they fit together.

Everything you need to know about digital encryption and signatures but were embarrassed to ask.

Cryptography 101

Note from the future: I wrote most of this article in 2020 as my own notes on my research. Once 
I discovered Tailwind, I stopped writing and got back to work, and nearly forgot about this piece. 
It’s now 2023, and in the interim Tailwind has gone from being super controversial to being a safe, 
even <a href="https://npmtrends.com/bootstrap-vs-tailwindcss">obvious</a> choice. So this all feels a bit like 
beating a dead horse at this point; but I’ve gone ahead and finished it and I’m posting it anyway, 
with the original title and date for context.

What tools would you reach for today to style the UI for a 
<a href='https://medium.com/all-the-things/the-trouble-with-saas-279694551b25'>hyper-customizable app</a>? 
I just spent an unreasonable amount of time trying to understand the current landscape and settle on the perfect
framework. I was surprised by what I found.

Thinking about front-end frameworks and CSS during interesting times

How to style React applications while the world burns around us

I started writing this as a way of organizing my own thoughts. I originally intended just to share it internally with my team at DevResults, but when it was nearly done I thought I might as well put it online. 
Less than an hour after I hit publish on Medium, DHH himself <a href="https://twitter.com/dhh/status/1097937474003685376">tweeted it out approvingly</a>. By the next day it was coming back to me in my own Twitter feed, as software developers who I admired responded to it: Nicholas Zakas, Anil Dash, Dan Abramov, Simon Willison. 
At a conference a few weeks later, I met Amir Sahlifendic, the founder of <a href="www.todoist.com">Todoist</a>. The conversation turned to the question of mature codebases. Amir said, “You should really read this article that came out a while back, about rewriting software…” 
At any rate, this is by far the most widely-read thing I’ve ever written. According to Medium, it’s been read from beginning to end by nearly 40,000 people – not bad for a half-hour read!

A new take on the age-old question: Should you rewrite your application from scratch, or is that “the single worst strategic mistake that any software company can make”? Turns out there are more than two options for dealing with a mature codebase.

Lessons from 6 software rewrite stories

Rewrite, refactor, or reinvent?

In a traditional client/server/database web app, so much effort goes into transferring and translating data between layers. Can we eliminate unnecessary complexity with a peer-to-peer design?

Rethinking the stack

A web app with no web server?

A key difference between a table in a spreadsheet and a table in a database is that database columns are typed: If you’re storing dates in a column, all the values in that column have to be dates.
Consistent data types are a big part of what makes a database a database — and what makes a database more powerful than a spreadsheet.
But, <a href="https://medium.com/all-the-things/a-single-infinitely-customizable-app-for-everything-else-9abed7c5b5e7">as we’ve seen</a>, data types are also a key point of friction for non-technical users, because they have to be nailed down too early in the process.
To get the best of both worlds, we want to avoid being heavy-handed: We don’t want to force Celeste to lay out her table schema in advance, but we do want to be smart and try to figure out her intentions. So if she types in a date, we’ll assume this is a date column. We’ll pick up the date format from the way she’s typed it in. And if she changes her mind and needs to change a column from one type to another, we want to make that as effortless as possible.
<figure>

<img src="https://miro.medium.com/max/3844/1*OZBN_m-HIuntmyIsHJfPyQ.png" alt="">
We want a column to pick up the date format from the way it’s typed in — just as it would in Excel.
</figure>

Before we go any further with this, though, let’s talk about data types.
<h2 id="towards-humane-data-types">Towards humane data types</h2>
I propose that the friction here doesn’t necessarily all come from having to choose a type in advance. The problem is also that most database systems force you to think about types from the computer’s perspective.
You only have a few fundamental types to choose from (text, number, date, boolean) and within those categories you have to make some commitments about how much space the computer should set aside. So for example, if you’re going the store numbers in a column, the system really wants you to make a lot of decisions in advance about size, precision, and so on.
<figure class='figure-lg'>

<img src="https://miro.medium.com/max/5002/1*8GaD1XKAYTfiyZ8MLwaaBg.png" alt="">
SQL Server’s datatypes. Do you want int or bigint? Any programmer worth her salt can tell you a story about making the wrong choice and coming to regret it.
</figure>

Human beings have different concerns. Does this number represent an amount in Euros or in Dollars? Or is it a measure of weight, or length, or area? What if it’s a distance in kilometers, but I want to show it in miles to some users? Does this text represent phone numbers or RGB colors?
With today’s databases, if you care about any of those things, you’re going to have to do the heavy lifting.
So basically you, the human, have to give a lot of thought to implementation details that should be the computer’s concern; but then when you want the computer to help you out with silly things like currencies or countries or weights or measures or oh, I don’t know, BASICALLY ANYTHING HUMANS CARE ABOUT, you’re on your own.
I think that’s the opposite of how it should be.
So what data types would actually be helpful to Celeste?
I think these categories ought to be front and center:
<ul>
<li>Dates and times</li>
<li>Physical measures: length, distance, weight, etc.</li>
<li>Names, addresses, and other multi-part human artifacts</li>
<li>Countries, states, currencies, languages, and other canonical lists</li>
<li>References to other things in the same system</li>
</ul>
Let’s look at each of these in turn.
<h2 id="dates-and-times">Dates and times</h2>
The passage of time is of fundamental concern to us humans, perhaps owing to the fact that we all die.
Most computer systems have something like a <code>datetime</code> datatype that allows you to identify an instant in time, down to the millisecond: for example, <code>Wednesday, June 7, 2018, 9:45:23.1234 (Greenwich Mean Time)</code>.
And maybe that is, in fact, what we are interested in.
But it’s often either a date with no time or a time with no date:
<ul>
<li><code>June 7, 2018</code></li>
<li><code>9:45 AM</code></li>
</ul>
Or, it could be a related, but distinct, concept such as:
A duration:
<ul>
<li><code>30 milliseconds</code></li>
<li><code>5 days</code></li>
<li><code>42 years</code></li>
</ul>
A period of time:
<ul>
<li>the year <code>2007</code></li>
<li>the month of <code>May 1997</code></li>
<li><code>Quarter 1 of 2019</code></li>
</ul>
A standalone day of the year, month or day of the week:
<ul>
<li>every <code>March 16</code></li>
<li>every <code>February</code></li>
<li>on <code>Tuesdays</code></li>
</ul>
Today’s databases are of very little help with managing these constructs.
Now, there are libraries in most programming languages to help programmers represent these concepts cleanly. For example, the <a href="https://js-joda.github.io/js-joda/">js-joda</a> library for JavaScript (based on the <a href="https://github.com/JodaOrg/joda-time">joda-time</a> package for Java) does a great job. But database tools supposedly designed for non-programmers — like Access or Airtable — mostly leave those users to struggle unassisted with the database’s raw data types.
This is going to be a recurring theme: There are so many great programming libraries and frameworks and concepts that have made things very clean and simple and straightforward for programmers. But no one has taken these ideas one small step further and wrapped them up in a form that makes them usable to civilians!
<h2 id="physical-measures-length-distance-weight-etc">Physical measures: length, distance, weight, etc.</h2>
Another amusing human idiosyncrasy is that we have multiple incompatible systems to measure the physical world.
The good news is that the rules for converting, say, <code>5&#39;8&quot;</code> to <code>172.72 cm</code> are well-known and involve simple math. Computers are better at simple math than people are. A thoughtfully designed system would take advantage of that fact to transparently take care of those conversions — so that one user could enter data in feet and inches, while another users would see the same data expressed in centimeters.
Again, there’s no shortage of free libraries to help with these concerns: For example, the <a href="https://github.com/ben-ng/convert-units">convert-units</a> library for JavaScript can handle not only everyday things like length, distance, weight, and temperature — but also more specialized physical measures like acceleration, frequency, or volume flow rate. But again, unless you have the wherewithal to write your own software, these don’t do you much good.
<h2 id="names-addresses-and-other-multi-part-human-artifacts">Names, addresses, and other multi-part human artifacts</h2>
If you’ve ever created a table of contacts in Excel, the first thing you’ve had to deal with is human names.
Perhaps you started with a single “name” column. Then, you realized you wanted to sort by last names, so you figured out how to automatically split a full name into a first name and a last name:
<img src="https://miro.medium.com/max/1260/0*Uny5s1vCSrUEtHCE.png" alt="">
But then start seeing names like these…
<ul>
<li><code>J. Shane Kunkle</code></li>
<li><code>Herbert “Herb” Caudill III</code></li>
<li><code>Josep Puig i Cadafalch</code></li>
<li><code>Dr. John P. Doe-Ray CLU, CFP, LUTC</code></li>
<li><code>Brigadier General Zebulon Pike</code></li>
</ul>
…and you decide to call in sick and spend the day under the covers.
And then there are addresses. Do you need one or two address lines? Three? Do you support international addresses? Do you need to break out street numbers? Apartment numbers?
What would the “pretend it’s magic” solution for handling names and addresses look like?
What I’d like would be to just paste them in, as-is — but have the computer figure out their component parts. That way I’d still be able to sort people by last name, or pull out their initials or their preferred given names. And I’d be able to display addresses in a standard format, or group them by zip code or by country.
<figure class='figure-xl'>

<img src="https://miro.medium.com/max/5496/1*G9H2sL8gsDaKF-TjgKIrgg.png" alt="">
</figure>

The good news is that we don’t need magic in this case, neither do we need to write a lot of code. Parsing and normalizing human names is hard, but it’s mostly a <a href="https://www.google.es/search?q=parse+human+names">solved problem</a>. Same for <a href="https://github.com/openvenues/libpostal">addresses</a> .
The same approach applies to other fascinating human artifacts like <a href="https://github.com/catamphetamine/libphonenumber-js">phone numbers</a>, <a href="https://github.com/bgrins/TinyColor">RGB colors</a>, and <a href="https://github.com/beaugunderson/ip-address">IP addresses</a>. There are tons of existing services and open-source libraries to choose from for each of these. There’s no reason to force people to come up with their own home-grown solutions to these basic needs.
<h2 id="countries-states-currencies-languages-and-other-canonical-lists">Countries, states, currencies, languages, and other canonical lists</h2>
Another hassle you might have dealt with in your contact spreadsheet is the problem of inconsistently entered names — for U.S. states, countries, and so on.
If you’re given this data and you want to do anything at all with these columns, you have to go through and make them consistent by hand.
<figure class='figure-lg'>

<img src="https://miro.medium.com/max/2784/1*O_msxV9aPmK-hM47iyux7Q.gif" alt="">
</figure>

And if you want to make a spreadsheet that forces people to choose from as set of options so that it stays clean, you have some hoops to jump through.
<figure class='figure-lg'>

<img src="https://miro.medium.com/max/2768/1*yd9gk-MNC8I7UQRBORrCyQ.png" alt="">
If you know how to make this happen, you’re in the elite 1% of spreadsheet power users.
</figure>

A humane system would take into account that (a) this is a really common scenario! and (b) we have standard lists for these things! — and make it easy to create a column with a datatype of <code>Country</code> or <code>US State</code> (or <code>Canadian Province</code>, or <code>Kyrgyz Oblast</code>, or what have you). Same goes for <code>Currency</code> and <code>Language</code>.
<h2 id="references-to-other-things-in-the-same-system">References to other things in the same system</h2>
One of the big limitations of spreadsheets is that it’s hard to link data from one table to another. Relational databases have this capability as a core strength — as the name implies — but as we’ve seen, most databases force us to think in an indirect and abstract way that doesn’t come naturally to most people.
This point — where Celeste realizes that what goes in a cell is actually a reference to a complex object that is defined elsewhere — is one of two key mind-melting junctures.
This shouldn’t be complicated.
Suppose I have a table of <code>Classrooms</code> and a table of <code>Teachers</code>. I want to assign a <code>Teacher</code> to each <code>Classroom</code>, so I make a column called <code>Teacher</code>. What’s the field type of that column?
It’s a <code>Teacher</code>.
What it’s NOT: A <code>Link</code>, a <code>Reference</code>, a <code>Foreign key</code>, or any other nerdy term that focuses attention the underlying implementation and makes it scary to civilians. Celeste shouldn’t have to decide whether this will be stored in a normalized or denormalized way. Celeste shouldn’t have to know an embedded document from a pointer. These are implementation details and they shouldn’t leak through to the UI.
Suppose I’m building a system that includes <code>Tasks</code>, <code>Projects</code>, <code>Companies</code>, <code>Customers</code>, and <code>Employees</code>. I should be able to treat each of these as a data type, and see them listed right there with all the other data types we’ve discussed. What goes in this field? A <code>Company</code>. Or a list of <code>Employees</code>.
Again, we’ve made this kind of thing straightforward and intuitive for ourselves as programmers. We’ve been living in an object-oriented world for over three decades now. We the have built-in primitives, yes, but we can define our own custom objects. Objects have properties that are in turn primitives or objects. This elegantly recursive paradigm makes it possible for us to model pretty much any real-world concept we can dream up, in all its detail and complexity.
So why don’t we put these tools in the hands of users?
Instead we restrict them to stupid grids and give them a handful of stupid datatypes to choose from. It ain’t right.
<h2 id="collections-of-things">Collections of things</h2>
The elegance of the object model depends, crucially, on our ability to represent arrays of objects in addition to individual objects. So there might be multiple <code>Employees</code> working on one <code>Project</code>, or multiple <code>Customers</code> in the same <code>Company</code>.
This is the second mind-melting juncture for Celeste: How do we fit a list of things — or a whole other table — into this little cell?
Again, as programmers we have no problem modeling this. You can have a thing, or you can have an array of N things.
In the latter case, it’s easy to end up with a clunky UI. But that’s a solvable problem.
What’s required is a willingness to try, and a willingness to stop infantilizing the user.
<h2 id="stop-treating-civilian-users-like-babies">Stop treating civilian users like babies</h2>
On that last point, a brief rant.
Since the early days of personal computing, we’ve made great progress towards making software easier to use. We’ve learned to say <a href="https://signalvnoise.com/posts/1626-the-most-powerful-word-is-no">no to new features</a>, we remind ourselves to <a href="https://www.amazon.com/Dont-Make-Think-Revisited-Usability/dp/0321965515">Don’t Make Me Think</a>, and we aspire to create software that <a href="https://en.wikipedia.org/wiki/Unix_philosophy">does just one thing well</a>.
That’s all healthy and positive — for the most part.
But I feel like one result of that has been that we software makers have collectively and individually ratcheted down our level of ambition. We’ve internalized a condescension towards end users that has led us produce more focused applications that do less and less, on the theory that anything more would overwhelm our poor customers.
As a result, there’s a huge chasm these days between what you can accomplish with user-facing tools and what you can accomplish by writing code.
Turns out, there are lots of intelligent non-programmers out there. There was a time — I know because I remember it — when there was tons of competition for that market. Throughout the late 1980s and early 1990s, there were dozens of innovative database systems designed for non-programmers to use: dBase. Hypercard. FileMaker.
Part of what happened, I think, was an accident of history: After Microsoft Office steamrolled the “office productivity” segment of the software industry, we were left with just two viable products in the user-facing database space: Excel and Access. Access has never sparked much enthusiasm or creativity, on account of it sucks and has always sucked. Excel has proved to be approachable and versatile enough that, in spite of not being a database, it has become the general-purpose database of choice. In doing so, it’s almost completely suffocated innovation in this space.
But those users are still out there, and I think they can be trusted with the concepts of arrays and objects and recursion.
It’s our job, as programmers, to offer up these facilities in a way that maps naturally to people’s intuitions; and there are a million ways to get that UX wrong. It’s our job to hide everything but what’s essential, to provide affordances, to avoid exposing distracting or intimidating implementation details. But it can be done, and it should be done, because it’s not right that all that creative energy is being wasted on twisting Excel into unspeakable contortions.
I’m going to stop there for now. Hopefully we’re a bit closer to having a tangible idea of what this tool might look like.
<figure class='figure-xl'>

<img src="https://miro.medium.com/max/3844/1*9d8mCVO-6fPfawXnPRXzTA.png" alt="">
</figure>

One possible way of presenting data types.
We want to make a tool that serves as a platform for humans to organize information for themselves and for teams they work on. To get there, I think we need to give a lot of thought to the types of data that humans are likely to organize. It believe it will be crucial not to dumb down this part down, but rather to use the familiar concept of the data type as a way of sneaking in the full flexibility and power of the object model.

Today’s databases force us to think about data in terms of what’s important to the computer. What if we had systems that started with things that are important in human society - things like timespans, birthdays, weights and lengths, names, addresses, and countries?

Software should know what kinds of information matter to people

Data types for humans

I have a long-standing love-hate relationship with the way computer interfaces are portrayed in films.
On the one hand, they’re so unrealistic! Lynne has patiently sat through any number of post-movie rants about this kind of thing:
<img src="/images/posts/20181222-secret-agent-database-for-the-rest-of-us/2.gif" alt="">
<img src="/images/posts/20181222-secret-agent-database-for-the-rest-of-us/3.gif" alt="">
On the other hand, they’re kind of awesome! At their best, film UIs capture the way we wish computers worked:
<img src="/images/posts/20181222-secret-agent-database-for-the-rest-of-us/4.gif" alt="">
<img src="/images/posts/20181222-secret-agent-database-for-the-rest-of-us/5.gif" alt="">
In the movies, no one ever spends five minutes looking for a file. In the movies, no one struggles to print something from their laptop.
In the movies, when you type someone’s name into the computer, you’re immediately presented with everything you need to know about them:
<img src="/images/posts/20181222-secret-agent-database-for-the-rest-of-us/6.png" alt="">
<img src="/images/posts/20181222-secret-agent-database-for-the-rest-of-us/7.png" alt="">
<img src="/images/posts/20181222-secret-agent-database-for-the-rest-of-us/9.jpeg" alt="">
<img src="/images/posts/20181222-secret-agent-database-for-the-rest-of-us/10.jpeg" alt="">
<img src="/images/posts/20181222-secret-agent-database-for-the-rest-of-us/11.jpeg" alt="">
<img src="/images/posts/20181222-secret-agent-database-for-the-rest-of-us/12.png" alt="">
<img src="/images/posts/20181222-secret-agent-database-for-the-rest-of-us/13.png" alt="">
<img src="/images/posts/20181222-secret-agent-database-for-the-rest-of-us/14.jpeg" alt="">
<img src="/images/posts/20181222-secret-agent-database-for-the-rest-of-us/15.jpeg" alt="">
<img src="/images/posts/20181222-secret-agent-database-for-the-rest-of-us/16.jpeg" alt="">
Mugshots, surveillance photos, maps, scans of paper records, ID cards, maps, fingerprints, 3D body scans: It’s all there, in one place, when you need it.
When I see the computer systems used by spy organizations in the movies, I always have the same three thoughts:
Thought #1: Why do secret agents type so loudly?
Thought #2: I’d love to just get a glimpse at the systems real-world spies use. Going by the Agency databases I have seen, I’m guessing they’re not all that great:
<figure class='figure-2up'>

<img src="/images/posts/20181222-secret-agent-database-for-the-rest-of-us/17.png" alt="">
<img src="/images/posts/20181222-secret-agent-database-for-the-rest-of-us/18.png" alt="">
Maybe the CIA saves all the good software contractors for their internally-facing systems — who knows!
</figure>

Thought #3: It sure would be nice if, at my job, I could type in, say, a customer’s name — and see everything I need to know about them, in once place.
So let’s take the secret agent use-case as a way of anchoring our thinking. Let’s focus on an imaginary user — let’s call her Celeste. Celeste has a desk job at the headquarters office of an intelligence organization. She’s smart. She’s not a programmer, and she doesn’t have a team of programmers at her disposal. It’s her job to put all the organization’s information about secret agents at her team’s fingertips.
Specifically, Celeste wants to be able to type in an agent’s name (klackety-klackety-klack) and see something like this:
<figure class='-xl'>

<img src="/images/posts/20181222-secret-agent-database-for-the-rest-of-us/19.png" alt="">
</figure>

This shouldn’t be rocket science, should it?
Now, this screen has no slowly rotating 3-D models, no live satellite tracking in grainy grayscale, no ANSI 381-encoded fingerprint templates. Just a bunch of facts. It doesn’t even need to make little futuristic beeps as it loads. This shouldn’t be hard, right?
Unfortunately, none of the tools we have today are of any use to Celeste.
So how is she making do?
Perhaps she already has a spreadsheet with a row for each agent, with their names and vital stats. But it’s limited: There’s no place for photos or documents. And when she was trying to come up with a good way of capturing agents’ language capabilities — a table inside a cell? a couple of linked tables? numbered columns? — <a href="https://medium.com/all-the-things/a-single-infinitely-customizable-app-for-everything-else-9abed7c5b5e7">her brain melted</a>.
<ul>
<li>She tried some database-spreadsheet hybrids like Airtable and Fieldbook, but they were not much better than spreadsheets: Neither one lets you customize a form like this; and neither one can handle things like the log of notes or the language skills table.</li>
<li>She looked at easy-to-use off-the-shelf systems for storing information about people— contact managers, CRMs, ERPs, etc. — but they weren’t made for the espionage business. With a little imagination, maybe you can stretch things a little bit — think of <code>Agents</code> as <code>Customers</code>, and <code>Missions</code> as <code>Deals</code>, and so on—but at some point you run into the limits of whatever customizability these systems offer.</li>
<li>And she probably considered highly customizable application builders — Mendix, Salesforce, Microsoft PowerApps, Google App Maker —but found them intimidating and unapproachable.</li>
</ul>
So it’s up to us to design a system that could help Celeste. Her happiness is in our hands.

Why shouldn’t our tools for organizing information be as cool as the ones in the movies?

A secret agent database for&nbsp;the rest of us

Computer, enhance

Before we create a “single infinitely-customizable app for everything else”, we need to understand
why we’re currently using spreadsheets for everything.

What would it take to replace the spreadsheet?

Excel hell: A cautionary tale

No matter what applications we choose, there’s lots of duplication and overlap between them. We’re paying for lots of stuff we don’t need or use. And there are still gaps that we have to cover with home-made systems. Surely there’s a better way?