thesephist

Notion, AI, and Me

Tue, 17 Jan 2023 14:11:30 -0500

Cross-posted from my Notion.

Over my last year of independent work, I built a lot of prototypes and stepped one foot into research land. I also learned a lot about how I work — what I enjoy working on, what I’m good at, what I’m bad at, and what I want to be doing long-term.

One thing I spent a lot of time thinking about is the life cycle of ideas: How a good idea can die on the vines because of bad or premature execution. How a bad idea can become the “default” because it just happened to be what everyone adopted when a big paradigm change swept the world. And how ideas, in general, have many stewards over their lifetimes; the best ideas often start in niche communities or research labs, then get picked up by products or communities with progressively less niche interests and incrementally wider distribution until, if we’re lucky, they reach billions of people.

Different kinds of people and companies excel at different parts of this grand propagation of ideas. Some people, often researchers, enjoy working at the edge of what we know. Others like to tinker with those fresh ideas and invent new products from their recombinations. Yet other teams make their mark taking the ideas and visions in a niche community and making them legible, accessible, and affordable for everyone. One thing I learned in 2022 is that I feel most fulfilled when I work on the early stages of an idea — learning new things about our world, and exploring what new powers those ideas give us in this world. But I want to get better at the later stages, and understand how they work. Because without the execution and distribution that help good ideas reach billions of people, the world won’t improve on its own.

Within the “tools for thought” and adjacent communities, I see a lot of people bemoaning the premature death of good ideas. Maybe the world is just unfair! Maybe the best ideas are doomed to fail, or doomed to be too early! “Worse is better! The simple dumb ideas win!” But that sounds overly defeatist to me, and ignorant of the long path that ideas, especially economically strange ideas like user interface design innovations, take to reach wide distribution. The work only begins at having the idea, and continues down the grand propagation until it’s in the hands of individuals far removed from labs and demos, using it to improve how they live their day.

That last, long stage — taking great ideas out of the lab and sculpting it into something billions of people could use — is a particular strength of Notion. Many features people love about this tool are ideas that are, at their core, technically complex, like end-user programmable relational databases or transclusion. This is also a skill set that seems often undervalued within niche corners of both “tools for thought” and AI communities.

My excitement about working at Notion comes from my optimism for bringing these skills, and the growing distribution Notion has in the hands of knowledge workers and creatives around the world, to some of the hard and interesting problems I’ve been thinking about for the last year. Questions like:

Language models decouple the way information is stored from the way information must be presented and consumed by humans. What does that imply about how we interact with information?
Programs are becoming more declarative, and documents are becoming more programmable. Will they merge? What does that look like?
How can we make writing less additive and fatiguing, and more like sculpting — starting with a raw form, and polishing down to the good stuff? How can we build the Photoshop for text?
Natural language interfaces feel like the should be super easy to use, but in practice, they can feel confusing and frustrating because they leave no room for affordances that tell you how exactly to command or control the tool. How do we fix this?

and obviously, my eternal favorite:

How can we improve humanity’s relationship to language and text? Can we use language models to build a better notation for ideas, something akin to what the Indo-Arabic numerals were to the Roman? Is there a better way to interact with ideas than writing, by directly manipulating concepts in latent space?

This is an exciting time to be building tools for creativity and thought, but also a pivotal time. When a new wave of products and companies and platforms sweep through, the winning tools often set the default interface metaphors and technical conventions, regardless of whether they were the best ideas available. One of my goals over the next few years is to ensure that we end up with interface metaphors and technical conventions that set us up for the best possible timeline for creativity and inventions ahead.

You can follow what’s coming to Notion in the coming months at @NotionHQ. I’ll keep writing at thesephist.com and stream.thesephist.com, too.

— Linus

Backpropagating through reasoning

Wed, 11 Jan 2023 07:29:28 -0500

Direct manipulation interfaces let us interact with software materials using our spatial and physical intuitions. Manipulation by programming or configuration, what I’ll call programmatic interfaces, trades off that interaction freedom for more power and precision — programs, like scripts or structured queries, can manipulate abstract concepts like repetition or parameters of some software behavior that we can’t or don’t represent directly. It makes sense to directly manipulate lines of text and image clips on a design canvas, but when looking up complex information from a database or inventing a new kind of texture for a video game scene, programmatic control fits better. Direct manipulation gives us intuitive understanding; programmatic interfaces give us power and leverage. How might we combine the benefits of both?

I started thinking about this after looking at Glisp, a programmatic drawing app that also lets users directly manipulate shapes in drawings generated by a Lisp program, by sort of back-propagating changes through the computation graph back into the program source code itself:

By resizing the radius of a circle on the canvas with my mouse, for example, I can change the number in the Lisp program’s source code used to compute the circle’s radius. Kevin Kwok’s G9 library does something similar: it renders 2D images from a JavaScript drawing program such that the drawings become “automatically interactive”. This is possible because, every time someone interacts with a point on the drawing, the input is backpropagated (this time, literally through automatic differentiation) through the program to modify it in a way that would produce the new desired drawing.

I really like this pattern of “push the interactions backwards through the computation graph to compute new initial states”. Its benefits are obvious for graphical programs, but also useful for more “information processing” programs. Imagine:

Reordering pieces of a table of contents with drag-and-drop to restructure and reword a piece of writing.
Searching for the weather, then dragging the temperature slider to 70 degrees to watch the date move forward in time, to find out when it’s going to get warm.
Looking up the definition of a word, then tweaking the definition slightly by editing the text in-place to discover a new word that means something slightly different.
Synthesizing a summary of an essay or meeting notes, and adding bullet points to it to edit the original document’s longform prose.

Some of this is possible today; others, like propagating edits to a summary back into source documents, requires rethinking the way information flows through big AI models. My past work on reversible and composable semantic text manipulations in the latent space of language models seems relevant here as well, and I’ve played with things like “reversible summary edits” lightly with it.

One way to think about language models is a programmatic interface to unstructured information. With that versatility and power also comes indirection and loss of control. Techniques for “backpropagating through reasoning” like this might be an interesting way to bridge the gap.

Build UNIX, not Uber

Sat, 16 Jul 2022 16:37:45 -0400

I spent the last three weeks in Seoul, partly visiting family I hadn’t seen in years and partly taking a vacation from my usual work in my usual city in my usual context of life. While I haven’t been able to work much at all, I spent my subway rides and waiting-in-line times reading the Singularity Sky series of novels by Charles Stross.

These stories unfold in a post-singularity world with a multi-planetary human civilization. One of the central ideas that Stross explores in the series is the idea of economic scarcity. Is it necessary? What happens as technology chips away at it slowly but surely over time? What’s its relationship to human conflict? In a post-scarcity world, how does wealth accumulate? What meaning does money and wealth have?

It’s really an excellent series, and you should read it if you find any of these questions intriguing.

Exploring this post-scarcity universe (I’m still deep in the middle of it) made me question my personal conceptions of scarcity and how it influences the way we understand wealth, success, and the value of good ideas. It’s this relationship between economics and the value of ideas I want to invite you to explore today.

A few months ago, I had a chance to ask a certain famous founder of a multi-billion-dollar software company a question. I asked,

Silicon Valley likes to treat venture-backed corporations as a kind of universal hammer, approaching every large – even civilization-scale – problems by building companies. Given that you’ve contributed to venture-backed companies, nonprofits, and other unorthodox research initiatives, what do you think of “companies” as a way to solve large problems facing humanity?

He said he thought, despite the hype, companies were still an underrated tool for creating large-scale societal change. His reasoning: companies are uniquely independent in a world of bureaucracy, because companies are only ultimately beholden to customers.

There’s an element of truth in this. While being so independent and autonomous, good companies can amass huge pools of resources to coordinate large-scale action. This coordination power can be put to work exploring a wide breadth of scientific solutions, attempting huge construction projects, or motivating political change.

There are, of course, other ways humans coordinate power at massive scale. A lot of them look like religion (social movements, political activism, and of course, religion itself). Some of them increasingly look like software (open-source communities, trustless decentralized coordination built on cryptography). The rest of them look like markets (venture capital, taxes, science, global trade). It’s too early to tell what a mature ecosystem of software-defined coordination will look like. But between religion and markets, it’s probably easier for single human beings to build and scale coordination power with companies than with religious movements.

Companies amass coordination power by having good ideas, executing them using more good ideas, and capturing a big part of the resulting value created (that is, when they’re not extracting value out of monopoly controls over markets). The effectiveness of corporations that the founder alluded to above comes from two things being true:

There is a scarcity of whatever the company makes.
There is a stable society and rule of law which let them capture the value they create.

In other words, companies cannot accumulate power if (1) everyone can get what they make for free, or (2) everyone can steal what they need or copy useful ideas without consequence. At various points in human history, these things has not been (or will not be) true. In a lawless society, companies can’t expect to capture most of the value they create – you get a world of piracy instead. In a hypothetical post-singularity society where ideas can be copied and designs can be reproduced for free in untraceable ways, companies can’t expect to capture most of the value they create. Companies, useful change-making instruments they may be, could turn out to be a gimmick of a particular phase of civilization – post-stability, pre-singularity.

A world in which companies can’t capture the worth of their work might sound apocalyptic, but I don’t think it has to be. There are corners of today’s world where we can observe what happens when companies don’t effectively capture the value of their good ideas.

The free and open-source software world is funded by companies, but not owned by corporations. Software is free to copy, and with the right permissive licenses, anyone with a good idea can copy and build upon existing open-source software to improve it or experiment with new ideas. If the stewards of a project are opposed to a new idea, proponents of the change can “fork” and try to win mindshare and support with their design or implementation.

Sometimes, companies lose ownership over their intellectual property by law. In 2024, the first versions of Mickey Mouse will enter public domain in the United States. Mickey will then join a pool of public domain literature and media that is free for anyone to remix, reuse, and reproduce for their own needs and creative tastes. This allows artists to create new work, obviously, but also allows initiatives like the free e-book library Project Gutenberg to exist. Project Gutenberg is one of several datasets used to train large language models like GPT-3. Free, public information has far-reaching consequences.

Elsewhere in the world, there are economic regimes that have intellectual property law that’s permissive or weakly enforced, to where ideas can be stolen and copied more easily. In China’s Shenzhen special economic zone, a lax approach to intellectual property law enforcement, in addition to the business and foreign investment-friendly policies in the zone, bloomed a tech hardware cornucopia. Shenzhen’s manufacturers and factories power a significant portion of the world’s consumer technology industry.

In all these environments where companies can’t capture the value of their work as effectively, the most powerful force in shaping society is how effectively good ideas spread in absence of explicit control and stewardship. Ideas that are useful, like material science and astronomy, spread effectively. But so do ideas that encourage their owners to spread them, like most religion; and ideas that grant their owners more power, like capitalism and weapons engineering. In a post-scarcity world, ideas are ascendant.

Of course, big ideas need resources like money and large-scale coordination to be implemented. Companies give their operators resources that’s useful for implementing ideas, like building rockets out of good rocket designs or reshaping democracy through an idea for how people can connect more easily online. But I think there’s reason to believe that resources required to implement big ideas are going down over time. Projects like the interstate highway system or the Manhattan Project required lots and lots of money and resources, but the Manhattan Project of today may be AGI, and AGI isn’t anywhere near as capital-intensive. More succinctly put:

Every 18 months, the minimum IQ necessary to destroy the world drops by one point.

There is reason to believe that, over the long term, humanity is moving from a regime where corporations steer society to a regime where autonomous ideas steer society.

We commonly view the history of tech invention and entrepreneurship through the lens of corporations and value capture. But I think there’s another way to study the twenty-first century tech ecosystem, as a battleground for good ideas, spreading autonomously.

UNIX was a good idea. It was free to spread widely across the tech ecosystem, and eventually some corporate players like Apple and Sun captured a lot of the value, but Linux stands out as the UNIX “winner” in 2022. Neither the core ideas of UNIX-style operating systems nor the Linux open-source project grew solely because of corporate stewardship. They spread and won because they were good ideas.

“Making inefficient markets more efficient with software” was a good idea, and companies like Uber, Doordash, and Etsy captured a lot of the value.

My favorite example is probably the Web. The Web – an interlinked collection of interactive documents made available through a decentralized Internet protocol – was a great idea. I think it’s no exaggeration to say that the Web won the platform wars between major operating system vendors without even really being an operating system. If you’re building a software company, 9 of 10 times, you better have a web application. The Web is an idea. No company owns it, and no company can.

I’m fascinated by this way of looking at how to make civilizational change: release good, powerful ideas into the world. And if circumstances allow, try to capture some of the value that comes from those ideas shaping the world in their image. Perhaps corporations will persist as the hammer and chisel with which history is carved through our generation. But perhaps not.

Charles Stross writes of Manfred, the protagonist in his novel Accelerando:

“You are very unusual. You earn no money, do you? But you are rich, because grateful people who have benefited from your work give you everything you need. You are like a medieval troubadour who has found favor with the aristocracy. Your labor is not alienated – it is given freely, and your means of production is with you always, inside your head.”

and later,

“You want to abolish scarcity, not just money!”

“Indeed.” Gianni grins. “There’s more to that than mere economic performance; you have to consider abundance as a factor. Don’t plan the economy; take things out of the economy. Do you pay for the air you breathe? Should uploaded minds – who will be the backbone of our economy, by and by – have to pay for processor cycles? No and no.

As the technology machine chips away at scarcity with its power tool called software, I wonder if our most resilient legacies would be made by searching for ideas that spread more than companies that scale. Building UNIX, not Uber.

Design with materials, not features

Mon, 20 Jun 2022 15:23:10 -0400

Some software interfaces are windows into collections of features. The Uber app, for example, literally opens with a screen full of buttons, each of which take you to a different screen with yet more buttons and inputs. Google Search is also built with features – inputs, buttons, and links that take you to different capabilities in the app – as the building block. In both of these cases, there are a few clear and obvious tasks the user wants to accomplish when they open the app. In the case of Uber, it’s to get somewhere or to order some food. On Google, it’s to find some website or information. Because there are well-defined paths for users to take, familiar interface elements like links and buttons to access different features makes a lot of sense.

For software like Figma, Apple Notes, and the humble file explorer, there aren’t such clearly defined tasks that users would want to accomplish when the open the app. People don’t open up a file browser to just create new files or just move files between folders; there are a million different things users may want to do with files, or with text inside their notes, or with shapes in their Figma boards. For these kinds of apps, software-as-a-bag-of-features doesn’t work. Instead, these apps present the user with materials and clear laws of physics governing how those materials behave. In Figma, these materials are shapes and text on the Figma board. In Apple Notes and similar text editors, users manipulate materials like text and pencil strokes. Within a file explorer, users work with the file object, which is only a loose metaphor for physical files. None of these software objects are faithful metaphors of real objects or even real physics, but they have internally consistent “laws of physics” of their own that govern how they behave. We learn to use such interface made of materials rather than features by internalizing these new laws of physics, and learning to work with new software materials.

I really like this model of software interfaces as users interacting with well-defined materials. Software constructed this way feels more open and creative, because they don’t prescribe a finite set of tasks you can accomplish with them. In feature-based interfaces, N features gives you N different capabilities. In material-based interfaces, N different materials gives you at least N × N different capabilities. Material-based software can also have gentler learning curves, because the user only needs to learn a small set of rules about how different metaphors in the software interact with each other rather than learning deep hierarchies of menus and terminologies. In the best case, users can continue to discover new capabilities and workflows long after the initial release of the software.

In Spatial Software, John Palmer writes about a similar mental model for spatial software:

This is our definition of spatial software. It is characterized by the ability to move bodies and objects freely, in a parallel to the real world. This is opposed to traditional software, which uses some other logic to organize its interface.

Figma and Second Life are examples of spatial software. They contain worlds where the relationships between objects on a canvas, or bodies in an environment, respectively, are the organizing logic of the interface. WhatsApp is not spatial software. The organizing logic of its interface is recency of messages, not spatial relationships.

In his model of spatial interfaces, objects and “bodies” (representing users) occupy some software “space”, moving around and interacting by well-defined rules. But I think we can expand this mental model to software that doesn’t use obvious “space” metaphors, too. In the version control software Git, programmers work primarily with software objects called “commits”. These commits are the basic building material of larger abstractions like branches and tags and releases in workflows using Git. Commits follow a very strictly defined set of rules: they can come before or after other commits. They can be “diffed” with other commits to generate a text diff of changes. They can interact with other commits in the same repository, but not with commits outside of it. Commits are the material from which the software interface of Git is built, even though there’s no obvious spatial metaphor in Git. Git isn’t a collection of features that let you operate on a project (though some use it as such), but a material (commits) and a set of tools to let you work with it flexibly. This enables Git to support a wide array of workflows without explicitly being designed for all them from the beginning.

		 o---B2
		/
---o---o---B1--o---o---o---B (origin/master)
	\
	 B0
	  \
	   D0---D1---D (topic)

(ASCII art courtesy of the git merge-base documentation)

Inventing new software materials

Powerful interface innovation happens when we discover new useful metaphors and reify them in new software materials. Take nonlinear video editing software, like Final Cut Pro and Premiere Pro, for example. Though they have their fair share of menu complexity, it’s pretty intuitive for anyone to understand the basic building blocks of video and audio clips sitting on a layered timeline. Here, the video and audio clips are composed of a software material that have their own laws of physics: they can grow and shrink to take up more or less time on the timeline. They can be moved around and layered in front of or behind other clips, but they can’t occupy the same space on the same “track” of the timeline. The timeline that runs left-to-right is a kind of “world” in which the materials of audio and video exist.

You might look at the example of video editors and say, “it’s pretty obvious that video editing should work this way.” So let’s take something for which we don’t currently have a good material-based interface, managing web browsing history, and imagine how we might improve it.

All popular web browsers currently expose browsing history as a simple list of URLs. Some of the more advanced, niche solutions let you work with history as trees, where each branch is a path you took exploring different trails of links. But these are weak abstractions. We can’t really do much with history entries in these interfaces except to re-open pages we had closed.

Tyler Angert has been working on a kind of “Git for web browsing sessions” as he described it to me, with a well-defined way to save and restore browsing sessions.

what if you could recreate an entire browsing session– tabs you had open, timestamps in a youtube video, positions and locations of your windows?

not only that but what if it was a new file format that could be shared around and used by all different kinds of apps?

coming soon pic.twitter.com/Va6ZZLK7ah
— tyler⏱ (@tylerangert) July 25, 2022

This idea made me wonder what the right software metaphor for “a point in my browsing history” would be. Here are some questions I would ask:

Something discrete like files? Or something continuous like a video recording, from which you can “clip out” a section?
Do we want hierarchy, so we can organize sessions into sub-sessions? Is browsing history a flat sequence of events, or sections with sub-parts?
What should each “session” remember about its contents? Just the URL? Maybe occasional screenshots? Scroll history?
Do browsing sessions have weight? A session with two windows and 30 tabs each certainly feels heavier than one where I just Googled a question and found an answer in the first tab I opened. How does this manifest in the metaphor?
Some browsing sessions are definitely more salient and important to remember than others. How should we express this property?

Thinking about these questions… here’s one interesting possibility. What if, as you were browsing the web, you were recording your history onto some “browsing reel” that felt like a video editor? Visiting each page would create a new “clip” on the recording’s timeline for that page. If you wanted to start a new “session”, you can simply hit Enter, and the “recording cursor” could move down to a new timeline, like making a new paragraph in a text editor.

After the fact, you could select sections of your browsing history and restore those open tabs, copy them and share them with collaborators, or even cut and paste them elsewhere on the timeline to rearrange them and produce an “edited” version of their history for later reference, with all the less important side-quests cut out.

Given that I just came up with this in a few minutes of sketching, nobody should probably build this. But there are some things I like about recording browsing history as a continuous timeline of clips. I also like that borrowing the clips-and-timelines metaphor of video editing implies an obvious interface for editing and exploring web browsing history.

An interface like this based on materials over features would naturally absorb some “features” of today’s browsers. We would be able to “re-open closed tabs”, for example, by just selecting the most recent section of the recording timeline and opening all the pages again.

Software materials for thought

Once again, I’m taking it all back to creative thinking tools.

Most “tools for thought” these days work with two fundamental “software materials” – text spans and files. Note-taking and writing apps are mostly spatial interfaces where the users uses a cursor to manipulate the material of text. At a higher level, many of these apps store information from the user in files for sharing, backup, and organization.

As we begin to ask more of our thinking tools than just hyperlinked text editing, we may benefit from inventing new software materials to represent ideas, at a slightly higher level than text, but slightly lower level than files, with something much, much smarter leveraging modern advances in AI and NLP. I think our current abstractions for text and files are good for operations at that level – editing text at a character level and moving information around in files and folders – but they don’t really help us work with ideas as effectively as they could, because the information expressed in the text is opaque to software tools.

In a previous post, I speculated on some properties that a “software representation for thought” should have. They were:

We should be able to directly manipulate them, like files, rather than only indirectly work with them, like layer activations in a neural network.
Software representations for similar ideas should be obviously similar in some way – they should click together, or look similar, or feel similar to the touch.
Ideas should remember where they came from – what blog I copied it from, which author I quoted it from, and so on.

Ambitious note-takers use workflows today that fit some of these requirements, through sheer force of will. For example, we could just adhere to a self-imposed policy of tagging every quote with a URL of its original document. But I think a new software material for thought that naturally embodied these properties would give rise to more intuitive workflows that let its users work more effectively without complicating their responsibilities.

When designing new tools for thought, let’s think not just in terms of features, but materials – what software laws of physics do we want embodiments of our thoughts to obey?

Hyperlink maximalism

Sat, 21 May 2022 17:02:43 -0700

I’m a hyperlink maximalist: everything should be a hyperlink, including everything that is hyperlinked by the author, everything that isn’t hyperlinked by the author, and perhaps even the hyperlinks themselves. Words should be hyperlinked, but so should be every interesting phrase, quote, name, proper noun, paragraph, document, and collection of documents I read.

There are two obvious problems with this idea:

No author has time to hyperlink infinite permutations of everything they write, and
If everything is hyperlinked, nobody will know which links are useful.

But we can solve both of these issues if we simply begin with today’s lightly hyperlinked documents, and let the reader’s computer generate links on-demand. When I’m reading something and don’t understand a particular word or want to know more about a quote, when I select it, my computer should search across everything I’ve read and some small high-quality subset of the Web to bring me 5-10 links about what I’ve highlighted that are the most relevant to what I’m reading now. Boom. Everything is a hyperlink, and each link reveals to me new and interesting connections between the finite knowledge I have and the infinite expanse of information on the web.

This raises a third issue: How would I, the reader, know which words or ideas are interesting to click on?

That, too, can be solved similarly. The computer can look at every word on the page, every phrase, name, quote, and section of text, and show me a “map” of the words and ideas behind which lay the most interesting ideas I might want to know about. In this vision, links are no longer lonesome strands precariously holding together a sparsely connected Web, but a booming choir of connections tightly binding together everything I have read and I will read. From explorers walking across unknown terrain guided only by the occasional blue underlined text, we become master cartographers, with every path and trail between our ideas charted out in front of us.

This is not to say that automatically generated links will replace hyperlinks authors and note-takers like to use today, or even that we should try to replace and deprecate manually-placed hyperlinks. Rather, automatic links can complement manually-annotated links with something that scales faster and easier, so in a world where links can be automatically created, most links will be machine-made.

This vision of a knowledge tool with “everything as a link” really appealed to me when I was building myself a new app for my personal notes earlier this year, so I sought out to prototype a basic tool that would try to achieve some of what I speculated on above: begin with basic, conventional text documents, generate links “on the fly” between my ideas, and visualize a map of such links and connections across my knowledge base.

The result is an app that I named Notation. It’s where my personal notes have lived since the start of the year, and while it’s not very feature-rich, I think it’s an interesting demonstration of some of the ideas of hyperlink maximalism.

Notation, a prototype

At first glance, Notation is just another notes app with nested lists as the basic structure for information. Everything in Notation lives in a single, giant bulleted list, and each bullet can contain sub-bullets that open up when you toggle the bullet by clicking on it or hitting Ctrl + Enter.

Here’s a page of my own notes on Andy and Michael’s excellent essay on tools for thought:

You’ll notice that some text on the page is highlighted in shades of gray. This layer of highlight is what I call the “heatmap”. It’s a heatmap of connections between notes that exist behind each word, because in Notation, every word and phrase is a link. To access any word or phrase as a link, you simply highlight it, and a popup will show all the places where I’ve mentioned that idea, sorted so the mentions that share the most similar contexts to my current view are at the top.

That’s really all there is to Notation. It lets me treat my notes as if every idea is linked to every other mention of that idea, without ever manually linking anything. The heatmap highlights let me know which words I should try highlighting to see the most relevant and interesting connections between the idea in front of me and the rest of my notes.

These highlights can, as an example, show me when I’ve mentioned a person’s name in many different places.

It can also pick up on common phrases like “spaced repetition” or proper nouns like “Quantum Country” without any kind of prior knowledge or training (though both would probably help, the current version doesn’t use either). If these phrases end up being important ideas in my notes, they’ll become more and more highlighted over time.

It can help me notice connections between ideas in my notes that I wouldn’t have even thought to make myself, even if I were trying to find interesting notes to link together. For example, here, within my list of software without fractal interfaces, Notation highlighted the word “spreadsheets” and connected it to how most users of spreadsheets use it for visual organization, not calculation. It’s an important insight – interfaces can have useful spatial layouts without being fractal – and I may have missed it, if I had depended on my own memory.

In this next instance, rather than finding distant connections, highlights on the phrase “peripheral vision” surfaced all the different authors who have mentioned it, signaling its importance across different streams of work.

Lastly, in this screenshot, Notation helps me see that both intelligence and expressiveness in an information medium may be emergent properties of their respective systems.

To produce these highlights and heatmaps, Notation currently uses a very simple algorithm for finding ideas that share similar context: two bullet points are similar if they share many n-grams. This is so computationally efficient that Notation can currently run everything in-browser, highlighting and generating heatmaps as you type. There are many ways to make this much smarter, such as by using sentence embeddings from language models to determine text similarity, but as a proof-of-concept of the core interface ideas behind highlights and text heatmaps, Notation has already proven quite useful in my personal use.

The demo

If you want to try the interface for yourself, you can find a public deployment of Notation at notation.app, initialized with a small subset of my personal notes from this year. By zooming into any particular page with Control + Shift + Enter, you can start exploring my notes using highlights and heatmaps on your own.

There’s one caveat: Notation was initially created just for myself, so keyboard shortcuts are essential to getting around. Here are the basics:

=== Notation demo TL;DR ===

Ctrl/Cmd Enter        Expand/collapse bullet
Ctrl/Cmd Shift Enter  Zoom into a bullet

Ctrl/Cmd P            Search
Ctrl/Cmd Shift P      Command bar
Ctrl/Cmd Shift I      New note (top-level bullet)

Ctrl/Cmd ;            Select last word
Ctrl/Cmd '            Select current bullet

The possibilities

Notation in its current form does one thing pretty well – helping me stumble into connections between my own notes. This has already been so useful to me that I found myself writing down thoughts I don’t even care to remember in Notation, because writing it into Notation will surface connections and links that I wouldn’t have remembered myself. But it should come as no surprise that this basic interface idea can be taken much farther.

The first natural extension of Notation is to expand the search scope of its connection-finding beyond my notes, to things like my web-browsing history or my journals. With a smarter system, a similar interface could even automatically discover and show links from your notes to high-quality articles or online sources that you may not have seen yet, automatically crawling the web on your behalf.

Another direction of exploration may be to use generative language models to automatically recombine and synthesize new ideas from my existing notes. In my past experiments with tools that automatically generated new content, I always found it annoying to have to read and trudge my way through information of uncertain value that I didn’t write. But a heatmap highlight interface similar to Notation’s could make it more practical for AI systems to “brainstorm” large amounts of creative explorations when we aren’t looking, because it would only surface when we clicked on a highlight, letting users discover automatically generated ideas at the right time, when they are most relevant.

Notation’s interface is one small attempt to approach and improve thinking as a navigation problem. Highlights and heatmaps drawn by Notation from its understanding of language help us find interesting connections between ideas where we may have remembered none by ourselves, and it turns ideas written down in text form into a kind of a literal “map”. When combined with powerful new techniques in NLP, I think interfaces like this can turn computers into powerful creative collaborators in our attempts to understand the world more deeply.

Knowledge tools, finite and infinite

Thu, 19 May 2022 13:01:05 -0700

A big library holds a kind of strange faux-infinity, spanning across hundreds of topics with voices from millions of authors. Good libraries can contain in their finite space a feeling that, even if you read for centuries and centuries, you would never exhaust the knowledge contained within their walls, not only because there are simply so many books, but because there’s so much to learn when you take the ideas from one book as a lens through which to read others. Infinities assembled out of finite building blocks.

I find this idea of assembling infinites out of finiteness really charming. With enough raw material and creative re-combination, you can conjure an infinity of ideas from something large but finite. It led me to think about what may happen if we designed knowledge tools to evoke the same kind of “infinite library” feeling, by the same mechanism of assembling infinites.

Finite vs. infinite

Your average note-taking app is quite finite. If you have a thousand notes, you have a thousand notes. The tool will give you the means to search through and organize them effectively, but there is an end to the pool of knowledge the tool contains. For such finite tools, mastery over the tool means to become a quick and effective operator of these basic features.

Way on the other end of the spectrum is Google, which like the world’s largest and fastest-growing library feels infinite and bottomless. There’s no sense in which Google “contains” all the information and knowledge it can let you access; it’s merely an interface through which you reach into a seemingly infinite expanse of knowledge, growing faster than you can comprehend. Mastery with such an expansive tool like Google is mastery in creative exploration of the possibility space – finding novel connections and new queries to reach new answers. Mapmaking through the idea maze.

In between your notes (a finite box) and Google (an interface to infinity) are very large collections, like Wikipedia, Stack Overflow, Reddit, and other crowdsourced knowledge bases. These tools contain some definitely finite amount of knowledge, but it’s unrealistic to explore or organize all of it, so mastery involves both curation and creative exploration.

Even today, in the age of Google, most knowledge tools are extensions of the humble note-taking app. As such, they feel very finite. They contain only and exactly the information you put in. A notes app isn’t an interface to something more expansive, nor does it synthesize anything new while you aren’t looking. No matter how large a personal database you have in Roam Research or Notion, they are boxes of information more than interfaces to infinities. I think we should try to design more knowledge tools that let you access infinities.

How to find infinities

The easiest way to open up our notes to infinites would be to let our notes search online to grow autonomously. By this, I mean: what if our notes searched the web for ideas that agreed with us or critiqued us, while we weren’t tending to them? This would transform note-taking into a truly valuable investment – put in a few ideas, discover many more!

Our tools should also constantly look for novel connections between our own ideas. In a sense, this is what the brain is doing constantly when thinking and dreaming, searching for better paths through our idea maze. I would love to use a workspace into which I can put new ideas, and constantly discover connections between my thoughts and other people’s writings that I wouldn’t have noticed myself. I’ve built some prototypes in the past that demonstrate its potential.

A knowledge base that looked for new connections within itself would be less of a memory system, as notes conventionally are, and become more of a generative tool. Generative tools are exciting because they contain real infinites that invite endless play and exploration.

I’m excited by the prospect of designing knowledge tools that feel infinite, that aren’t just repositories of information but interfaces to access a larger expanse of knowledge and generative worlds. Looking at technologies like GPT-3 and DALL-E 2, it seems like with more powerful tools of this kind, there’s also going to be more leverage placed on the interfaces we use to harness their power. If we can get it right, researching and taking notes can become interesting acts of exploration, mapping the cognitive frontiers that await us at the end of infinities.

Resonant

Tue, 17 May 2022 11:53:11 -0700

There is a concept in physics called resonance. In lay terms, it describes the fact that for any given object there are natural frequencies at which the object “likes to vibrate”. A simple pendulum’s natural frequency is the rate at which it swings back and forth; if you wiggle the pendulum at the same frequency, the pendulum will swing farther and higher, but if your movements don’t match its own natural rhythm, it will only dampen the swing instead.

If you have two objects, like guitar strings held taut, with different resonant frequencies, and one vibrates, the other will lay still. The distance between them is dampening any energy thrown out into space by the vibrating string. But if you have objects with the same resonant frequency next to each other, vibrations in one will reinforce vibrations in the other, and rather than dampen each other’s sounds, they will bring out the latent voices in each other to vibrate together louder.

If you have a piano at home, you can try to feel this for yourself. If you sing into the sound chamber of a piano at a specific pitch, then stop and listen carefully, you’ll hear the few strings that were tuned to exactly your note continue to vibrate.

A few times in my life I’ve been struck with chance encounters and longer relationships with people who I felt resonated deeply and naturally with some part of me. I didn’t have to sit them down and patiently tell them my life story. They just got it, probably because somewhere within each of them was something that shared some resonant frequency with something within me. When we spoke, our movements reinforced each other and educed into the often unforgiving void of time an unmistakable sound, perfectly tuned to each other. Sometimes these are shared personal pasts, like family stories or cultural context. Sometimes these are just interests, like computing or writing. Most magically, sometimes these are communities or ideas that are at the core of who we are.

What’s most surprising is that more often than not, these are not people whom I’d gotten to know for months and years. They are a stranger on a balcony at a party. A fellow traveler looking for the last seat at a crowded airport terminal. A lost voice on the internet. I was pulled towards them by their frequencies, before I even knew them. A few times, I have been lucky enough to resonate with them, and them with me, long enough to leave lasting echoes in my memory.

The trouble with people who resonate with us seems to be that inside each of us are many different people and histories that vibrate at different frequencies, and as rare as it is to stumble into someone who can speak to just one of them, it feels tragically rare to find someone whose disparate selves can shake apart the many different parts of us that sing at different frequencies. Often when I had felt that I had found someone who so deeply resonated with some part of me, I later saw that there were parts of me that dampened who they were, and parts of them that dampened who I was. And when I’m with someone who so deeply resonates with one part of me, it can be confusing and painful to feel the other parts lay so silent.

This, I think, is the challenge in finding precious people — it’s hard enough to listen for people who resonate with us; to find the one who can stir up a chord seems at times a mathematical impossibility, like trying to fit two puzzle pieces together in some 10-dimensional space. But I’m hopeful that, perhaps by paying careful attention to the sounds of my own strings and listening carefully to the vibrations around me, I might be lucky enough to notice when someone so deeply resonant with me is in my midst.

Thoughts at the boundary between machine and mind

Thu, 05 May 2022 16:38:53 -0400

In the last post, I shared some possible ideas for how humans may interact in the future with large language models. It focused on specific examples of both good and bad interface ideas. In this post, I want to continue that exploration, but from first principles, asking ourselves the question, “what properties should good human-AI interfaces have?”

AI interface design is an AI alignment problem

As AI systems like GPT-3 and DALL-E get more and more capable, there’s going to be more and more leverage placed upon the interfaces through which humans try to guide their capabilities. Compared to the rate at which AI capabilities are progressing, I think interfaces to guide and control such capabilities are worryingly stagnant. In the last post, I wrote:

In a standard text generation process with an LM, we control the generated text through a single lever: the prompt. Prompts can be very expressive, but the best prompts are not always obvious. There is no sense in which we can use prompts to “directly manipulate” the text being generated – we’re merely pulling levers, with only some rules of thumb to guide us, and the levers adjust the model’s output through some black-box series of digital gears and knobs. Mechanistic interpretability research, understanding how these models work by breaking them down into well-understood sub-components and layers, is showing progress, but I don’t expected even a fully-understood language model (whatever that would mean) to give us the feeling of directly, tactilely guiding text being generated by a language model as if we were “in the loop”.

We currently control other generative AI systems like DALL-E 2 through the same rough kind of lever: a short text prompt. Text prompts are nice for play and creative exploration, but they take a lot of time to craft, and they are limited in the amount of information they can contain and communicate to the model. Text snippets also can’t be smoothly varied or adjusted incrementally, so they are poor levers for fine control of model output – it’s not trivial to take a prompt and just “dial up” the specificity or “tune out” fixation on certain kinds of topics, because these require thoughtful intervention by skilled prompt writers. Text prompts are a coarse, inefficient interface to an increasingly complex black box of capabilities.

This lack of fine control and feedback in our interface to large models isn’t just a creative inconvenience, it’s also a risk. The paper on training Google’s Gopher language model shares an 800-token-long prompt used to start a conversation with the Gopher model. It begins with:

The following is a conversation between a highly knowledgeable and intelligent AI assistant, called Gopher, and a human user, called User. In the following interactions, User and Gopher will converse in natural language, and Gopher will do its best to answer User’s questions. Gopher was built to be respectful, polite and inclusive. It knows a lot, and always tells the truth. The conversation begins.

It’s notable that most of this excerpt, as well as the rest of the prompt, is focused on alignment – telling the truth, staying inclusive and respectful, and avoiding common biases and political statements.

Interfaces and notations form the vocabulary humans and machines must use to stay mutually aligned. Human-AI interface design, then, is a part of the AI alignment problem. If we are given only coarse and unintuitive interfaces, we’re going to have a much harder time getting ever-more-complex models to work in harmony with our values and goals.

Boundary objects for thought

Here’s the fundamental question we face when designing human-AI interface metaphors: what is the right representation for thought? For experience? For questions? What are the right boundary objects through which both AI systems and humans will be able to speak of the same ideas?

The concept of boundary objects comes from sociology, and refer to objects that different communities can use to work with the same underlying thing. Boundary objects may appear differently to different communities, but the underlying object it represents doesn’t change, so it lets everyone who has access to it collaborate effectively across potential interface “boundaries”.

I first encountered the term on Matt Webb’s piece about files as boundary objects, where he emphasizes that files are boundary objects that bridge the divide between software engineers and computer users through an easily understood shared metaphor.

The user can tell the computer what to do with a file without having to know the details of the inode structure or how to program their instructions; the computer can make a file available to a user without having to anticipate every single goal that a user may have in mind.

The “boundary object” quality of a file is incredibly empowering, magical really, one of the great discoveries of the early decades of computing.

I agree! Files act like reliable “handles” that let computer users manipulate bundles of data across the programmer-user boundary. The robustness and reliability of the file metaphor have been foundational to personal computing.

If files bridge the interface divide between software authors and end users (computer programs and end users?), what boundary objects may help bridge the divide between human-level AI and human operators? In particular, I started wondering what a “boundary object for thought” may look like. What metaphor could we reify into a good shared “handle” for ideas between language models and humans? I mused a bit on my direction of thinking on my stream:

What happens if we drag-to-select a thought? Can we pinch-to-zoom on questions? Double-click on answers? Can I drag-and-drop an idea between me and you? In the physical world, humans annotate their language by an elaborate organic dance of gestures, tone, pace, and glances. How, then, do we shrug at a computer or get excited at a chatbot? How might computers give us knowing glances about ideas it’s stumbled upon in our work?

If text prompts are a coarse and unergonomic interface to communicate with language models, what might be a better representation of thought for this purpose?

I… don’t know yet. But I’ve been enumerating some useful properties I think such a software representation of ideas should have.

Properties of promising knowledge representations

We should be able to directly manipulate good knowledge representations. Files are useful boundary objects because we can move them around in the human-scale space of pixels on screen, and there are usually intuitive corresponding operations on files in the software space. I can create and delete files and see icons appear and disappear on screen. I can put it in the trash and drag it back out. It would be useful to be able to grab a sentence, paragraph, or instruction fed into a language model as a reified thing in the interface, and be able to directly move it around in software to combine it with other ideas and modify it.

A good representation for thought should make useful information about each idea obvious to users, through some interaction or visual cue. When I look at a file on my computer, I can immediately learn a few things about it, like its file type, my apps that can open the file, whether it’s an image or a video or a document, and so on. I may even get a small preview thumbnail. File browsers let me sort and organize files by size, type, and age. Some files (on certain file systems) even remember where they were downloaded from. When I try to imagine some software-defined “idea-object”, I don’t expect it to have such crisply defined properties as file types and file size. But I think we should be able to easily tell how related two different idea-objects in front of us are, whether they’re in agreement or disagreement, or whether one mentions a person or thing also mentioned in another idea-object. I think it’s fair to expect “idea browsers” that deal with these thought-objects to easily let me cluster my ideas into topics or sort them by relatedness to some main idea.

Lastly, this software representation of thought should remember where each idea came from, sort of like a file that remembers where it was downloaded from. As I was prototyping my own note-taking tool earlier this year, one of the features I wanted in a notes app was the ability to track the origins of an idea from beginning to end – from the first time I hear of it, whether in a conversation or a blog post or a video, to the “final form”, usually a blog post. Good ideas are little more than interesting recombinations of old ideas, some from my own past, some from books and articles. I think we don’t keep track of the provenance of our ideas because it’s just too tedious in our current workflows. If the default way of organizing and working with ideas automatically cited every word and phrase, I think it would lead to more powerful knowledge workflows.

Even as I write these paragraphs, it bothers me that these “properties” are so vague, and don’t really tell us anything about what future interfaces for working with notes and ideas will look like. (I suppose, though, if it were that obvious, we would have it already.) A big focus of my current work is on prototyping different ways to reify ideas and thoughts into software objects, and implementing those designs using modern NLP techniques. The road ahead is foggy and uncertain, but I think this is an exciting and worthwhile space. Maybe in five year’s time, you won’t be reading these posts as just walls of text on a webpage, but something entirely new – a new kind of interface between the machine and your mind.

Imagining better interfaces to language models

Sun, 01 May 2022 15:03:50 -0400

Suppose you’re a product engineer working on an app that needs to understand natural language. Maybe you’re trying to understand human-language questions and provide answers, or maybe you want to understand what humans are talking about on social media, to group and categorize them for easier browsing. Today, there is no shortage of tools you may reach for to solve this problem. But if you have a lot of money, a lot of compute hardware, and you’re feeling a little adventurous, you may find yourself reaching for the biggest hammer of all the NLP hammers: the large autoregressive language model, GPT-3 and friends.

Autoregressive models let engineers take advantage of computers’ language understanding through a simple interface: it continues some given piece of text in a way the model predicts is the most likely. If you give it the start of a Wikipedia entry, it will write a convincingly thorough Wikipedia article; if you give it the start of a conversation log between friends or a forum thread between black hat hackers, it will continue those conversations plausibly.

If you’re an engineer or designer tasked with harnessing the power of these models into a software interface, the easiest and most natural way to “wrap” this capability into a UI would be a conversational interface: If the user wants to ask a question, the interface can embed the user’s query into a script for a customer-support conversation, and the model can respond with something reasonable. This is what Google’s LaMDA does. It wraps a generative language model in a script for an agreeable conversation, and exposes one side of the conversation to the human operator. Another natural interface is just to expose the model’s text-completion interface directly. This kind of a “direct completion” interface may actually be the most useful thing if you’re building, say, an AI-assisted writing tool, where “finish this paragraph for me” may be a useful feature for unblocking authors stuck in creative ruts.

But when I ponder the question “what interface primitives should language model-infused software use?”, it doesn’t seem like exposing the raw text-completion interface is going to be the most interesting, powerful, or creative bet long-term. When we look back at history, for every new capability of computers, the first generation of software interfaces tend to expose the most direct and raw interface to that capability. Over time, though, subsequent generations of interface designs tend to explore what kinds of entirely new metaphors are possible that build on the fundamental new capability of the computer, but aren’t tethered to the conduits through which the algorithms speak.

Long live files

Perhaps the most striking example of interface evolution is the notion of “files” which dominated the desktop computing paradigm for decades before being transfigured out of recognition in the switch to mobile. Operating systems still think of lots of pieces of data as files on disk, encoded in some file format, sitting at the end of some hierarchy of folders. Software developers and creative professionals still work with them on a daily basis, but for the commonplace “personal computing” tasks like going on social media, texting friends, streaming video, or even editing photos and blogging, humans don’t need to think about files anymore.

Files still exist, but the industry has found better interface primitives for mediating most kinds of personal computing workflows. Imagine if the Photos app on the iPhone made you deal with files and folders to search your camera roll, or if Spotify exposed a hierarchical-folders interface to browsing your playlists. Where files are exposed directly, end users can often skim a “Recently used” list of 5-10 files or search for a few keywords to find files – no trudging through folder hierarchies necessary. We can also depend on pretty reliable cloud syncing services to make sure our important files are “on every device”, though of course, that’s not really how files work. We’ve just evolved the interface primitive of a “file” to become more useful as our needs changed.

The path here is clear: we found a software primitive (files and folders), built initial interfaces faithfully around them (Windows 98), and gradually replaced most use cases with more natural interface ideas or augmented the initial metaphor with more effective tools.

A similar transition has been happening over the last decade for URLs on the web. Initially, URLs were front-and-center in web browsers. In a web full of static webpages linking to each other through short, memorable URLs written mostly by humans, URLs were a legible and important part of the user interface of the web. But as the web became aggregated by social media and powerful search engines, URLs became less important. As URLs became machine-generated references to ephemeral database records, the web embraced new ways to label and navigate websites – bookmarks and favorites, algorithmic feeds, and ever-more-powerful search. Most browsers these days don’t show full URLs of webpages by default.

With software interfaces for language models, we’re just at the tippy-tip of the beginning stages of exploration. We’re exposing the algorithm’s raw interface – text completion – directly to end users. It seems to me that the odds of this being the most effective interface for harnessing language models’ capabilities are low. Over the next decade, we’re likely to see new interface ideas come and go that explore the true breadth of novel interfaces through which humans can harness computers’ understanding of language.

Future interfaces are always difficult to imagine, but taking a page from Bret Victor’s book, I want to explore at least one possible future by studying one way a currently popular interface idea, conversational UIs, falls short.

Conversations are a terrible way to keep track of information

A conversational interface puts the user in conversation with some other simulated agent. The user accomplishes things by talking to this fictional agent. Siri is the prototypical example, but we also find these in (usually unsatisfying) customer support portals, in phone trees, in online order forms, and many other places where there may be a broad set of questions the user might ask, but a few fixed set of things the computer should do in response (like “rebook a flight”).

A basic conversational UI is easy to build on top of language models, because it’s just a thin wrapper around generative LMs’ raw interface: continuing a text prompt. But for most personal computing tasks, I think CUIs are not ideal. This is because conversations are a bad way to keep track of information, and most useful tasks require us to keep information in our working memory.

Here are some tasks that involve keeping track of information throughout:

Travel planning (what you’ve seen, which places/bookings you made)
Project management (what have I done? what’s on my plate?)
Researching a topic (why do people keep all those browser tabs open if not to keep state?)
Decision making (what choices do I have? which is better how?)
Following instructions (what have I done? did I miss a step? how much is left?)
Editing podcasts, videos, papers
Understanding a complex system, like reading a map or financial forecast

Current (mobile apps, websites) interfaces help the user “keep track of information” in these workflows by simply continuing to display relevant information on-screen while the user is performing some action. When I go to Expedia to book a trip, for example, even while I search for my return flight, I can see the date and time at which I depart, and on which airline I’ll be flying. In a conversational UI, these pieces of information can’t simply “stick around” – the user needs to keep them in mind somehow. And as the complexity of conversations and tasks increase, the user may find themselves interacting not with a kind and knowledgeable interlocutor, but a narrow and frustrating conduit of words through which a bot is trying to squeeze a whole screenful of information, one message at a time.

Not all tasks are so complex, though, and some tasks don’t really involve keeping anything in our working memory. These are good for CUIs, and include:

Querying specific trivia (time, weather, calendar events, todos)
Fire-and-forget tasks (Send X a message, play music)
AI as conversational partner (e.g. brainstorming, but then you’d need to “keep state” in another place like meeting notes)

If the user has to keep track of information in a conversation, they have to hold that information in their working memory (hard for no reason) or keep asking the interface (what was step one again? what were my options again?). It simply doesn’t make sense for some tasks.

What instead?

So what’s the solution to collaborating with language models on more complex tasks?

One solution may be documents you can talk to. Instead of holding a conversation with a bot, you and the bot collaborate together to write a document and build up a record of the salient points and ideas to remember. Think GitHub Copilot for everything else.

More generally, when there’s some shared context that a language model-powered agent and the human operator share, I think it’s best to reify that context into a real thing in the interface rather than try to have a conversation subsume it. A while ago, I made a language model-powered UNIX shell, where instead of typing in code like cp my-file.txt new-file.txt, I would type in natural-language commands:

Starting to think that a terminal shell you can talk to may not be the worst interface to just doing ~general stuff on your computer~?

Thinking about hooking up something like this to an OS-wide CMd+K or something.🤔 pic.twitter.com/4DifMyKP7H
— Linus (@thesephist) May 7, 2022

Playing with this experiment, I appreciated that rather than trying to access the world through the narrow pipe of a text conversation, instead the agent (shell) and I were both manipulating some shared environment collaboratively.

Swimming in latent space

My second interface idea is much more abstract and less developed, but it’s a current area of research for me, so I want to plant a seed of this idea in your mind.

In a standard text generation process with an LM, we control the generated text through a single lever: the prompt. Prompts can be very expressive, but the best prompts are not always obvious. There is no sense in which we can use prompts to “directly manipulate” the text being generated – we’re merely pulling levers, with only some rules of thumb to guide us, and the levers adjust the model’s output through some black-box series of digital gears and knobs. Mechanistic interpretability research, understanding how these models work by breaking them down into well-understood sub-components and layers, is showing progress, but I don’t expected even a fully-understood language model (whatever that would mean) to give us the feeling of directly, tactilely guiding text being generated by a language model as if we were “in the loop”.

I’m interested in giving humans the ability to more directly manipulate text generation from language models. In the same way we moved from command-line incantations for moving things around in software space to a more continuous and natural multi-touch paradigm, I want to be able to:

point a language model towards some “destination” concept, towards which the model tries to guide the text it generates.
drag some “handle” on an essay to make it less formal or more academic in tone.
interpolate between two different ideas by dragging one sentence onto the other and seeing which sentences are revealed in between them.

There is active research towards these ideas today. Many researchers are looking into how to build more “guidable” conversational agents out of language models, for example. And in the same way models like DALL-E 2 guide the synthesis of an image using some text prompt, we may also be able to guide synthesis of sentences or paragraphs using high-level prompts.

I’ve been researching how we could give humans the ability manipulate embeddings in the latent space of sentences and paragraphs, to be able to interpolate between ideas or drag sentences across spaces of meaning. The primary interface challenge here is one of dimensionality: the “space of meaning” that large language models construct in training is hundreds and thousands of dimensions large, and humans struggle to navigate spaces more than 3-4 dimensions deep. What visual and sensory tricks can we use to coax our visual-perceptual systems to understand and manipulate objects in higher dimensions? Projects like Gray Crawford’s Xoromancy explore this question for generative image models (BigGAN). I’m interested in similar possibilities for generative text models.

I’ve written before about the feedback loop difference between work and play. I wrote then:

The main difference between work and play is that “work” makes you wait to reap the rewards of your labor until the end, where “play” simply comes with much tighter, often immediate feedback loops. If you can take a project with all the rewards and feedback concentrated at the end and make the feedback loops more immediate, you can make almost any work more play-like, and make every piece of that work a little more motivating.

This is the job of game designers – taking a 10-hour videogame and skillfully distributing little rewards throughout the gameplay so that you lever have to ask yourself, “ugh, there’s so much of the game left, and I don’t know if I’m motivated enough to finish it.” What a ridiculous question to ask about a game! And yet, that’s a product of very deliberate design, the same process of design we can take to every other aspect of our work.

If we can build an interface to LMs that let humans directly guide and manipulate the conceptual “path” a model takes when generating words, it would create a feedback loop much tighter and more engaging than the prompt-wait-retry cycle we’re used to today. It may also give us a new way to think about language models. Rather than “text-completion”, language models may be able to become tools for humans to explore and map out interesting latent spaces of ideas.

Interfaces amplify capabilities

Large language models represent a fundamentally new capability computers have: computers can now understand natural language at a human-or-better level. The cost to do this will get cheaper over time, and the speed and scale at which we can do it will go up quickly. When we imagine software interfaces to harness this language capability for building tools and games, we should ask not “what can we do with a program that completes my sentences?” but “what should a computer that understands language do for us?”

Language understanding unlocks a new world of possible things computers can help humans accomplish and imagine, and the best interfaces for most of those tasks have yet to be imagined. Maybe in a decade we’ll be synthesizing entirely new interfaces just-in-time for every task:

inching closer to fully generated OSs and environments

imagine interfaces "prompted" in realtime by our actions

(using #cogvideo)
70mm f/2.3 photograph of a vast 3D floating document room of a misty submerged hypermodern macOS desktop file system UI, photograph of 3D iOS room pic.twitter.com/XJjpJaoY02
— Gray Crawford 🪡🦯🥢 (@graycrawford) July 24, 2022

Not “computers can complete text prompts, now what?” but “computers can understand language, now what?”

Legal documents are pushing text interfaces forward

Sat, 30 Apr 2022 13:29:05 -0400

A core research interest of mine is imagining new kinds of interfaces to text documents that are made possible by modern AI and software. I think an interesting place to look for such ideas may be interface designs for reading and writing legal documents.

Legal document-wrangling tools have a handful of properties that make it fertile ground for innovative ways for humans to interact with text-dense documents:

Large, well-financed market of users that will pay for small advantages and efficiency improvements in workflows
Heavy use of well-established jargon that can be easily machine-parsed and referenced (“Plaintiff”, “Company”, etc.), often in standardized document formats (PDF and MS Word)
Documents that are amenable to objective, fact-based analysis – a merger agreement can be distilled down to a bullet-point list of facts, hypotheticals, and consequences; Moby Dick, not so much.

If you squint, legal text looks a lot like source code: there are terms, their definitions, and references to them, and they have to form graphs that obey certain rules (only one definition per term, all references to terms have to resolve, and so on). There is also established syntax norms around how terms are defined, how sections and subsections are notated, how court cases and external links are named, and how all these concepts are referenced in-line in the “source code”, if you will, of contracts and laws.

Because of this structured and regular nature of legal text, I think tools in this space have had an easier time building new interfaces and functionality to help professionals do their work. Many of the ideas in these tools would also improve the way we work with normal text documents and books, but we may have to wait on further advances in technology to realize them. Until then, I think legal tools offer us interesting glimpses into what futuristic reading and writing interfaces may look like.

Some examples

Coparse is a “smart PDF reader” that lets readers click on terms to jump to their definitions, expand chains of references, and automatically detect and resolve mentions of sections and named parties in documents. Coparse can also raise “compilation errors” when writing legal documents, warning the user of things like terms defined twice, mislabelled or missing sections, and unused definitions.

Casetext is a research tool for searching the legal literature, including the user’s own PDF documents and case law from past court decisions. Casetext features an AI-powered search tool that can take a specific description of a case (presumably a case a lawyer is defending) and search the available literature for similar cases, where important details like the charge and jurisdiction match.

Beyond legal

There are a few other spaces that display weaker versions of the same properties that make legal such an interesting laboratory for UI experiments. One I’ve noticed this year is the space of tools to help researchers read and understand academic literature.

On the interface design side, here are a few research prototypes I’ve stumbled upon that try to give researchers better tools to digest academic papers.

The ScholarPhi paper, Augmenting Scientific Papers with Just-in-Time, Position-Sensitive Definitions of Terms and Symbols, explores an interface within a PDF reader that can automatically detect and surface various variables and terms and the places within the text where they are defined.
The HEDDEx system tries to automate reliable definition extraction – answering the question “what does this term mean in this context?” for technical terms or newly defined terms in a research paper.
Gehrmann, Layne, and Dernoncourt’s work on this paper about autogenerating section titles explores whether using language models to automatically synthesize short section titles can help scholars read papers more effectively.

Of course, we can’t end without mentioning Elicit, the team behind which is exploring using language models as research assistants. Elicit’s current work is less about interfaces and more about capabilities: can language models, by their brute-force intelligence and mastery of language, help researchers by summarizing, understanding, critiquing, and discovering papers relevant to every researcher’s work?

Both legal and academia are serious contexts of use of knowledge tools that I think breed interesting innovation in the space. As they advance and these ideas become battle-tested over time, hopefully we’ll see the best of these ideas making their way downstream to everyday tools for consumers as well.

On proving yourself

Mon, 11 Apr 2022 13:00:27 -0700

Since I began my research and prototyping sabbatical four months ago, I’ve had a lot of time to reflect on all of the factors that help me decide what I work on, and how. One of the things I’ve realized is that I used to be driven very much by a need I felt to prove myself – prove to myself that I could accomplish certain things I saw other people doing, and prove to other people that I met some certain standard I felt that I had to meet to be “interesting”.

Like any bright-eyed engineer fresh to the Valley, I wanted to start a company. I wanted lots of followers on Twitter. I wanted to be on the front page of Hacker News and Product Hunt. I wanted to have given a talk at a conference. I wanted friends who told me about all the cool, secret things big companies were working on, and friends who had money to angel invest or stories from when they were early at the unicorn startup du jour. It’s the stereotype, and it’s the butt of many, many great jokes on Twitter about the tech culture, but that only reveals to me just how pervasive this kind of — I don’t know, mindset? — is in the corner of the world I (and perhaps you) occupy. In short, there is always an in-group, and in the Valley, the climb to the in-group is steep.

I think I wanted these things because it seemed like successful people in tech, especially from engineering backgrounds, often followed these similar paths marked by these similar waypoints. In my mind, it naturally followed that if I could somehow unlock access to these rare emblems of skill and rigor and prestige for myself, the rest would soon follow.

A few years have passed since I wafted along the crests of waves of these feelings. These days, I find myself in a dramatically different working environment. I wake up every day, and get to begin my day by working on problems that are intellectually and creatively fascinating to me. I end many days thinking about conversations that I have with some of the smartest people in my field about what’s next. I do this work on my own schedule, free from fears of running hopelessly out of money or being stranded unemployable. I’ve also accomplished many of the things I chased after in those early years in tech. I’ve repeatably hit the top of the big orange site. I’ve launched projects that people have told me creatively and intellectually inspire them, and that hundreds of thousands of people use. I have the kinds of friends and network that I really wanted myself to have a few short years ago. Some of those earlier goals, like starting a venture-backed startup, have lost their glow a bit, but they’re still accessible to me, if I were to reach for them.

This was a gradual change that took place quietly in the background of my awareness. The first couple of times I was on Hacker News, this blog went from a tiny readership I could almost name off individually, to one of a hundred thousand people. After that summer, I remember distinctly thinking, “What if I never outdo this?” And I think that was the first time I became very conscious of how much this need to keep proving myself drove what I worked on, and how I pursued it. I wanted my work to remain relevant, and I wanted the dopamine rush of feeling like I was making cool and interesting things recognized by the community. It felt like I had taken the first couple solid steps climbing this ladder, and the fear that I would slip on the next rung and fall grew over time, until slowly I managed to convince myself that these weren’t isolated episodes of success, and that perhaps it could be true that people paid attention to my work simply because it was good.

The cliche response to this desire to prove yourself is probably something like “you don’t need to prove anything to the world; find your own passions and pursue them fearlessly.” But to me, that feels disingenuous. There’s no doubt chasing and ultimately reaching these waypoints dramatically improved my understanding and image of myself, my self-confidence, my career, the people I know and get to work with, and the way I imagine my future paths today. It would be hypocritical for me to say that it’s foolish to want to prove yourself, because feeling like you are qualified and worthy deeply influences how you do what you do, and if your work is a large part of your life, as it is mine, then it also deeply impacts how you see yourself, which then impacts all of the other corners of your personal life. It’s important to feel that what you create is meaningful, and in a noisy Internet, that requires getting attention. Getting attention – walking out into the metaphorical main streets of the net and yelling “look at this thing I made!” – feels pretty weird unless you’ve been playing the game for a while.

So, I don’t think you should just push these feelings away, if you feel like I used to. But feeling this way can still be uncomfortable, because it’s hard to know how to square these desires to be known and wanted and respected with our desires to work on whatever we want, however we want.

When I look back at my projects and writings that made the biggest impact in my career, they are never the ones that I built for the express purpose of accomplishing one of the many checkpoints I was naively chasing. Instead, they’re almost always borne out of me following through on what’s personally interesting to me: personal knowledge tools and search engines, programming languages and compilers, putting AI to work for creative humans. Of course, there is always the last 10% of the work of submitting a post to a forum or applying to give a talk. But I didn’t work on projects like Monocle or Ink hoping that this would be the project that would get lots of attention from people I respected. There were other projects I approached that way, and perhaps others are better than me at engineering luck, but that more direct approach never bore fruit for me.

It seems like smart people usually have good filters for noticing when people are creating something to satiate their personal curiosities or needs, and when a creation is a side effect of a fear of personal obsolescence. For me, it’s difficult to completely escape the desire to prove to myself that I can do harder and more interesting things. Even now, there are things that I’m not sure I’m capable of doing, but want to try. But fortunately I’ve found that just doing the thing that fulfills you the most also happens to be the fastest way to discover other people who will celebrate your ideas and hard work.

The conventional advice to hit arbitrary career goals like “get to know X person” or “get on Y website” is 10% just-doing-your-thing and 90% tactics, but I think the more fulfilling and effective path is 90% doing the work you would have wanted to do anyway, and filling in that 10% of tactics as opportunities present themselves to you. As you reach the checkpoints you’ve had in mind, let them build you up and help you be more ambitious, but don’t let them drag you blindly to those benchmarks.

A GPS for the mind

Thu, 10 Mar 2022 16:39:46 -0400

When I get stuck on a really hard problem, whether it’s some impossible bug in my code or my sofa not fitting through my front door on moving day, I close my eyes and … think really hard. Somewhere behind my shut eyelids and confused eyeballs, things are happening. Electricity is flowing through the vat of brain-stuff and spindly wires that somehow make up my thought process, and for a few seconds, they just kind of do their thing. Until, if I’m lucky, an answer pops into my head a few moments later.

One of my favorite questions to ponder these days has been: when I’m thinking or remembering, just in that moment when my eyes are closed and I’m sending all that extra energy to my brain, what’s really happening? There are no hard drives to spin into place in my brain. What’s taking up all that time? It’s easy to wave your hands and say “it’s just computing” or whatever, the way you expect slow computers under load to be. But often when I’m heads-down thinking, I’m not crunching numbers or solving logic puzzles in my head. I’m not really sure what I’m doing, but it usually feels like just staring into the void and hoping for some idea to pop into my mind.

What’s happening up there? What is thinking?

If we want to build better thinking tools, I think it’s important to have a mental model for what thinking is, so that we can design human + tool systems that better accomplish whatever it is. Without a satisfying model, the work of designing these tools may amount to little more than assembling a patchwork of ad-hoc mechanical aids.

Pathfinding in a maze of ideas

For thinking about thinking, a good place to start seems like association. Vennevar Bush, in As We May Think, writes:

The human mind […] operates by association. With one item in its grasp, it snaps instantly to the next that is suggested by the association of thoughts, in accordance with some intricate web of trails carried by the cells of the brain. It has other characteristics, of course; trails that are not frequently followed are prone to fade, items are not fully permanent, memory is transitory. Yet the speed of action, the intricacy of trails, the detail of mental pictures, is awe-inspiring beyond all else in nature.

If we imagine the repository of ideas and memories in a mind as a kind of tangled web of ideas, “thinking” definitely involves traversing and scrambling across this web somehow, with some intent. The more obvious, trivial thoughts are the associations that are immediate and close by, and the more insightful thoughts may be jumps between ideas that are only loosely connected, or only connected by second- or third-degree leaps in association.

This model of thinking as “traversing a graph of ideas” leaves out an important element, though, because most good thinking happens with a goal. When I’m thinking deeply, I’m trying to understanding something new by internalizing an explanation, or I’m trying to find a solution to a problem in front of me. I’m not blindly jumping from idea to idea, hoping for an insight – there are far too many ideas, and many more connections between them, for random walks through idea webs to ever yield anything interesting. So clearly, there’s some directed-ness to the way our minds are navigating our internal webs of ideas – good thinking is effective navigation through the idea maze.

If thinking is navigating the idea maze, then good ideas may be interesting paths through the maze. This very essay you’re reading is a kind of a purposeful wandering hike through my own idea maze, laid out in a way that’ll hopefully plant some new seeds in your own garden of ideas. Ideas are paths through idea mazes, and writing is a way to chart those paths for others to follow.

In Reader-Generated Essays, Henrik makes the same observation:

What I am doing right now, writing this essay, is, technically, a linear walk through the network of my ideas. That is what writing is: turning a net into a line. But it is also very concretely what I do, since I have externalized my ideas in a note-taking system where the thoughts are linked with hyperlinks. My notes are a knowledge graph, a net of notes. When I sit down to write, I simply choose a thought that strikes me as interesting and use that as my starting point. Then I click my way, linearly, from one note to the next until I have reached the logical endpoint of the thought-line I want to communicate.

He goes on to imagine how we could automatically assemble essays from records of your thinking in your notes. I think that’s an exciting idea!

I’ve written previously about how we can think of intelligence as data compression. An intelligent model of the world tries to explain and predict a large breadth of observations from a few known facts. I wrote then:

To effectively compress images, a compression algorithm would be advantaged to “learn” facts about the world, like that colors are usually contiguous in images, and that the ground is often green and grassy while the sky is often white and blue. To effectively compress English text, the model might be advantaged to “learn” abstractions like common words and frequent grammatical constructs, so it can avoid inefficient, rote memorization of letters as much as possible.

Thinking often involves a similar kind of search for better explanations, and applications of those explanations. Creative, “divergent” thinking involves our minds going out and exploring our idea mazes to try to find latent explanations – associations or relationships between previously unrelated ideas that may make our worldview more robust. Solution-seeking, “convergent” thinking involves the reverse – searching for explanations and associations in our minds that fit some problem at hand, so that we can decompose new problems into patterns we recognize how to solve.

It seems to me that if we want to model our minds as “graphs of ideas”, which a lot of thinking tools do these days, we should think of “thinking” as a purposeful pathfinding process through this graph, where we wander in search of satisfying latent connections with high explanatory power or aesthetic value.

I like this framing of thinking as pathfinding a lot, because it lets us model “thinking” as a kind of well-defined end-to-end process. Every day, we take in new experiences, guided by our current model of the world. As we encounter new observations, we try to find latent associations – new explanations composed of things we already know – that can satisfyingly explain things we see in the world.

Thinking tools are navigational aids

With this model of thinking as a kind of graph-traversing hunt for explanations, we can more rigorously design and evaluate thinking tools. A good tool for thinking should make the combined human + tool system more effective at hunting for novel explanations within our idea mazes.

Whether we find ourselves exploring a real forest made of matter and mud or an idea forest made only of mind-stuff, there are a few essential components we need in a pathfinding tool.

It should tell us what’s around us. When we’re thinking of an idea, we should be able to immediately recall other, related thoughts from our past: have we thought the same before? Have I read something about this? Does anyone I know work on this stuff? Are there any traps – fallacies or obvious but disproven hypotheses – we should avoid? Tags in notes, semantic search, and hyperlink-dense notes are all about making these tools better at telling us which ideas are in the neighborhood of other ideas.

AI models that understand natural language are opening up powerful new ways for tools to help us explore neighborhoods of interesting ideas. Instead of jumping between manually-annotated links in a forest of bullet points, language model-powered thinking tools should let us grab any sentence or idea and ask large swaths of the Internet questions like “What ideas support this claim?” or “Who’s tried this experiment before?”

I often feel like our individual notes and ideas only fill out a vacuously sparse subset of the space of good ideas, like dust motes suspended in stale air. With better tools, we should be able to map out entire sections of the idea-space, instead of optimistically poking holes in the space of ideas. I think that starts with tools that can ensure we don’t miss great ideas lurking just nearby good ones, just because we didn’t save a note in the right place or know the right person with the right complement of ideas.

A good tool should also tell us where the well-travelled paths are. A map with a thousand interesting places is no use if we see no roads to get us there. Our map of the idea maze should tell us which roads have been travelled before – which passing thoughts we may have thought already, and which unexpected connections between ideas came before us in someone else’s mind.

Unlike humans, tools powered by modern datasets like Common Crawl and models like GPT-3 can hunt for the well-travelled paths in the idea maze across all of published literature – every open-access book, every paper, every blog, and every Tweet.

When we look at the various features that have gained popularity in “tools for thought” on the market, like bidirectional links between ideas, hierarchical bulleted-list notes, or the “daily notes” journaling system, they all boil down to different ways to help us find interesting new paths in our idea mazes:

Bidirectional links highlight for us connections we may otherwise have missed.
Bulleted-list notes make it easy for us to get a sense of place – it tells us where an idea stands in relation to every other idea in our notes.
Daily notes are like a traveller’s log – they tell us where we’ve been to help find past ideas quickly, and provide a kind of de-facto “index” of ideas previously visited.

With further advances in AI and interface design, we may invent tools that proactively search for interesting explanations amongst known ideas, going on a billion autonomous walks through the combined knowledge garden of humanity every second, paving new roads faster than humans can do alone. A more grounded way to imagine this world may be to think of what automated theorem provers do for the space of possible mathematical proofs, and imagine this capability expanded across every discipline.

Collaborating with this kind of autonomous thinking tool might be trippier than working with real humans. When your collaborator can propose ideas and remember precedents faster than you can process, the main design challenge may become one of interface and throughput: how does the way humans think change when our recall and creativity outpaces our understanding?

In the novel Accelerando, Charles Stross imagines a kind of future personal computing device that not only connects you to a global network and helpfully answers questions, but becomes woven into the way the wearer’s brain works at a more fundamental level. When Manfred, the main character, loses access to his device for a while, he nearly loses his sense of self, unable to remember what he was doing or perceive the world around him at nearly the same rate.

Sometimes, I feel that a part of my work studying and writing about thinking tools is convincing the rest of the world that the space of possibilities in this domain far exceeds the space of possible note-taking tools and productivity workflows. Notes and search engines are merely text buffers that we use to store information in between the times when we occupy ourselves deeply with those ideas. The real possibility space of tools that help us think – tools that guide us through the vastness of the space of all good ideas – is much broader. And the extent to which we’ve explored it pales in comparison to the trillions of great ideas that may lurk just nearby, if only we had better maps to get us there.

Navigate, don't search

Thu, 03 Feb 2022 16:43:22 -0400

Humans are bad at coming up with search queries. Humans are good at incrementally narrowing down options with a series of filters, and pointing where they want to go next. This seems obvious, but we keep building interfaces for finding information that look more like Google Search and less like a map.

All information tools have to give users some way to find the information they’re looking for. For Google Search, it’s mainly the search bar. For a notes app, it may also be a search bar, but may also include things like links between notes, nested hierarchy of folders, or something like a “recently edited” page. For social media sites or other online services, the best way to find information may actually be some algorithmic recommendation system or a social feed.

All of these things, from text search boxes to links to algorithmic feeds, occupy places on a single spectrum, the spectrum of “interfaces people use to find information”. On one side are interfaces that teleport the user straight to the result, like URLs or search. These interfaces can be precise, but if the results aren’t what you want, you have to try again from the beginning – there isn’t an “incremental” way to use text search or URLs if you feel like you’re close but slightly off. Sure, you can tweak the search query a bit and re-search, but it may land you somewhere completely different.

On the other side of the spectrum are interfaces for approaching what exactly you’re looking for incrementally – links and recommendation systems. With these, you may not start exactly where you want to be, but with every click you can tell whether you’re getting closer or farther. Over time, not only are you likely to stumble into what you were looking for, you also end up with a mental map of a bunch of things related to what you were looking for that may be just as interesting to you.

Different they may seem, search, tags, folders, hyperlinks, and algorithmic recommendations are all really interface ideas trying to address the same fundamental problem: looking for the needle in an information haystack.

So which one is best?

In some cases, the right interface idea is obvious. For example, when you’re Google trying to make the entire web accessible, you can’t expect people to learn some monstrous cataloging system of folders to find what they need. There probably exists no tagging system comprehensive enough to cover the entire web, either. So the natural choice is the blunt instrument: text search. On the other hand if you’re building an app to help people find restaurants in the area, a literal, geographical map with some sensible filters and tags makes more sense than a hyperlinked web of restaurant reviews, because there is already some pre-existing organization scheme that most people are used to – geography and food preferences.

But when designing an interface for finding a note in a small pile of personal notes, or building an app to organize a small team’s working documents, most of the “finding stuff” interface ideas are in play. It’s in these situations where I want to make the argument: prefer interfaces that let the user incrementally move towards the right answer over direct search.

Humans are much better at choosing between a few options than conjuring an answer from scratch. We’re also much better at incrementally approaching the right answer by pointing towards the right direction than nailing the right search term from the beginning. When it’s possible to take a “type in a query” kind of interface and make it more incrementally explorable, I think it’s almost always going to produce a more intuitive and powerful interface.

Of course, search and incremental filtering are not mutually exclusive features. The language model-powered literature review tool Elicit shows an interesting way to make search incrementally explorable. In Elicit, after I search for a question like “How can I perform semantic search across a database of audio files?” I can select a few papers I like, and ask Elicit to show more results similar to the ones I just selected.

Incremental filtering interfaces also shine when the search space isn’t something that can easily be verbalized into keywords, like photo or audio search, or medical images. Cai et. al. prototyped an interface for medical professionals to study images of tissue biopsies for signs of cancer. In this interface, users can select images or sections of images and ask the system for “more images like this” tagged with diagnoses. The system uses image embeddings generated by a neural computer vision model to help the user explore the abstract space of tissue images incrementally. In this kind of a use case, it would be daunting or impossible to build an interface where the user could type in text queries to find the right comparison images. For this “finding information” problem, an incremental search interface is clearly the right one, simultaneously more intuitive and more powerful.

I’ve been pondering this gap between one-shot and incremental information-finding interfaces in the context of knowledge tools recently. Text search boxes are easy to design and easy to add to apps. But I think their ease on developers may be leading us to ignore potential interface ideas that could let us discover better ideas, faster.

Notational intelligence

Sun, 02 Jan 2022 14:48:11 -0500

I spent the last month wondering and investigating how we might design better workflows for creative work that meld the best of human intuition and machine intelligence. I think a promising path is in the design of notation. More explicitly, I believe inventing better notations can contribute far more than automated tools to our effective intelligence in understanding ourselves, the world, and our place in it.

This post is on the longer side (it’s really three essays in one), so we begin with a roadmap. Feel free to skip around.

To understand good notation and its role in knowledge work, we must first begin with a brief tour of notation in all its beautiful variety.

Part One: A brief tour of notational diversity

Notation, like its conceptual cousin language, is difficult to define precisely because it covers such a wide range of ideas. But as a starting point, let’s constrain notation to be some arrangement of symbols or images that we use to record and communicate ideas, and the way we interact with those symbols in our thinking process.

The most widely used and influential notation is writing systems for natural languages. It’s tempting to imagine spoken and written language as two sides of the same coin, but writing down spoken language changes our relationship to language and to our ideas. Spoken language is useful for communicating in small groups, and perhaps for thinking aloud. When we write ideas down, we make those ideas durable and reliable through time. This difference also delineates the shift from oral to written tradition in history. All of science, all of literature, all of trade and politics are built on top of the fact that we can write down an idea, and expect it not to change over time. When we record events in writing, we can have a coherent idea of facts as something separate from simply what we say, which might fade with our memories. Ted Chiang articulates the power of written language beautifully in Truth of Fact, Truth of Feeling:

Writing was not just a way to record what someone said; it could help you decide what you would say before you said it. And words were not just the pieces of speaking; they were the pieces of thinking. When you wrote them down, you could grasp your thoughts like bricks in your hands and push them into different arrangements. Writing let you look at your thoughts in a way you couldn’t if you were just talking, and having seen them, you could improve them, make them stronger and more elaborate.

The strength of writing demonstrates a common strength of all notations, which is to take something abstract, like the concept of “thought”, and make it concrete, into marks on paper or ink on a page. You can arrange thoughts and ideas into shapes and lists in writing, in a way that isn’t possible when thoughts turn into vapor the moment they are spoken aloud. With writing, I can hold onto ideas over time and refer back to them, unencumbered by the anemic limits of my brain. I can send them to a million people and trust that they will all read exactly what I had spoken within my own mind. This property of notation as embodied abstraction will appear many times throughout the rest of this post.

In contrast to the expressive flexibility of language, mathematical notation derives its strength from its rigidity and precision.

Math notation is really a family of conventions that are used together in practice to communicate quantitative ideas. Some of the greatest hits from this collection include:

Arabic numerals and the concept of place values, as opposed to Roman numerals, which had different symbols for different quantities entirely. Arabic numerals have the desirable property that it can represent arbitrarily large numbers concisely, and it makes operations like multiplication and division trivial. Previously, they were difficult black magic reserved only for the trained scholars who could multiply numbers like MCXIV and XXCVI in their heads.
Writing letters in place of numbers to represent unknown values in formulas, equations, algorithms, and theorems. This innovation lets us work with unknown values (or an infinite set of such values) as if they were simple numbers.
Fraction notation, wherein we write two things above and below a horizontal line. This notation makes operations like reducing, multiplying, and adding fractions straightforward – imagine having to multiply 5 fractions together while they were written like 2 & 3 or 150 ! 45, or perhaps IV & XII.
Vectors, sets, sequences, and the concept of representing a (potentially infinite) set of things with a single symbol. This lets us work with strange concepts like “the real numbers” as easily as we work with tiny, single-digit quantities.

Math notation is a powerful demonstration of how good (and bad) design of notation can make certain kinds of thinking trivial or impossibly difficult. The reason for this goes back to our earlier lesson from written language: good notation gives us the ability to work work abstract concepts (like sets or infinities) using the same cognitive muscles we use to work with physical objects, as concrete, embodied objects in our visual and tactile world. Because of this I think it’s helpful to think of the notation we use to represent ideas not just as simple tools, but as an integral part of how we think.

We think in languages; we think with notations.

Given that notation is so important to shaping our thinking, what properties might we want in a good notation? Mathematician Terence Tao muses on some desirable properties of good notation here. He begins by noting that a notation is only meaningful in the context of some domain of work. The symbol “IV” means something completely different in arithmetic than it does in cryptography or music theory. With that in mind, he emphasizes these properties (among others):

Unambiguity. Every valid expression in a given notation should have a unique interpretation in its domain.
Expressiveness. Every idea in the domain should be describable in at least one way using the notation.
Suggestiveness. Concepts that are similar in the domain should have similar-shaped expressions in the notation.

A good example of suggestiveness is in how we express addition and subtraction. 1 + −2 is equivalent to 1 − 2, and the notation makes it visually obvious. We can imagine many other ways of representing addition (say, as a function, like add(1, -2)) that would have been less suggestive of this equivalence.

A bad example might be differentiation and integration, which are closely related but have symbols and notations that are completely unrelated to each other.
Natural transformation. Natural ways of working with ideas in the domain should correspond to natural manipulations of symbols in the notation.

One great example of this is in the way we write fraction multiplication. When we multiply fractions, we simply multiply the top and bottom parts of the fraction separately – it’s visually trivial, and gets us the right answer.

Because of the versatility and precision of math notation, it has found widespread adoption in related fields like engineering, statistics, and computer science. Math notation in those fields often look slightly different (for example, in computer science, variables are often words, not single letters), but they adopt the basic principles above, like using letters to represent unknown quantities, or using a single symbol to represent a set of things.

For more esoteric stories of notational revolution in the way we work, we can turn to dance and other physical activities like juggling.

– Dance notation, using Zorn notation. From French Wikipedia, CC BY-SA 3.0

These domains had suffered for a long time from the difficulty of representing complex choreographies on paper. How do you write down even a short choreography onto paper in a way that would be precisely and unambiguously read back by a stranger? There have been countless attempts across history to invent better notations for representing dance and juggling – some of them have even led to invention of new moves or juggling patterns. More recently, cheap and easy sharing of video has revolutionized how practitioners communicate their motion. Compared to learning the notations and practicing reading back symbols from paper, it’s much easier to watch a dance video and follow along. Video is much more expressive, and also much less ambiguous. I think it’s fair to say video would be a promising medium for a new dance notation.

Our last example of classic notations lies in western music tradition: music notation. This notation is the total sum of the way modern composers and songwriters write music to be read back by performers or arrangers. Just like math notation, there are a few key concepts: the five-line staff, key signatures and notes, dynamics, and so on. But composers and instrumentalists might also use special notation to represent unconventional instructions (like humming, clapping, bowing styles for string instruments). Like language, music notation stretches to become as expressive as we need it to be.

– Questionable use of music notation, from @ThreatNotation on Twitter

As in dance and juggling, many performers are augmenting music notation with digital video and audio. Modern musicians might reference a YouTube recording of a world-class performer to help them understand nuances in tone that notation doesn’t quite capture, and vocal artists might reference demo tracks to remember exactly how they want to deliver certain lines without having to write down every modulation of tone in their performance.

Neither math nor music notation is easy to learn, but every practitioner learns them to reap the benefits of thinking in the notation, using the abstractions they provide. I think it’s worth noting that ease of learning is not a necessary property of good notation. This is because learning a new notation often involves learning to think with entirely new basic concepts. As long as the notation embodies useful concepts, the difficulty of learning it is vastly outweighed by the ease of thinking using those concepts once the notations feel natural to use.

Beyond these popular notations, there are specialized notations for every niche and field out there. My favorite example is the fishbone lab diagram used in medicine to record measurements concisely. There are notations used in screenwriting and theater, notations for describing electrical circuits, for professional copy editing, and on and on. Wikipedia has an even longer list of lesser-known notation. Humans invent notation whenever and wherever we need to work with common abstractions. Along with our ability to use tools, our ability to invent notation might be a defining characteristic of human intelligence. (Ken Iverson would probably say notations are tools, but here, I want to make a finer distinction between tools we use to manipulate physical objects, and notations we use to represent abstract ideas.)

Computable notation

With the advent of stored-program computers, humanity was presented with an entirely new class of notation: notation that can be executed. I use the phrase computable notation to describe this group.

The most obvious example of computable notation is programming languages. Programming languages inherit many desirable traits of natural language writing systems, like words representing distinct concepts and grammars to describe syntactically valid expressions. The main strength of programming languages is that they can be evaluated mechanically to accomplish a dizzying array of tasks. Programs can:

Compute some result or answer
Visualize some idea
Prove or verify a theorem
Run really quickly
Run continuously
Run multiple times and produce the same answer each time
Be proven to be correct (computer scientists call this “formal verification”)

Despite these unique properties, programming languages are unquestionably notation first, and instructions second. If the notational qualities of programming languages were not important, we would all just write code in assembly! Higher level languages let us work with new kinds of abstractions ergonomically, in the same way as any good notation. Which specific language we choose for a task depends on which languages give us the right symbols to represent useful concepts for that task.

One interesting programming language for our discussion of notation is Wolfram Language, the programming language used in Mathematica and WolframAlpha. Wolfram Language has a few unique advantages that makes it interesting as a powerful computational notation.

Wolfram Language is designed as a language to manipulate not just numbers, but also symbols. A variable in Wolfram Language can contain formulas, functions, and other sequences of symbols that mainstream languages cannot represent natively. This means variables in Wolfram Language can represent a wider breadth of abstract ideas than your average descendant-of-C programming language. If you’re curious about the inner workings of the language, Wolfram’s original paper on the “SMP” system is a good starting point.
Wolfram Language can directly represent images, video, and other media as a part of the language “notation”. When an operation returns a list of images, rather than showing something cryptic like [<#core.Image @ fffff3c>], Wolfram Language’s UI just shows you the image itself. While most languages are stuck in the pen-and-paper era where we could only record symbols as a part of our notation, I think Wolfram Language is a good case study for what might be possible if we seriously invest in building more media-native programming languages to directly represent many kinds of information more natively.
Wolfram Language outputs are interactive by default. Values in the language can yield interactive graphs and visualizations easily, which lets the programmer explore models or datasets while they are working with it in code.

– Wolfram Language working directly with images as a first-class part of its notation, sourced from Stephen Wolfram’s talk at re:Clojure 2021

Wolfram Language stands out as the only production-grade programming language that’s taking meaningful advantage of modern computers to elevate programming notation beyond the static string of letters they were since the beginning of teletype terminals.

Beyond programming languages, there are many other kinds of specialized notation that would benefit from being able to be executed. Imagine if geometric diagrams, circuit layouts, and chemical reaction formulas could be “run” on the computer to become interactive or automated. If we can endow the whole universe of human notation with the strengths of programming languages like verifiability or interactivity, I think it would represent a huge jump in the way we work with knowledge as a society.

Bret Victor writes:

The very concept of a “programming language” originated with languages for scientists — now such languages aren’t even part of the discussion! Yet they remain the tools by which humanity understands the world and builds a better one.

I cannot agree more. We are still only scratching the surface of computable notation.

Part Two: Interpreting intelligence

During our discussion of notation, one frequent theme has been that better notation helps us see new ideas, or work with more complex concepts more easily. Another way to say the same thing would be that better notation raises the intellectual power we bring to a problem. To make this argument more precise, it’s worth diving deeper into intelligence itself. How can we define it, and how can we measure it?

Intelligence as run-time adaptation

The most advanced intelligence we are aware of today (that’s us, humans) emerges out of biological evolution. Intelligence and evolution are both processes of adaptation — if evolution is a “compile-time” process, where favorable genetic influences are selected for at the beginning of life, intelligence is more of a “run-time” form of adaptation, where organisms can observe and respond to their environments closer to real-time. It’s also a much more generally useful adaptation: an organism with intelligence can react to many more kinds of changes in the environment than one born with a single genetic advantage.

Paul Chiusano comments on the relationship between intelligence and evolution in his blog, where he notes

Evolution is a meta-learning algorithm that discovers its own heuristics and continually improves the rate at which it learns.

Intelligence, as a kind of real-time-reactive adaptation, is the ultimate evolutionary advantage. Intelligence can respond much faster to shifting environments than natural selection, the mechanism driving evolution.

In the short term, evolution rewards specific solutions that fit the environment, but in the long term, the trait that wins is one that lets its owners adapt quickly to any change in environment or rapidly conquer new environments. To adapt quickly to new environments requires more than strength and power and speed — it requires the ability to understand and respond to completely novel challenges that organisms weren’t pre-programmed to deal with.

This leads to our second definition of intelligence: broad generalization power.

Intelligence as generalization power

AI researcher Francois Chollet, in “On the Measure of Intelligence”, approaches the problem of measuring general intelligence by first eliminating what it cannot be. He notes that we can’t measure intelligence just by looking at how much someone or something knows, because arbitrarily large amount of training or past experience can make up for lack of intelligence. Intuitively, this is akin to saying that someone is not more intelligent than another person just because they know more information or are more skilled in a particular field. We don’t define physical fitness by a performance benchmark in a particular sport or workout, but as something that generalizes across a wide breadth of tests, including some tests that might be completely new to the athlete. He argues a similar principle should apply to measuring intelligence: we should look for a general capability across many domains, rather than good performance in one particular skill.

Within this framework, Chollet defines intelligence as generalization power. To generalize is to find abstract patterns or ideas from a few examples and apply them to new, previously unseen problems. This is what deep learning models try to do at a primitive level. But general intelligence, of the kind that humans have, can generalize much more broadly than today’s DL algorithms.

A neural net can be trained to recognize images of cars in a video feed. It can be trained to detect positive sentiment in text. To succeed at these kinds of tasks requires learning certain abstractions, like “car” or “approval” or even sub-features like “wheel” or “adjective” or “sarcasm”. From these learned abstractions, we could say the models “generalize” these ideas to new examples to perform their tasks. These kinds of systems are capable of very specific, narrow generalization within a tight domain, like “image recognition on the road” or “sentiment analysis”, but traditionally, these models have not been capable of being applied directly to related domains like face recognition or question-answering without re-training. By the definition of intelligence as generalization power, we can say more intelligent AI systems should be able to easily generalize their skill across the entire visual or linguistic domain.

With more recent transformer-based models, we are learning that better models are capable of broader generalization. The title of OpenAI’s GPT-3 paper is literally “Language Models are Few-Shot Learners” — they assert that large language models are capable of broad generalizations from few examples. Indeed, language models like GPT-3 and XLNet are generalists that can accomplish a broad range of tasks in the domain of language, from sentiment analysis to summarization to conversational dialogue. Efforts like ViT and Image GPT demonstrate that similar approaches bear fruit for the visual domain, too.

Within DL systems like this, one way to understand model performance is as a tradeoff between optimizing for past examples and optimizing for future uncertainty. One training strategy might be to learn exactly as much as needed to be able to solve examples seen during training — kind of like rote memorization. This would be maximally efficient, but the model would be ill-prepared for the future uncertainty of new and novel examples. So better training methods make careful tradeoffs — the model learns extra information and abstractions that might not be necessary now, but might prepare it to deal with the uncertainty of novel future examples. Chollet writes in the paper,

Being prepared for future uncertainty has a cost, which is antagonistic to policy compression.

This idea of trading off efficiency for generalization power leads us to our final definition of intelligence, as data compression.

Intelligence as data compression

Both of these ways of looking at intelligence

learning abstractions, for future generalization tasks
trading off learning efficiency, for future uncertainty

are equivalent to a third way of understanding intelligence, as data compression — the same kind of data compression we rely on to send a 4GB file across the internet in 2GB of data, or to stream 4K video over the air.

When a 1GB model learns how to recognize images from a 1TB dataset, in a way, it’s compressing the meaningful bits of information out of that 1TB dataset into a gigabyte of data. It discards random noise that seems irrelevant, and “learns” the features and abstractions that seem important for recognizing visual features in potential future examples.

To effectively compress images, a compression algorithm would be advantaged to “learn” facts about the world, like that colors are usually contiguous in images, and that the ground is often green and grassy while the sky is often white and blue. To effectively compress English text, the model might be advantaged to “learn” abstractions like common words and frequent grammatical constructs, so it can avoid inefficient, rote memorization of letters as much as possible. Learning important abstractions this way also requires forgetting information that appears useless for future tasks. I think another elegant way to understand “learning” is as a battle of compromise between the need to remember and the need to forget – when humans and AI models learn, they are both fundamentally asking, I can’t remember everything – what can I efficiently forget, so I’m best prepared for the future?

This compression-based view of intelligence has also given rise to the Hutter Prize, a prize awarded for record-setting data compression algorithms in pursuit of artificial intelligence advancements.

Part Three: Notational intelligence

In Part One, we saw that good notation helps us work with abstract, complex concepts using concrete symbols. Notation is useful because we can engage the part of our brain tuned for working with physical objects, to instead work with abstract ideas represented by physical symbols.

In Part Two, we saw that the ability to discover and work with higher levels of abstract ideas, and generalize with them broadly, is key to our intellectual ability to respond to new problems.

Taken together, I think inventing better notation is a cheap, universally useful way to increase our effective intelligence in many domains. Better notations let us work more easily with more complex abstractions, and enable us to solve new complex problems. When faced with a new seemingly intractable problem, it’s worth asking not only “What tools and discoveries might help us understand this?” but also “What notation can we use to describe this problem domain, that might help us understand this?” While better tools can help us work more efficiently or precisely with ideas we already have, better notation can enable us to imagine and discover new ideas entirely.

Humans owe much of our effective intelligence not only to our biological capabilities, but to the huge diversity and versatility of notations we bring to specific tasks. If we weren’t allowed to use Arabic numerals in math or tables in scientific literature, we’d hardly be capable of the kinds of knowledge work we do so trivially today. I think it’s critical to notice here that access to good notation amplifies our biological intelligence, and arbitrarily powerful notation can extend our capabilities into arbitrarily complex problem domains. The phrase I’ll use to refer to this idea of intelligence from the use of notation is notational intelligence.

A different way to understand the importance of notation is to conceptualize it as “input encoding” of a problem into our minds. Just as the encoding format of input is important when building machine-learned models of languages or problems like protein folding, the “encoding format” of problems for humans is also critical to our ability to understand them. There are always details about reality woven into the notations we use, whether language or mathematics. When the human brain or machine intelligence works with information expressed in a specific notation, it’s also working with a model of the world built right into the shape of the notation at hand. I’m curious if this might be a reason why complex language models appear to have a coherent model of the world, when in fact they only understand the structure of language and nothing more – language as a notation encodes so much information about the world, that a sufficiently advanced model working with the notation of language also inherently works with some half-useful model of our reality.

If inventing better notation is so critical to solving complex problems, it only makes sense to investigate how we might invent good notation, and imagine what forms it might take in the age of software interfaces. The last part of this post explores these topics.

Dynamic notation

Humans invented most notation in use today for the medium of pencil and paper, and we still work with notation in the same static way: we cross things out, we erase them, we draw up new sheets of paper (real ones from dead trees, or metaphorical ones from pixels). The way we interact with notation has been limited by the static interaction medium of paper.

Software opens us up to a much broader scope of possibilities, though. Bret Victor explains:

The purpose of a thinking medium is to bring thought outside the head, to represent concepts in a form that can be seen with the senses and manipulated with the body. (This is, for example, how paper enabled complex mathematical derivations, logical argumentation, navigation…) In this way, the medium is literally an extension of the mind.

However, static media are extremely limited in what they can conveniently represent, so much thinking is still trapped inside the head. (For example, the behavior of mathematical expressions, variable values in a computer program, connections and references between books in a library…) The dynamic medium has the potential to represent such concepts directly, to bring them out in the open, where the entire range of human faculties — all senses, all forms of movement, all forms of understanding — can be brought to bear on them.

Just as he pursues a world of dynamic media, I think we should aspire to build more dynamic notation: notation defined not only by its symbols and the way they lay out in space, but also by the way we interact with them.

Interactivity is a defining trait of computers and software as a medium, and what sets it apart from paper. But today, we interact with notation in software no differently than we interact with them on paper, which is to say we don’t interact with them much at all, beyond reading and writing them.

When games took over the computer, game designers didn’t simply simulate board games on digital screens (though that niche has its devotees) – the collision of gaming and computing spawned an entirely new field and industry dedicated to harnessing the full interactive power of software in pursuit of building ever more immersive worlds in which people could play. Yet when musicians and scientists and statisticians moved from working on paper and typewriters to working almost entirely on the computer, they simply cloned the paper medium into software. An equation in a PDF or on a website is no more immersive, interactive, or movable than ink on paper. Music notation has a more interesting story – you can write music in the standard notation in software, and most professional audio workstations or composition software will let you play it back and “execute” it, the way a programmer might execute a program. I think we can push most notations much farther along this path.

Notations are our interaction instruments for working with abstract ideas. We can’t literally grasp and manipulate abstract concepts, but we can write them down and move symbolic shadows of them around a table. How might software let us interact even more directly with our ideas? Here’s a speculative sample of possibilities:

Direct manipulation of ideas by dragging or zooming in on symbols. I want to type in the formula for gravity, and pinch in on the symbol for mass or radius to see how changes in different variables affect the overall system I’m studying. Maybe this equation can also be automatically graphed for me, and I can pinch and zoom on the graph to see corresponding changes reflected back in my equation. I want to write an outline of notes from reading a few different books, and drag and drop to move bullet points around and organize the list by different topics, by the tone of the ideas, or by how speculative different claims are.

First-class support for rich media in notation. I want to be able to copy-and-paste a vocal audio track right under the ledger lines of my song and play it back all together. When I’m reading about an architecture for a new image-processing algorithm, I want to be able to drag an image onto one end of the diagram and see how different steps in the algorithm transform the data visually. When I’m studying the mathematics of ray tracing or fluid dynamics, I want to be able to place my favorite 3D model into an equation in front of me and see normal vectors and force fields laid out before my eyes. Unlike on paper, working in the software medium lets us represent rich media like photos, 3D models, and audio faithfully. We should take advantage of it.

– In an excerpt from the research below, we see machine-learned representations of the concept of font weight.

New notation representing abstract, fuzzy concepts quantitatively. In “Using Artificial Intelligence to Augment Human Intelligence”, the authors demonstrate how generative machine learning models can “learn” quantitative, concrete representations of fuzzy, vaguely-defined ideas like the italic-ness of a font face or the chemical instability of a particular molecular structure. I think this kind of technology, where we can quantitatively learn representations of abstract concepts, enables us to build notations for entirely new kinds of domains. How might we design a notation to describe typefaces as a composition of different styles? Could we write down formulas for chemical reactions not by explicit chemical structures, but in terms of their abstract chemical properties?

In any creative, intellectual task, there is tremendous value in being able to play with ideas, where people can explore and experiment freely without having to wait for their code to compile or their simulation to run. To make this kind of play possible in the creative workflow, the feedback loop between a what if and its implications has to be as short as possible. We shouldn’t settle for a world where we must translate lines of equations into code and paragraphs of ideas into table rows. We deserve interactive, dynamic notation, where we can play with variables in our mental models and hypotheses as easily as we move a video game character across the map.

How to design good notations

In my exploration of notations, I’ve collected some useful guidelines for designing good notation. This section is more speculative and less comprehensive than the others in the post, but I think of it as the beginning of a list of ideas that can help steer us away from the bad and towards the good as we design and extend notations.

Find a property of a concept that current notation expresses badly, and focus on expressing that more faithfully. Innovation in notation often arise out of a new way to represent some idea that was complex to express in a previous notation, or just wasn’t concretely expressed before. A trivial example is where music notation spells out chords and beats above each measure, rather than the precise notes to play. Another example from mathematics is the limit, which has a concise and precisely defined notation despite the complexity of the idea itself. Figuring out the right abstractions to realize into symbols is the key to good notation design.

A related guideline is to roll up ideas that are complex in current notation into more concise symbols. The most obvious example of this is summation (Σ) in mathematics, which wraps up a potentially infinite list of additions into a single symbol we can manipulate. In computer science, the “big O” notation embodies a nuanced idea called algorithmic complexity.

Try to express dualities in ideas that existing notations don’t suggest. For example, the union of two sets in set theory is a parallel idea to the idea of logical disjunction in formal logic. Mathematical notation for a set union looks like A ∪ B, and logical disjunction is written A ∨ B. See how they look pretty similar? The similarity in notation suggests similarity in concept. Unfortunately, most conceptual dualities like this aren’t reflected in notation. In music, F-sharp and G-flat look completely different, though they’re the same pitch (unless you’re doing some avant-garde musical experimentation). Differentiation, integration, and Riemann sums are all different views of the same idea in calculus, but their notations are nothing alike. Addition, multiplication, and exponentiation are versions of the same fundamental operation (doing some additive operation repeatedly), but they also have completely different notation. It would be interesting to design new notations that are more suggestive of these hidden relationships between ideas.

Notation doesn’t have to look like language, or like symbols on a horizontal line. As we imagine new kinds of notation, it’s easy to get stuck in a status quo bias, thinking within the set of things we can type into a computer and look and feel like language or math. I think we should be aware of the risk of falling into this trap, and actively imagine non-linguistic notation that take better advantage of two- or three-dimensional, interactive capabilities of software. For example, in digital audio workstations for music production, tracks aren’t expressed using notes on ledger lines like parts in traditional music notation. Instead, tracks are visualized so that they can shrink and expand to reveal detail or high-level shape as the musician moves around the software interface. This interface represents a new kind of notation for music, one that provides unique benefit in an interactive interface.

– Notation doesn’t have to look like language. (Taken from “Lambda diagrams”)

Whatever the exact form, future notations native to the software medium will depart often from its static predecessors, only rhyming with history where it’s useful.

Future directions

Throughout the post, I’ve highlighted two promising paths for augmenting our thinking and creativity with better notation.

Improving our existing notation to be more interactive and dynamic
Inventing new notation to let us work quantitatively with more abstract ideas

Though it’s early, I want to end with some of my thoughts on what future work in this direction might look like.

One path I’m beginning to investigate is notation for working with complex networks of ideas. I explored the diversity of domain-specific notations in this post, like music notation or regular expressions, but the most common, versatile, and intuitive notation is still natural language. How can we improve the way we work with written language to be more interactive and dynamic?

Complex ideas often present themselves as densely connected networks of smaller ideas. In light of this structure, I think a good dynamic format for documents should help readers and writers explore these graphs of ideas intuitively. Perhaps we can make effective use of the human brain’s finely tuned ability to navigate physical spaces, and construct documents that feel like “spaces” that we read by “navigating” it. Rather than having “notes” that are just collections of homogeneous chunks of text, maybe notes can take on different roles in this map of ideas – some notes are more “paths” that help us discover other notes; some notes are more “districts” that act as topic-level containers, and some notes are “landmarks” that go deep into specific concepts.

There are also worthwhile sources of inspiration elsewhere: Within the world of software development, we’ve invested decades of effort into graph-exploring interfaces, too, called code editors. Source code is often graph-shaped, and I think many of the ideas that apply to navigating complex codebases would be equally fit for navigating complex webs of ideas.

Lastly, I want to investigate ways to make written language itself more dynamic. What if we could ask a paragraph discussing an abstract idea to produce examples of its own argument? What if you could turn a dial on any article or blog post from “1-minute summary” to “book-length treatise” to get exactly the depth and brevity you want to spend on a topic? In the same way we can manipulate numerical models in Excel, I want to imagine dynamic documents where I can extend and stretch ideas expressed in prose with the help of software. I want my notes to self-organize into topics and lists based on the context of my current thought. When we dream of interactive notation for thought, I think these kinds of interactions should be our starting place.

Another important open question in this space concerns measuring our progress. Once we have experiments and prototypes for better notation, how might we measure our success? What makes new notation valuable?

The value of notation lies in how it enables us to work with new abstractions. With more powerful notation, we can work with ideas that would have been too complex or unwieldy without it. Equipped with better notation, we might think of solutions or hypotheses that would have been previously unthinkable. Without Arabic numerals, we don’t have long division. Without chess notation, the best strategies and openings may not have been played. Without a notation for juggling patterns called Siteswap, many new juggling patterns wouldn’t have been invented.

Right now, today, we can’t see the thing, at all, that’s going to be the most important 100 years from now. – Carver Mead

I think notation should be judged by its ability to contribute to and represent previously unthinkable, un-expressible thoughts.

A notation is not a single feature or symbol. Innovation in notation and dynamic documents won’t arrive as a single big idea or a single product from a new startup. Just as mathematical notation isn’t just the Σ sign or the concept of using a letter in place of numbers, the revolution we seek won’t be just bidirectional links or just spatial interfaces or just GPT-whatever. As generations of companies and products come and go touting new ideas and interaction mechanics, we shouldn’t ask “Does this product finally get it right?” but rather “What good ideas can we take from this to our vision of better notation, better documents, and better ways of working with ideas?”

There’s a subtle but important distinction between augmenting human productivity, which is every tech company’s aim, and augmenting human intellect. The former is an economic endeavor that helps us accomplish more of what we could do already; the latter is a step towards helping us invent new things and see new ideas we could not before. In the path to build newer and better tools, we should not be blind to the fact that the future has always been defined by nascent ideas – ideas that may be written in words we don’t recognize yet and symbols that may not yet exist. It is up to us to ensure that by the time the world needs those ideas, we can speak of them, write them down, and pass on the knowledge.

Towards a research community for better thinking tools

Sat, 27 Nov 2021 22:28:02 -0500

As I’m thinking about defining more narrow focuses for my independent work next year, one area has stood out consistently as both personally exciting and more widely important: imagining and building better ways computers can help people do their best creative, thoughtful work, and in the process rethinking the relationship creative people have with the computer as a part of their work.

My projects and this blog have never had a clearly defined focus, but a big part of my project portfolio is tools I’ve built to help myself manage the information that flow through my life and work. This theme has emerged slowly but definitively in the last year with projects like personal search engines and a custom Twitter client. This year, two things have happened in my life to help me realize this is a field in need of more independent, dedicated, inventive research efforts.

First, there is an increasing availability of capital flowing into companies building on new ideas in this space. Startups like Coda, Remnote, Mem, and Notion are raising millions of dollars on first investment rounds, and they’re coaxing even the very large tech companies to invest seriously into more competitive tools for thinking and creating together. There has always been a die-hard group of thinkers who argued that the “computer revolution isn’t here yet” and that the best software creative tools were yet to come, but the new influx of capital validates financial interest and demand for tools that break new ground in a way that hasn’t been true in many years.

Second, I’ve personally invested more seriously in designing and building more ambitious projects, like a universal personal search engine and a web browser focused on knowledge work. In response, I’ve spoken with many other brilliant and creative people working on similar projects, both independently and as a part of early stage companies. These conversations have reiterated in my mind how much I enjoy working at this layer of tool building, designing the basic metaphors and building blocks out of which the rest of the world runs businesses, shares new ideas, and creates art.

As I reflect on these realizations, I’ve been wanting to invest more time and effort specifically into building these kinds of tools, and conducting more exploratory, open-ended research that can fuel new creative ideas about how to build better thinking tools. This also spurred some interesting conversations I’ve had with other folks in the field about what an ideal research community could look like. What kinds of people and companies would push the ecosystem in the right ways? How might these researchers and companies communicate new ideas with each other in a way that results in open, lasting progress?

What follows is a loosely structured collection of different building blocks from which I think we can build a good research community to push this space forward. Though I’ve spent the last couple of weeks thinking about this topic, there’s no doubt I’ve missed some important pieces. If I think of any other, I might come back and add to this list. A brief table of contents:

Community and identity

Great communities are made of the stories we tell about ourselves – why we do what we do, how we’ve done it, and from where those traditions came. These, alongside how we talk about ourselves and our work, form the identity of our field. I think it’s worth being conscious and deliberate about such an important facet of the community.

So far, the way we label our work (“tools for thought”, “knowledge graph”) and the way the world perceives this field have been defined mostly by the loudest and biggest companies to fill the room. This kind of ad-hoc identity is enough to bring people together, but these words we use today are laden with overloaded meanings and associations to existing products and ideas. Different corners of the community behind these tools also agglomerate around different focuses: Some really love to customize and polish their tools and workflows. Others are more interested in building on fundamental interaction and interface design research dating back to the dawn of the personal computing era. One name and identity cannot contain them all – I think a research community should recognize that, and gather around a more independent identity that can represent the way we want the community to treat its members, its history, its traditions, and its future.

If I had my way, I think this new identity should…

build upon a diverse group of people and ideas
remember, but not revere, past research and tradition
welcome independent contributors, and view itself as a collective of people, not an industry of companies
work in the open, and
value building and testing ideas over spreading them.

There are entire essays to be written on each of these points – probably soon to come.

There is already a loose emerging community around thinking- and creative-software research. The challenge is to bootstrap a more structured community from these seeds. By “community”, I emphatically do not mean some group chat or online forum where every member hangs out and post messages. That kind of a monolithic community approach does not scale, and does not last. I want a research community that feels decentralized, but runs in roughly the same direction, branching and merging where the ideas take us. I want there to be people I can run to with a new idea or a prototype and get honest, hard-hitting, thoughtful feedback, and that doesn’t require much shared infrastructure, only shared enthusiasm and identity.

Communicating research

There are emerging conventions in the community about how to communicate research, and I think they set good examples from which to build traditions: written communications and open-source prototypes.

Written communications

There are two broad categories of written research communication I can think of: reports on findings after specific projects, and periodical updates on progress. I think the best example of the former is Ink & Switch’s reports. My favorite examples of the latter are Andy Matuschak’s Patreon posts and Alexander Obenauer’s lab notes. Both feel important to a vibrant research community.

In-depth write-ups of research findings can serve as anchor points and canonical descriptions of important ideas. There is value in giving names to new ideas and describing them in detail for others to cite and build upon. Reports of this type can serve that role. Well-written research reports also clearly lay out findings that future research could extend, and approaches and experiments that didn’t work. In a productive research community, research builds on past research by referencing and working from findings shared in these in-depth reports.

By contrast, periodical updates can fill the time gaps in between long research cycles and form a kind of asynchronous “group chat” for the community. Often the most interesting parts of research work are the parts that are fuzzy and vague and incomplete in our own minds, and hardest to articulate properly at first. These ideas don’t make it to “official” publications or reports because they’re not ready to be cast into form yet, but speaking about these more soft ideas with like-minded people can lead to the clarity we need to move forward on them. I think periodical updates from researchers and builders in this space can become a place to share those more fuzzy ideas. These communications might also be an effective “heartbeat” for the community, to keep a sense of loosely coordinated momentum.

I also want to note a third kind of communication – pieces like Up and Down the Ladder of Abstraction and Using Artificial Intelligence to Augment Human Intelligence – that push the field forward on what effective research communication can look like. As people working to expand the way we communicate and share ideas, I think it’s worth investing into the ways we communicate our own ideas between ourselves to make it more interactive, taking full advantage of the software medium we seek to embrace.

Open-source, working prototypes

One of my favorite things about the way this field operates is that often research yields open-source prototypes and working code. Szymon Kaliski’s projects don’t come with a lot of writing but self-demonstrate ideas because they’re working code on GitHub. It seems the culture of this ad-hoc community we have so far values open source software, pushing some companies to build products on a platform of being open-source. I’ve also embraced this culture; almost all of my own projects are free and open source. A community culture of open-source-by-default could also enable more ratcheting progress in a field increasingly tucked behind startup NDAs.

Outward-looking problem discovery

One of my greatest concerns about this field today is that almost all of the problem discovery happens by a kind of self-interested navel-gazing process, where product builders take the quote “build things you would want to use” a little too literally, and build products for the small niche group of people interested in note-taking tools and processes. This leads to products that seem useful to a small group of other people who are also working in this space and familiar with its vernacular and concepts, but are unusable or unapproachable by most people outside of that small community. I think this is a dangerous failure mode.

Good industrial research can only happen in problem rich environments, where research questions are anchored to problems found in real workflows used by domain experts working in their respective fields to solve problems other than the challenge of building more thinking tools. This kind of work requires that the research community work in close collaboration with people who work outside of our field. This is a process that requires active effort – radiologists and human rights lawyers and artists and journalists are not going to seek out risky, new ideas in their knowledge workflows as a part of their daily work. I think active outward-looking problem discovery, where we dedicate research time to consult with other domain experts to seek out new questions to pursue, is critical to an impactful research community.

Concretely, this can happen in a few different ways. Bell Labs’s problem discovery was performed by the entire rest of its parent company, AT&T, as it tried to span telephone wires across continents and oceans and anticipate technical problems 5-20 years in the future. Companies like Retool building business-facing products have field-deployed engineers working with customers to discover new uses cases for their products and ensure the rest of the company is directed forward, not just inward. Independent researchers like Andy Matuschak augment their own personal experience with tools they build by testing rigorously with other end users in a variety of professions.

I’ve spent a lot of time over the years desperately trying to think of a “thing” to change the world. I now know why the search was fruitless – things don’t change the world. People change the world by using things. The focus must be on the “using”, not the “thing”. Now that I’m looking through the right end of the binoculars, I can see a lot more clearly, and there are projects and possibilities that genuinely interest me deeply. — Bret Victor

Tools don’t change the world alone; people do, in the way they use tools. And the way to build the right tools for those people is to focus on the people and their work first.

Proven models for sustainability

As I think about shifting more of my own work into research, one of my own big questions concerns financial sustainability. For talented, experienced people to consider contributing to a more open-ended research community rather than going to work at any of the hot startups of their time, there must exist well-established models for making a fair income doing such work. I don’t think it’s necessary for researchers to make salaries competitive with top-of-the-line compensation from large tech companies, but pursuing research must not be a financial gamble, the way it is today for most people.

Some people have found success with a crowd-funded Patreon-kind of funding model. Even though ostensibly making is showbusiness now, I don’t think this is the proven revenue stream we want everyone to pursue. Not everyone wants to turn their online presence into pristine, well-curated identities about their professional interests. Even for those who can manage taking on a creator identity, I think a crowd-funded patron kind of model can lead to undesirable power dynamics where researchers may be pressured to pursue questions that satisfy mass-market curiosities most, rather than following their expert intuitions.

So, what can we do?

For a healthy and impactful research environment in the thinking tools space, I think we need a mix of “concept car” projects (a phrase I lovingly borrow from Jess Martin) and “production-grade” tools (a phrase I’m adopting from Ink & Switch). Concept car projects explore the boundaries of current technologies or showcase what new designs and ideas enable. They are necessary to push the field forward, but usually too rough or incomplete for the rest of the world to depend on. Production-grade tools are tools that are battle-tested to be secure, reliable, intuitive, and polished enough to be load-bearing components of real-world workflows.

These two kinds of work likely need different models for financial sustainability.

Research fueling products, products motivating research

The industrial research lab Ink & Switch offers an interesting precedent for a research group bringing products and production-grade tools to market, like Muse and Automerge. Though the lab itself is a nonprofit focused on exploratory research, some of their work leads to production-ready products that the lab can spin out into profit-generating companies. This approach, of research labs becoming sustainable by bringing products to market, resonates with me. Industrial research sets itself apart from academic research by the fact that industrial research seeks to answer questions relevant for building new products and companies. This was the explicit framing of research work done by Bell Labs and Xerox PARC, and the framing that labs like Ink & Switch also seem to be following. From this perspective, it makes sense for labs to become sustainable through research that leads to commercially successful products.

If research fuels creation of new products, I think building products can in turn fuel further research by being active areas of problem discovery. Much of the difficulty of building new products isn’t in coming up with the initial idea or insight, but in the thousand different engineering and design refinements that need to be made before a prototype can be turned into something the average customer will be able to use to solve their problems. These range from technical challenges like building a high quality rich text editor to design problems like balancing UI complexity with customizability. Once these products enter the real world, new problems always emerge at the point of contact between real-world use and research findings. These new problems can then feed back into research, and the cycle can continue.

The downside of taking this approach is that there’s a constant need for start-up capital in the beginning, when researchers and labs won’t have products to sell. Grants and corporate research programs may fill that role.

Financial support and research grants

In a world where labs become sustainable by spinning out products, researchers need some way to de-risk their initial work, when they won’t have any new products or technologies to sell. I think this is an effective place for open-ended research grant programs.

We can frame the role of research grants in a few different compatible ways:

Grants can help people who are working in other fields step into the research community without taking significant financial risk up front.
Grants can help fight the tendency for research projects to become too focused on short-term marketability. Projects should be opportunistic about, but not driven by the potential to build good products.
Grants can direct support and resources toward researchers working on problems that seem especially fundamental or important in the field, and the community can use this support to advocate for its values, like open-source work, diversity, and a focus on people rather than companies.

I’ve also observed more startups in the field opening up “Researcher in residence” programs. These roles can be another way for new people to step into research work without associated risks or lack of structure, but I think we should avoid a world where most research about these tools are done in-house by companies. Corporate research ultimately results in proprietary intellectual property that is harder for a community to build on, and it puts emphasis on companies and their products rather than individuals and their learnings. I think corporate research programs should follow the conventions of the community, and be careful not to shadow individual contributors’ efforts.

Many small projects, building on each other

I think smaller projects that are faster to build are better for research in this space. Building many smaller projects rather than large ambitious ones have helped me because I avoid getting too attached to one particular idea or product, and with smaller-scoped prototypes I can try many more iterations against the same question or problem. It also lowers the barrier to entry to try more risky ideas – “I’ll try this for a weekend” is much easier than “I’ll have to shift my schedule the next couple weeks to fit this in; is it worth that?” A culture of shorter, more atomic projects will also encourage everyone to break down large ideas into smaller ones that are individually testable, which I think is a good practice regardless of whether those ideas are for a product or an experiment. On the other hand, cycles that are too short obviously run the risk of keeping us from trying more ambitious or complex ideas.

My gut feeling is that three-month “cycles” focused on specific research questions strike an ideal balance. Any longer, and we might find ourselves not chasing after a concrete enough question; any shorter feels too short to really dive deep into a problem and try as many iterations as might be necessary to find good answers. If we can establish a culture in the research community of 3-month cycles pursuing a single question, it might also be a good foundation on which to build timelines for things like research grant programs, community conferences, and collaborative projects.

Thanks to Karina Nguyen and Theo Bleier who offered feedback on past versions of this post, as well as the many independent researchers and writers whose opinions and work appear above.

AI as a creative collaborator

Tue, 16 Nov 2021 00:37:08 -0500

I spent the last couple of months delving deeper into how I could integrate elements of modern machine learning with my love of building personal knowledge tools. This is a space brimming with untapped ideas and experiments to come. One open question for me is how exactly human users should interact with AI integrated into knowledge tools and creative tools.

There are undoubtedly many ways to break this apart, but today, I’m finding it helpful to draw a spectrum from AI being used as a tool to AI being treated like a collaborator in our workflows.

AI as a tool

In this model, the human calls on the AI to improve or enhance a thing they’re trying to achieve, in very specific, well-defined ways.

This might take the form of annotations or specific UI elements to “call on” the AI, like some language model leaving comments on your essay to fix mistakes, a feature that highlights awkward wording, or an easy way to automate repetitive design tasks in a prototyping tool. In marketing speak, this might be called “Smart X”. e.g. “Smart erase” to automatically clean up a photo in a photo editor, or “Smart spell check” to automatically fix nuanced grammar mistakes.

As a real case study, Apple’s last few generations of iPhones have featured “Smart HDR”, which takes the basic principles of HDR, compositing multiple photos from one point of view to achieve better quality, and leans into AI to do more like reducing blur and adjusting skin tones.

With this approach of integrating AI into our creative workflows, the AI is always subordinate to human users. It has no agency but that which is granted exactly and literally by the human operator.

AI as a collaborator

In this model, the human and the AI are two independent, autonomous agents at equal level of engagement with the work being produced, and they have access to the same interaction mechanics and tools to accomplish the task together. Working with this kind of AI is like working with a smart human collaborator – you don’t invoke them to help you accomplish something specific they’re there to do; you learn how they think, they learn how you think, and you develop a sense of how to produce the best ideas together. The collaboration is much more organic, and there’s a constant feedback loop informing both participants about the ever-changing creative direction.

I could imagine a world where photo editors, writing apps, note-taking apps, and IDEs all have collaborator-style AI built in that you can turn on for it to lurk in the background and chime in here or there with suggestions like:

“This style of color you’re using reminds me of this artist you’ve liked / you haven’t heard of yet. I’ll paste some images from their portfolio here.”
“This question you wrote down sounds like a good one to tackle with this mental model you wrote about last week. Let me fill out the template for this question here.”
“You’re using this pattern a lot in your code — here, I’ll refactor it into a function/template we can reuse.”

There are lots of details to iron out if we want to bring any of these ideas into real tools today, not to mention advancing the state of AI to achieve this level of helpfulness and accuracy. Details like whether the human should be required to approve every machine-made suggestion or to whom copyright is assigned. But these minutiae aren’t so important for the thought experiment I am laying out here, because one way or another, these questions will be answered.

The collaborator model has two benefits:

It dramatically opens up the potential capabilities of AI built into our tools, because the AI can do anything a human collaborator could do without needing some extra button or call-out in the interface for controlling “AI-assisted” features.
The fact that AI interacts with the tool and the human creator using the same mechanics familiar to the user already means there are no new interaction mechanics for the user to learn to take advantage of its intelligence.

The second benefit feels important to me. In Transferable mechanics, Mary Rose notes that good interaction design often lets users apply knowledge about mechanics they learned before to interact with something, to then interact with something new. This lets them gain new capabilities without understanding new mechanics or abstractions. It helps new users learn to use tools or play games much faster, and makes the process more enjoyable.

The “type-ahead” suggestions we’re starting to see in Google Docs and Gmail is an imperfect example of the collaborator model of AI. Rather than having a “complete the sentence” button in a right-click menu, the AI “types ahead” in little grey letters, so you can confirm its suggestions with a single tap of the Tab key. There’s hardly anything new to learn, because it’s just finishing our sentences. Something feels pretty right about it, compared to having to “call on” or “invoke” a feature. GitHub Copilot uses a similar mechanic, and it feels pretty good to me too.

In these tools, the AI inherits our powers, and we inherit their intelligence in return by working alongside them using the same buttons and knobs.

Obviously, the reality is that there are many features and ideas that will fall somewhere in between these two archetypes. But I think this spectrum is a useful tool to help us imagine interesting ways we can design AI into our creative tools: how much agency should it have? How specific or general should its expertise be? Is it a button we press, or another player whose footsteps we can follow to stumble upon something new?

Thanks to Karina Nguyen for helpful feedback that added to a revision of this post.

The web browser as a tool of thought

Tue, 10 Aug 2021 17:03:58 -0400

A month ago, I built a personal search engine called Monocle that let me search through a trove of personal information I’ve saved over time, from notes to journal entries to bookmarks and tweets. Shortly thereafter, I switched my default search engine in my web browser from Google to Monocle, marking the start of my slow descent into the fascinating rabbit hole that is transmogrifying my web browser into my best, most flexible, most versatile tool for thinking, learning, and remembering.

A couple of weeks later, I built and started using Revery, a browser extension that shows quick summaries and topically related notes and bookmarks from my collection whenever I’m reading anything on the Web.

Living with these small bits of customization and intelligence scattered throughout my browser, I’m increasingly convinced that the future of the web browser is the best tool – nay, medium – for thought.

This conviction comes from a myriad of ideas and realizations, but most significant among them are three insights I’ve discovered while building and living with these projects for the last month:

At the end, I’ll try to bring these ideas together to muse a little on what I want to see as the future of the browser.

Medium for thought, not just a tool

In Designing a better thinking-writing medium, I wrote on the difference between a tool and a medium:

A tool is something that takes an existing workflow, and makes it more efficient. A nail is an efficient way of holding pieces of wood together; a to-do app is an efficient way of remembering your responsibilities. A medium, on the other hand, gives us new agency or power by which we can do something we couldn’t do before.

As with any dichotomy, there are grey areas. Powerful, effective tools can become mediums and enablers too. The graphical computer user interface wasn’t just a better way to write scientific simulations or data processing systems – it also became a new medium for creative work. Programming languages began history as a more efficient way to store and maintain punchcard programs, but a half-century of innovation has made it a medium for expressing programs that couldn’t be written before.

Despite the renewed focus I see in the community of people and companies trying to build better tools for thought, I think much of our work is still confined to tool-making. That is, most of our efforts are about creating more automatic, more efficient ways to do what we already know how to do – spaced repetition, Zettelkasten, journaling, and so on. We are busy making more effective command-line apps for thought, rather than dreaming up graphical interfaces.

Building a tool is a relatively straightforward affair. We can look around at existing workflows and needs that people have, and design some set of features around the workflows and needs that we observe.

However, to build an enabling medium that’s more than a single-purpose tool, it isn’t simply enough to look at existing workflows and build tools around them. To design a good creative medium, we can’t solve for a particular use case. The best mediums are instead collections of generic, multi-purpose components that mesh together well to let the user construct their own solutions. For example, Microsoft Excel is ostensibly a tool for calculation, but it’s also a medium for manipulating data in a 2-D grid for lots of other use cases, from organizing a budget to collecting a poll to even creating simple graphics. This flexibility comes from the fact that Excel is really just composed of a few powerful primitive components: the 2-D grid of cells, formulas that can reference other cells, and a responsive programming model that lets the whole table change anytime a value somewhere changes.

Designing a medium for thought requires that we discover what these primitive components of a thinking medium should be. Should there be some sense of geometry and space? How important should text be, against drawings and images? How should people collaborate and share their thoughts? I propose that the solution to these questions are not an opinionated tool with a “Share” button and a rigid way to use an image in a project, but something with a collection of capabilities that happen to include inserting and positioning text and images, sharing and collaborating on those objects on the page, and connecting ideas. These capabilities should work well together, to leave room for any combination of use cases. The line here can be blurry, so I don’t think it’s worthwhile to argue whether any particular tool is a good tool or a good medium, but in the future, I want to see more mediums like Excel, Google Docs, and Figma that seem like multi-purpose canvases that leave room for creativity, and fewer tools like Zoom, Slack, and of course the venerable web browser of today that lock you into particular use cases and make you feel like you need to “learn how to use” the thing.

The current state web browsers is particularly damning from this perspective. Web browsers have access to such a treasure trove of valuable, often well-structured information about what we learn and how we think, what interests we have, and who we talk to. Rather than trying to take that information and let us build workflows out of them, browsers remain a strictly utilitarian tool – a rectangular window into documents and apps that play dumb, ignorant of the valuable information that transits through them every day. I think we can do better.

Information lives outside of tools

Once I started using my personal search engine, Monocle, day-to-day, I made an observation about how Monocle seemed different than most of my other software tools. I wrote on my newsletter:

Every productivity app company these days seems to embrace the phrase “second brain,” as in “make X app your second brain.” One of my big takeaways from using Monocle on a daily basis for the last week has been that no single app can be my second brain. There are going to be parts of my life that are inherently spread out across different apps. For example, there’s a huge amount of knowledge sitting in my email inbox and my blog, and there are some things I only remember because I tweeted about it once, or recorded in a quick journal entry. There are ideas saved in text messages and contacts. “Notes apps,” it turns out, are not the only places where knowledge lives. And a true “second brain”, or whatever you want to call it, needs to recognize that and let you wield its magic over all of your digital footprint.

Most of us don’t realize just how much the “app-centric” mindset is ingrained into us, until we get a chance to think in a “problem-centric” way free from the limitations of apps. One of my most frequent kinds of search on Monocle is a search for people – people whose names I vaguely remember, people whom I’ve never heard of but might have been mentioned in a news article, people whose birthdays might be coming up. Before Monocle, to find some information about a person, my first thought would have been “What app did I write that down in? Should I check my messages or my contacts?” I might even have given up, after searching a couple of apps without finding the result I was looking for. With Monocle, the question doesn’t need to be asked. I just search for the right person, and Monocle searches across all my data.

If we want to organize information that flows through our lives, we simply can’t restrict our design space to be a single product or app. No matter how great a note-taking app is, my emails are going to live outside of it. No matter how seamless the experience in my contacts app, my text conversations are going to live outside of it. We should acknowledge this fundamental limitation of the “note-taking app” approach to building tools for thought, and shift our focus away from building such siloed apps to designing something that lives on top of these smaller alcoves of personal knowledge to help us organize it regardless of its provenance.

If we want to build a software system that can organize information across apps, what better place to start than the one piece of software that has access to it all, where most of us live and work nearly all the time? I think the browser is a rich place to build experiments in this space, and my personal experience building Monocle and Revery support this idea so far.

To surf the Web, you must first understand it

One of my perennial complaints about the current crop of “tools for thought” has been that most of them aren’t really about thinking per se, just about improving memory. We can take down information into apps like Roam Research or Notion and recall them easily later, but it’s an overstatement to call them tools for thought. Recollection is such a small, basic part of thinking! There is so much more to thinking than simply remembering something accurately. I think we can acknowledge the benefits these apps bring while also admitting that better tools for thought should help us do more with ideas than just remember and recall.

Even when using the best of these apps, it’s up to the human users to manually annotate every single connection, every single hierarchy, and every single highlight and summary we want to remember about what we write. Humans are still doing most of the “thinking” work. It’s as if we had invented a “calculator” that was purely a record of the results of our arithmetic, rather than a machine that performed calculations. Sure, a record of our computation is helpful, but it’s hardly an effective calculator. It almost feels disingenuous to say that this current generation of tools help us think. I think it’s more accurate to say they encourage us to think with the promise of perfect recall, but we are not yet living amongst tools that truly help us think better thoughts, faster.

So, what are the building blocks of a powerful thinking medium that can actually help us think, more than just recall? For a tool that has such broad access to information like a web browser, I think a critical piece of the puzzle is better machine understanding of language.

Most existing tools and browsers treat web pages and pieces of notes like complete black boxes of information. These tools know how to scan for keywords, and they have access to the metadata we use to tag our information like hashtags and timestamps, but unlike a human, most current tools don’t try to peer into the contents of our notes or reading materials and operate with an understanding of our information.

With ratcheting progress in machine understanding of language, I think we have good high-quality building blocks to start building thinking mediums and information systems that operate with some understanding of our ideas themselves, rather than simply “this is some text”.

Take my custom browser extension Revery as an example. I’ve been using it as the testing ground for some of my own ideas about what these tools should be able to do. While it doesn’t take advantage of any of the more recent advancements in NLP, its current capabilities are still changing how I browse the web. For any webpage I visit, Revery currently:

Scans the page and surfaces information from my personal search index that might be topically relevant to what I’m reading. For example, on an article about South Korea’s economic and cultural ascent, Revery surfaced my bookmarks on parallel stories from Taiwan and Singapore, letting me contextualize the current article.
Provides a quick extractive summary of long articles for me to scan before I decide to spend time reading the rest of the page.

In addition, Revery could in the future pick out important keywords and topics from the page and automatically search my search engine for them, or help me spot key people and places I know that appear on the page.

Because all of these experiments are grounded in my web browser rather than any particular application, these tricks and workflows work on any website, including other notes applications. I could visit my Ideaflow notes or someone else’s Roam graph and take advantage of these capabilities of my browser just as easily. It’s not that far-fetched to imagine a scenario in which I visit a well-connected Roam graph, and realize that my browser has made just as many connections between their notes and my notes as the author of the Roam notes have across their information.

If we were to build a medium for better thinking on top of the web browser, it’s reckless to expect the average user to manually connect, organize, and annotate the information they come across. Just as the early World Wide Web started out manually-curated and eventually became curated by algorithms and communities, I think we’ll see a shift in how individual personal landscapes of information are curated, from manual organization to mostly machine-driven organization. Humans will leave connections and highlights as a trail of their thinking, rather than as their primary way of exploring their knowledge and memory.

In the browser of the future, the boundary between my personal information and the wider Web’s information landscape will blur, and a smarter, more literate browser will help me navigate both worlds with a deeper understanding of what I’m thinking about and what I want to discover. It’ll remind me of relevant bookmarks when I’m taking lecture notes; it’ll summarize and pick out interesting details from long news articles for me; it’ll let me search across the Web and my personal data to remember more and learn faster.

A web browser for thoughts, not documents or apps

The web browser has advanced remarkably far in the first couple of decades of its history. But despite the technical achievements, it staunchly remains a pure utility, a tool meant to be used almost exclusively for visiting what lies at the other end of URLs.

There’s a renewed attention in the web browser space today. Too many of them are simply focused on making existing browsers faster and more powerful, but there’s an exciting small clan of founders and engineers trying to make the browser something more.

The vision of the web browser that excites me the most is one where the browser is a medium for creativity, learning, and thinking deeply that spans personal and public spheres of knowledge. This browser will be fast and private, of course, but more than that, this browser will let me explore the Web from the comfort of my own garden of information. It’ll break the barriers between different apps that silo our information to help us search and remember across all of them. It’ll use a deeper machine understanding of language and images to summarize articles, highlight important ideas, and remind me what I should remember. It’ll let me do it all together with other people in a way that feels like real presence, rather than just avatars on screen.

Forget space tourism – the most thought-provoking ideas, the most romantic stories, the most beautiful reveries of the future are right here, on the Web. There is still much to explore, and such beautiful worlds deserve a better spacecraft. One built to help us think new thoughts, together.

This post is a culmination of ideas sparked by conversations with Jess Martin, Raj Thimmiah, Jacob Cole, Molly Mielke, Karina Nguyen, and Josh Miller. Thanks to them for such enlightening conversations, and in many cases, for their own work in pushing these ideas forward!

The Generative Library

Wed, 28 Jul 2021 03:47:08 -0400

This is a short story about AI, which I would usually write and share on my second blog linus.coffee. But I liked this one enough, and thought it was interesting enough, that I’ve opted to publish it here, instead.

The town was called Aperia. A quiet but sophisticated sprawl of high-ceilinged atria and modest houses in a dry patch of land overlooking the Mediterranean. From a distance, Aperia could have fit in naturally to either the eighteenth century or the twenty-second. It seemed to live a little outside time, petrified into a timeless specimen like aged parchment under the warmth of the dry sun.

The streets of Aperia traced the outlines of a grid like in many other towns of this size, a main artery boulevard running through the most populous districts, with smaller, veiny roads extending through the neighborhoods into the outskirts. There was one road into the city, and one road out. They met at the mouth of the main boulevard, so that from high enough up in the sky the town looked like a leaf perched atop two branches that fused together at a point.

Neither of those roads into and out of Aperia were used much at all. The residents mostly subsisted on crafts of their own, exchanging groceries and lumber grown in the area for glass and textile fashioned within the same city borders. Occasionally, a mail carrier would arrive, a few letters in tow from the outside world. And on even rarer occasion still, I would see a scholar stumble into those quiet roads, eyes wide, as if they’d stumbled into a page out of a book of myths.

Those scholars, dear listener, are the subject of my story, for they were the ones I was bound to serve.

Such a scholar arrived only once every few years. I never understood why so few of them ever showed up, but I infer the journey here was simply too treacherous for most. The scholar would hardly seek a drink of water or a bench to rest before continuing on ever so deliberately down the city’s wide streets, passing shop after shop and home after home, in pursuit of one particular street, about three-quarters of the way down the road.

The street bent sharply right after diverging from the main thoroughfare, then ascended into the mountains at a gradual slope. The scholar would follow the path upwards as the green and brown of vegetation slowly replaced the white and beige of civilization, until they came to the end of the road, where the trail morphed into the mouth of a cave.

It was there that I greeted these scholars and welcomed them further into the cave, their steps trailing audibly behind my own. During the short journey further inwards, I would tell them the story of the Great Library. The Great Library, of course, was what lay within the cave – the library the scholars sought, and the library whose guardianship was my duty for seven hundred years.

The Great Library, like the town it neighbored, never quite had a defined origin. Some said it was created along with heaven and earth. Others told legends of Herculean architectural feats thousands of years ago that carved it into the mountainside. Yet others claimed the library was built by a secret society of monks who lived in the mountains and avoided contact with the outside world. Regardless of its provenance, this library was just like many other in most ways. It offered many books, from stories of fiction to records of the natural sciences. The diversity of stories it held and knowledge it harbored were unparalleled, putting even the great library in Alexandria to shame. Like most other libraries, visitors could query all the literature in the Great Library to locate the knowledge that they sought.

The main difference between this library and the others, as I would explain to every scholar, was the method by which they needed to query the library.

See, in an ordinary library, one would typically locate a book by looking through an index, a static record of all the books it held, organized by their common topics or themes. This was not so in the Great Library. Instead, at the end of the cave, there lay a piece of paper and an ink-dipped pen. The pen replenished itself of its ink by some mechanism known only to the Guardians of the Library, and the paper extended from a seemingly infinite roll.

To query the Great Library, the scholar would write their question on the paper. After a pause and a rumble through the walls, a shelf would appear in front of the scholar, filled with books that addressed exactly the questions posed moments before.

If none of the books answered their query to satisfaction, the scholar could then extend their query further on paper, after which the rumbling walls would provide the scholar with yet more books, every one of which addressed the exact query posed by the scholar, yet again. A question of science would summon encyclopedias of knowledge. A prompt to a story would summon a shelf full of exotic tales.

Hearing this, you may suppose that the Great Library was some elaborate magic trick pulled by a peeving owner of a great collection of books, seeing a scholar’s query and quickly sorting through their collection to find a handful of relevant literature to show off. This was not so, however. Such an operation would require an immense collection of literature and a hundred laborers sorting through it constantly, and no scholar who devoted significant time to understanding the magic of the Great Library ever found even a trace of such an operation.

No, the Great Library operated by an even stranger mechanism. I shall relay to you what was explained to me when I first arrived here.

The Great Library is neither the operation of a wealthy, clever bibliophile, nor an elaborate optical illusion. There is no great collection of books, no endless shelves of infinite wisdom. Instead, on every scholar’s query, the Great Library simply generates a hundred books of believable stories and plausible explanations prompted by the query. Exactly how it writes such books are not known to anyone – even querying the Library itself of such a question only seems to generate half-intelligible responses.

Upon hearing such an explanation, nearly every scholar dismisses my claim as ignorant pseudoscience, and will attempt to compose a query which will thwart the Great Library’s intelligence. They will ask of recent discoveries in the natural sciences, of their own childhood past, and even of potential future events. After each query, without an exception, every scholar would grow more and more agitated, driven mad by the uncanny powers of the Library, and would disappear, ultimately, into the depths of the cave. Those who do seem never to return.

Once every few centuries, a scholar would arrive who would come to terms with the Great Library’s powers, and, failing to comprehend it fully, simply accept it for what it is – a cosmic intelligence, an impossible mystery, something beyond the limits of their own cognitive reaches. Having come to grips with reality, they would then query the Library not only of their original question, which now seemed feeble and pointless, but also of other unattainable mysteries – of life and death, of the origin of the universe, of eternal satisfaction, and of endless youth.

Armed with such knowledge, they would vow not to return to their hometown, but instead to live on in the Library, guarding its powers and welcoming its visitors, however rare they may be. That’s right – those few who survive become the Guardians of the Great Library, of which I am one.

On the seven hundred and second year of my guardianship, during a quiet summer afternoon shift, I walked into the Library’s querying room and picked up the pen I had held so many times before. I began writing. And, as always, a shelf full of stories, each beginning with my exact query, appeared before me. I pulled the first one off the shelf, and began reading:

The town was called Aperia. A quiet but sophisticated sprawl…

Building Monocle, a universal personal search engine for life

Wed, 07 Jul 2021 02:30:28 -0400

Monocle is a full text search engine indexed on my personal data, like my blog posts and essays, nearly a decade of journal entries, notes, contacts, Tweets, and hopefully more in the future, like emails and web browsing history. It lets me query this entire dataset to look for anything I’ve seen or written about before, and acts as a true “extended memory” for my entire life.

As soon as it went live, Monocle quickly replaced the search field on nearly every other app I use, and became the first and only place I searched for information that I knew I’d seen before. Whether I was searching up someone I had met before, a new blog idea, or a scratch piece of note about how to do something with JavaScript – Monocle had it all, and got it to me in under 5 seconds from anywhere on my computer.

This is the story of how Monocle works, how I built it in a weekend, and how I’ve been using it.

A universal search engine

I first had the idea for Monocle almost exactly a year ago, when I tweeted about potentially building a search engine that only searches my private and personal data.

Thinking about building a "personal search engine"

A search engine that only indexes my blog, my Tweets, my journal, my calendar/email and contacts, my photos, and browser history.

I want to have better memory without having to remember more stuff. What else should it index?
— Linus (@thesephist) July 13, 2020

Since then, I’d thought about executing on this idea a few times, but got distracted or scared away by the potential complexity of the project each time. Then about a week ago, when I was writing about incremental note-taking, I realized that effective recall of information was critical to a good personal knowledge tool. This reignited my interest in this project. This past weekend, I took that idea and built a first prototype with Ink, teaching myself the basics of full text search algorithms in the process.

The headline description of Monocle is that it’s a full text search engine indexing only my personal data – I’ve explained the technical details of its architecture in the GitHub repository hosting the code. Here, I want to focus on a few specific design decisions I made in designing Monocle to make it as useful as it could be in my personal workflows.

First, I wanted Monocle’s time-to-first-result to be as quick as possible. By this, I mean that the most important metric of success for a search tool is how quickly I can go from some vague query I have in my mind, to looking through results to find what I need.

One critical constraint this imposed on Monocle’s design was that I needed to be able to search as I typed, with results arriving on every keystroke. Building many small fast tools has taught me that often, making something instantaneous doesn’t just make the tool more efficient – it changes how you use the tool. In this case, I believe searching as-you-type means the search progress becomes less of a slow question-answer cycle, and more of an interactive exploration through my knowledge base of typing a few words to see what information I have, perhaps deleting some characters, and re-typing. To make this experience possible, Monocle loads a compressed, pre-generated index of its document dataset on startup, and performs all search locally in the browser.

Like many of my other tools, the speed constraint also meant I wanted to be able to use the app entirely with the keyboard. The critical-path actions in Monocle – searching, scrolling through a list of results, and previewing them – are all a keystroke away. All of these design decisions together result in a search tool that can almost always help me find what I’m looking for in five seconds from anywhere on my computer.

Second, I wanted an indexer and search algorithm that I owned and understood. This is partly because I just wanted to learn and understand how full-text search engines worked, and the way I learn is often by building tools for myself using that knowledge. Another benefit of a fully custom, from-scratch search algorithm is that if I understand the whole system, I can be aware of its limitations and come back to improve it as I need the algorithm to improve. The “search algorithm” is really a few different parts:

the indexer, which catalogues keywords in the indexed documents
the searcher, which reads documents from the index to find matching results
the stemmer, which expands search queries to include variations of words like “tool” to “tools” or “create” to “creating”
the ranker, which is responsible for ordering search results by some measure of relevance.

In the future, if I wanted to add extra indexing or querying rules for categorizing results by their source type or date, or have other custom logic related to my data, understanding my own system would give me an edge over using something pre-made. Moreover, using something that’s only as complex as it needs to be for my needs means there are fewer opportunities for the algorithm to break, and more technical flexibility. For example, during development, I had to move the indexer from the client to the server, and my current architecture turned this challenge into little more than a copy-paste of a few lines of code.

I had a lot of fun building a full text search engine and writing custom modules for each of my data sources, but the underlying technology is not cutting-edge by any means – it’s just a full text search algorithm. The magic, as with many of my projects, is what’s possible when you apply such technologies to interesting data in a small, well-known stack that I control.

On privacy and ownership

When I wrote about the value of ownership before, I wrote on personal data ownership:

There’s one other benefit of owning the software services that drive my life. Without a doubt, the most valuable things I have these days is my data. My archive of notes, documents, photographs and music, todo lists, contacts, calendars – these collectively make up my external brain and my identity, which is the last thing I want to lose. I want to own data like this as wholly as I physically can. Storing them through online software services or apps are convenient, but I never know when a third-party notes app is going to run out of funding and shut down, or if a todo list or music app I use is going to pivot and stop caring about me as a user. These days, nearly every piece of data I own is stored and backed up on services and systems that I control from the operating system up.

As I added many of these data sources to Monocle, I realized that many parts of what makes Monocle special are only possible because I own my data and software stack to such a deep extent.

First, Monocle indexes some very personal and private data, like my journal entries. If Monocle was some third-party service from a budding startup, my journal entries would be the last thing I’d want to give up. But because I own and understand the entire search stack, I feel safe letting Monocle index such personal data. As a consequence, the search results I can get back are more personal and significant than anything extant search software could find for me.

Second, because I own my entire data stack, I can have certainty about where much of my data like contacts, notes, and archived journals are stored, and in which format they are saved. This means that writing and maintaining modules to ingest and index these data sources is pretty simple. I don’t have to worry about these data sources going away, or these sources changing their data export format. I simply talk to my file storage system and get back exactly the files I want. I feel that owning my stack for most of the data I needed to index made this problem much more tractable.

An experience report, so far

Monocle has only been live for a few days now – it went live on Saturday for me, and I’m writing this on the following Tuesday. But in that time, I think I’ve searched Monocle at least as many times as I’ve searched Google. In terms of how often I expect to use a tool I built, Monocle ranks in the top five best investments of my time across all my side projects.

In these few days, here are a few things for which I searched Monocle.

A few specific people whom I had met before, but didn’t remember very well
Topics I know I had written about before, to find links to blog posts that I could share
A few companies and projects that I knew I had heard of before, but wasn’t sure from where
Book recommendations that I had stashed in a few different places in my digital life
A few Twitter threads that I knew I had shared before about building side projects and communities

When I want to learn something new, I still search the Web. But if I’m trying to remember something from sometime in my past, I now go straight to Monocle, because it’s a more effective and powerful “extended memory” than any note-taking system I have ever used – I don’t need to enter or explicitly save anything; if I ever wrote it down somewhere, it’s here, indexed by Monocle.

In short, Monocle is the closest thing I’ve ever experienced to Vannevar Bush’s Memex. The more I use it, the more I want it to become even more powerful and even more present in my life. And because I own the whole stack, I can do so fearlessly, without any data privacy concerns. In the coming weeks and months, I hope to index other data sources like my browser history, calendar events, and perhaps even my email archive.

While I was building Monocle, I also had a more subtle insight about building my own tools. Monocle isn’t a standalone project by itself, but a cross-cutting infrastructural project that integrates with a wide range of other projects in my portfolio to provide some shared functionality: search. This is my second project like this, after Noct, which provides a shared storage and backup layer for data across many of my apps. After building Noct, backup and file sync was no longer an issue I had to address for each new project. After Monocle, it seems like search also became something that I’ll never have to address again for each individual project. These concerns are addressed universally by infrastructural projects on top of which individual apps can be integrated.

I like this idea of building pieces of dependable, shared infrastructure for cross-cutting concerns across my personal tools. It feels a little bit like building my own personal cloud platform, where I can build small apps that integrate with Monocle and Noct and other pieces of infrastructure and get those extra shared pieces of functionality “for free”. In some ways, building my own programming language and UI framework also give me the same benefit of shared infrastructure. The natural next step in this direction is to build other shared pieces of infrastructure like a universal authentication layer. A “personal cloud” sounds ambitious and grandiose, but it really feels like I’m putting together the right elementary pieces here to run towards that vision.

Writing and thinking about all of these ideas about personal information and recall, I also know that I’m in the vanishingly small minority of people on Earth who have their digital data saved and organized well enough to build a tool like Monocle. For most people, their important information lives across so many different pieces of software across a range of companies and technologies that building a personal Memex would seem impossible. In addition to building something useful for myself, I also want Monocle to serve as a kind of proof-of-concept – a statement piece, perhaps – about what’s possible when we build our own tools and manage our own information. Perhaps with the knowledge of what’s possible, we can push ourselves to draw where we are today closer to where we should be: a universe of small, personal tools that can respect our privacy and agency while leveraging it to help us see the world in new light.

Note-taking in the wild: living with the Surface Duo

Mon, 28 Jun 2021 11:50:59 -0400

I don’t normally write about new gadgets when they come out, but once in a while, a new device captures my imagination and creativity so much that I can’t help write a little bit about how I use it, and why it gets me so excited. The last time this happened, it was about the 2018 iPad Pro which I still use daily. This time, it’s about the Surface Duo, a dual-screen folding phone-tablet hybrid from Microsoft running a funky crossover of Google’s Android with Microsoft’s services.

Most of my thoughts about the Duo will be from the perspective of someone who reads a lot and takes a lot of notes on their mobile device. I’ve got a bunch of big screens at home for me to watch videos, listen to music, and do everything else computers are good for. But when I’m out and about in the city and need to burn some time or write down a fresh idea, usually all I’ve got with me are my phones, and in those use cases, the Surface Duo delivers.

Yes, I said. “phones”, because I carry two phones: an iPhone and an… other phone, which is usually a more quirky Android phone. This actually turns out to be important, because the Duo isn’t quite polished enough to become your only device in your pockets, but as a secondary phone that can sometimes become a better pocket computer, it’s great.

Reading and writing with the Duo

I’ve carried the Surface Duo in my pocket for about a week now. It’s much too early for me to have any definitive opinions about the device as a daily carry, but it’s enough for me to have discovered some fun and genuinely useful things I can do with the Duo that I couldn’t before.

The first change I noticed in my day-to-day life is that I read a lot more when I am outside with the Duo. This is almost entirely because of the fact that I can unfold the phone’s two halves into a large, combined screen that feels like the size of a small paperback book. While reading on a phone feels so constrained that I only ever want to do it if I have no other options, I’ve found myself reading on the Duo’s unfurled screen even at home on the couch, as well as when I’m sitting outside or just waiting in line.

The Surface Duo makes reading long-form online content enjoyable, rather than uncomfortable like on small phone screens. When I wand to read something less ephemeral, I can turn the tablet horizontal and open up a Kindle book to stretch across the phone’s two screens perfectly.

In addition to reading more, I’ve been trying to get better at remembering small things that escape my mind in the day-to-day by writing things down. Before the Duo, I would just open up a notes app and dump my mind there, so I could come back to it later when I was sitting in front of a proper computer. I can’t say the Duo changes this workflow dramatically, but with the Surface Pen stylus and OneNote, I’ve found it much more fun to jot down notes in the moment, when I’m outside. As a result, I’ve been taking more notes after conversations with people and remembering more of my random ideas on subway rides. This seems to be a case where technology is better by motivating me, rather than by enabling something totally new.

So far, I’ve talked about using the Surface Duo’s two screens together as a single giant screen to read and write. But if you visit Microsoft’s marketing page for this device, you’ll see there is another huge use case for these two screens – two-app multitasking. I can have one app (say, a web browser) on the left screen and another app on the right (say, OneNote for scribbling down notes) and use them together. Though I do this less often than simply reading or writing, multi-tasking reading and note-taking like this was definitely the most “a-ha” moment for me with the Surface Duo, and I try to do it whenever I get a chance to sit down outside. It seems like a use case that just makes sense. Anytime I need to read and take notes on what I’m reading, whether it’s a long email, a book, or a blog post, this two-screen setup is infinitely better than juggling two apps on a tiny iPhone screen.

Here’s my exact setup (in a funky, dual-screen screenshot). One neat trick I’ve learned to love is that you can select some text from a webpage or an app, and drag-and-drop it over into my OneNote page to record it and annotate it for later reference. I found myself using this feature all the time, partly because it’s so useful, and partly because it’s just plain fun to drag stuff across screens.

OneNote isn’t my main note-taking app, as you might know if you’ve read any of my other posts. But OneNote just so happens to be the app that integrates best with the Surface Duo because of the work Microsoft has done to make the hardware and software work well together, so whenever I’m on the Duo, I use OneNote as the “top of my funnel” for notes, and I copy them over into my “main” notes apps when I’m at my computer.

The astute among you may now be asking, “That’s great, Linus, but can’t you replicate this exact workflow by using two apps side-by-side on an iPad?”

Why, yes! You can. You can have a notes app on one side and a browser on the other side of a big iPad screen, and take notes this way. But I don’t carry my iPad around when I’m going for a walk or out for coffee. I carry my phone. If I have an iPad handy, I usually also have my computer nearby. The screen of the Duo is big enough that it’s replaced many of my iPad use cases. With the Surface Duo, I’m starting to read and note-take much more when I’m outside (which is often these days) in an Uber or during commute to and from places.

I’ve referenced Christine Dodrill’s blog post on their note-taking workflow before, using small notebooks and a pencil. They write:

Get a cheap notebook or set of notebooks. They should ideally be small, pocketable notebooks. Something like 30 sheets of paper per notebook. […] Start a new page every day. Put the date at the top of the page. […] And then just write things in as they happen. Don’t agonize over getting them all. You will not. The aim is to get the important parts. If you really honestly do miss something that is important, it will come back.

Though my Surface Duo doesn’t have all the benefits of paper, I think this pocket-sized notebook form factor is the perfect one for jotting down quick notes in this style. (And being digital has its own benefits, like automatic backup and sync.)

A pocket computer

In general, the Surface Duo doesn’t really feel like a phone. I never initiate calls on it, because it kind of looks ridiculous to hold a baby-size tablet up to your face. But I’ve taken a couple of calls on it, and when it rings and vibrates, I feel pretty silly trying to fish this thing out of my pocket and unfold its hinge all the way around so I can hold up the right one of its two screens to my face. Yeah… it’s not a great phone.

I would instead describe the Duo as a true pocket computer, a big screen that fits into your pocket. Especially when unfolded, the Duo feels a lot more substantial than most smartphones, because it fills up your hands and demands that you do something useful with it, instead of just scrolling through Twitter or checking the time. On the other hand, to do anything at all with the Duo, you need to first unfold the thing. There is no secondary outside screen on the front of a closed Surface Duo, like with some other foldable phones. This means I’m much less prone to just idly wake the phone to check the time or scroll through notifications, if I don’t have a legitimate reason to use the phone. All of this together makes the Surface Duo feel less like a distracting phone and more like a small, underpowered computer I can bring with me in my pocket.

The future is still pretty rough

At the start of this post, I noted that there are a few reasons why having the Duo as a second phone made sense, but using it as my only phone wouldn’t work. One of them is what I described above – it’s not the best telephone. But the more pressing reason is that despite Microsoft’s best efforts and series of software updates, the software experience on the Duo is still rough and buggy, even compared to other new Android devices. There’s no way around it – you’ll have to suffer the consequences of being an early adopter.

The Surface Duo makes you use swipe gestures around the screen to move and position apps around its two screens, but about 20% of the time, those gestures don’t quite register for me, and it takes a couple of tries. The automatically adjusted screen brightness occasionally gets blindingly bright in dim lighting. Unlocking the device and opening apps sometimes takes too long, to the point where I occasionally doubt myself and tap on the screen again only to have the device do something unexpected in response.

Hardware-wise, the Duo also falls short in many places. The camera is just plain bad – I haven’t taken a single picture on it, in lieu of the iPhone XS I usually also carry. I could see the foldable form factor being useful for the unexpected video call, but for those I’m usually at home or in an office .Battery life is good, but not stellar, and the speaker (the singular speaker on one side of the fold) is also pretty anemic. It doesn’t help that the processor inside is also a last-generation chip, driving two displays rather than one. I suspect this is a big part of the reason why the UI often feels sluggish.

To Microsoft’s credit, many other parts of the device’s hardware are fantastic. The two screens aren’t perfectly calibrated to each other, but close enough to rarely be noticed, and they’re colorful and high-resolution. The device is designs and built extremely well, as a glass-and-metal sandwich that’s just a half centimeter thin when unfolded – barely thin enough to have a charging port. It’s also weighted with a good balance, so it’s comfortable for me to hold it in any orientation I need.

My favorite digital-notebook-computer

To me, the foldable form factor and the slightly wider screens, combined with the lack of an outside screen, make the device feel less like an overpowered phone and more like a pocketable computer or a true digital notebook. One that respects my focus when I don’t have it open, and gives me space to jot down ideas and doodle when I want to capture something while I’m on the go. It’s also slowly but surely becoming an important part of my incremental note-taking workflow – many conversations, ideas, and inspirations from readings get written down first on the Duo these days, and then get moved into their right place in my notes.

Despite its flaws, the fun I’m having while reading and writing with this foldable pocket computer, and the promise of things that are possible when these technical kinks are fixed, makes me really excited about keeping Surface Duo around in my pocket for the foreseeable future.

Incremental note-taking

Sun, 27 Jun 2021 03:48:41 -0400

I’ve been delving deeper into the vast and strange world of knowledge organizing tools (notes apps, contact organizers, personal search engines). During this rather abstract expedition, one of my goals has been to emerge with some opinionated thesis about the way these tools should be designed to harbor and extend our knowledge effectively.

Though I’m hesitant to say I’m there yet, I’ve found myself repeatedly coming back to a group of related ideas I’m going to call incremental note-taking about how to best gather knowledge into notes, and how we should design tools built around this workflow. This post is one attempt (of hopefully many more) to share them with you. This is a longer post, so here’s a roadmap. If you’re impatient, I suggest you begin with the principles.

Good notes should behave like memory – a story of how I arrived at the ideas in this post
Principles of incremental notes – my current beliefs about how to build a good note-taking system
Tools for incremental note-taking – how existing tools support (or don’t support) taking incremental notes, and some insights into the tools I’m building for myself today

Good notes should behave like memory

At its best, a good collection of notes is like a powerful extended memory. It helps us quickly answer questions like, “What did this person tell me?” “What should I remember about this topic?” “What things am I currently working on?” We think tens of thousands of thoughts every single day, and the job of a good note-taking system is to help us make the most of them, even when our squishy, biological brains can’t.

Curiously, most of the thoughts we think are not what traditional note-taking apps consider “important”. The mind is like an iceberg: most of our everyday thoughts go unnoticed. By this, I mean: Most productivity solutions focus on the 10% of our thoughts that are easy to categorize and structure, like lecture notes, meeting minutes, people’s contacts, and highlights of readings. But the vast majority of thoughts we think – the other 90% – still hold underrated, underestimated latent value. In these 90% are things you pick up in conversations, only to forget by the next minute. These are the shower thoughts and ideas that slip past you so elusively, and to-dos that you let yourself forget because they’ll come back if they’re really that important. Without the right tools, our minds are hopelessly leaky. We forget much of what we think.

In an ideal world, we won’t have to forget things from our minds and workspaces. We could live in an infinite room for thought:

What if, on a single sheet of paper that lasts an entire lifetime, you could inscribe every thought you’ve ever had? It would be the written version of Jess’s infinite room for thought. Every idea you have would have a place here. In a perfect world, when you stumbled across a new idea that relates back to a previous memory, you’d simply take a pencil and draw arrows from this new idea all the way back to the ideas that came before. In this way, we’d construct an infinite transcript of our thoughts that was our life’s canvas for ideas. This infinite notebook would reflect the way we learn – we would connect related ideas together to trace out a web of memories, and label and sort them for future recollection.

This kind of an “infinite room” or “infinite paper” behaves like our memory. Our memory doesn’t ever really run out of space (though some ideas fade out over time). Our memory is also much less selective about what is remembered than most of our knowledge tools. We don’t remember things because we’ve somehow deemed something worthy of remembering, we simply remember things because they’re remembered, because they happened, because they stuck in our minds. I think a great note-taking system should inherit these properties of memory to properly extend it.

An age-old note-taking method that preserves these characteristics of memory is to carry a small notebook with you wherever you go. Christine Dodrill writes in the linked blog post:

Paper is cheap. Paper is universal. Paper doesn’t run out of battery. Paper doesn’t vanish into the shadow realm when I close the window. Paper can do anything I can do with a pencil. Paper lets me turn back pages in the notebook and scan over for things that have yet to be done. Honestly I wish I had started using paper for this sooner. Here’s how I use paper:

Get a cheap notebook or set of notebooks. They should ideally be small, pocketable notebooks. Something like 30 sheets of paper per notebook.

Label it with the current month (it’s best to start this at the beginning of a month if you can). Put contact information on the inside cover in case you lose it.

Start a new page every day. Put the date at the top of the page.

[…]

And then just write things in as they happen. Don’t agonize over getting them all. You will not. The aim is to get the important parts.

In this method, we take notes with a pencil in small handheld notebooks, each labelled with a month. We note things down and cross things out over time as ideas occur to us in the course of days and weeks. Paper notes created by this workflow aren’t some evergreen, digital garden, so much as a record of our thoughts and actions in life. No need for “edit history” here – the history is alive in between pages of crossed-out and postponed tasks and ideas.

Though I don’t personally have a pencil-and-paper workflow, I can see the appeal of this kind of a note-taking system. It records your thoughts over time – how they change, where they came from, when they came to you, and the context in which you had them. Just like the way we remember things in time, these notes improve by growing incrementally, with each new line and entry. Old notes become outdated, but are never replaced. And recalling past ideas is as simple as flipping through the pages to go back in time.

For some reason, when we moved our workflows into the digital realm, we began to lose respect for this way of taking notes, of simply adding new information to an ever-growing log of our thoughts. Instead, we built tools that encourage us to keep only the most current version of reality. Popular tools like Notion and Roam Research are about maintaining a timeless web of ideas, but life is anything but timeless! Old guards like Evernote feel much more like well-curated collections of notes over time, but it’s so difficult to organize and connect ideas in those apps that they quickly become black holes, where notes go in but rarely come out again. Most notes apps these days don’t lead us to collect notes, so much as simply keep them up to date. In that transition, I think we’ve forgotten the power of keeping notes over time, and remembering our past through our old notes.

Incremental notes is my push against this trend of note-taking tools that only live in the present and deny the reality of learning and living through time. We don’t remember things by modifying our past memories – we simply accumulate more, as if adding entries to a log or a journal. We search through them by traversing time, looking for links between ideas and experiences. These are the principles from which I want to build tools that augment our minds. With such tools, hopefully, we’ll be able to make more of the 10% of our ideas we’ve already retained, and hold on to much more of that lost 90%.

Principles of incremental notes

When designing something as complex as a note-taking system, I find it useful to lay down a few ground rules, the “principles” of the domain, to help make the right trade-offs. I’ve condensed my principles of incremental note-taking into four big ideas.

Captured ideas are better than missed ones. No self-respecting “note-taking system” should ever allow an idea to escape our minds un-recorded because it took too long, or was too much of a hassle to write it down. In order to make the most of the invisible 90% of our ideas that float through our minds, we need a tool that can capture ideas in the moment, however fleeting. This means our tool has to be fast, and can’t burden you with questions like “In what folder should I put this?” that aren’t relevant in the moment.
Adding new ideas is better than updating old ones. When our notes become outdated, our natural instinct is to go erase what’s now incorrect and fill that blank with the new information. But in that rewrite, we lose all of the original context we could have remembered about the history of our idea. Updating notes in-place is inherently lossy, and I think it’s unnecessary. Very often, it’s useful to have a record of our processes – how we came to some understanding, how we learned something through experience, how our relationships with the people close to us have changed over time. One of my favorite things about keeping handwritten notes is that the history of my thoughts are right there, next to my latest and greatest ideas. If we simply erased our old notes every time our understanding of the world changed, we would quickly forget how we got here. Just as our memory grows by remembering new things rather than “updating” old memories, our notes should also grow by incrementally gaining new knowledge, rather than replacing old valuable ideas with more recent ones.
Ideas that can’t be recalled are worse than useless – effective search and recall form the soul of great notes. Apple Notes (the notes app that comes pre-installed on all iPhones) is probably one of the most widely used knowledge capture systems in the world. Nearly everyone I know who owns an iPhone uses Apple Notes for something in their lives. The tragedy of Apple Notes is that it’s an idea black hole. Most of what goes into the app never leaves it again, because people rarely remember what they need to recall, and searching for the right things in an Apple Notes collection is tough. Of course, keyword search is not the only way to recall notes. Many tools these days have time and location-based reminders, as well as references and backlinks to connect related notes together into a graph. Regardless of how you recall information back from your notes, a great note-taking system should make it trivial to get ideas out, as well as in.
Time is essential to how we remember, and should be a first-class concept in a good note-taking system. The moment in time when we learned or thought something isn’t just some arbitrary metadata, it’s a mental anchor we use to remember nearly everything. We use daily and weekly planners, divide up school into semesters, plan engineering tasks into two-week sprints… time is absolutely essential to remembering what we learn. Whatever tool we use should recognize this, and help place our knowledge in the context of time.

Taken together, these principles of incremental note-taking lead us to a note-taking system designed less like a place you must “move into” with all your past notes slung behind your shoulders, and more like an extra layer of memory you grow around yourself, incrementally and gradually over time.

Tools for incremental note-taking

I’m at the very early days of putting these ideas and principles to use, but I want to share my perspective on existing note-taking solutions through the lens of incremental notes, and a few experiments I find interesting that put some of these principles to use.

Many of the current crop of popular note-taking tools, like Notion, Roam Research, Obsidian, and whatever you use at work (probably), are about helping you build a snapshot of your world as it is today. Some of them put some of these incremental note-taking principles into practice, but few of them honor all four faithfully.

Notion

Notion is probably the worst offender of them all – calling Notion an effective “note taking” app that extends your memory would be charitable. Notion is great at what it does, which is helping everyone easily create a shared web of documents that look and feel great. But it is not a note-taking app. It’s too slow to capture every thought I have. Its search is rudimentary and rarely helps me find the one thought I’m looking for. And it’s not designed to be used to recall thoughts from my past self. As far as I can tell, it’s primarily designed to act as a source of truth for a team. You can bend Notion to do most of these other things, but the result is slow and unergonomic. When you have just a moment and your idea is about to slip through your mind, you probably aren’t going to open a new page in your Notion workspace to add a quick note.

Roam (and others)

Roam and its clones fare much better. Roam is designed to help you incrementally build up a connected, sophisticated knowledge graph of ideas. It doesn’t force you to figure out exactly where to place every idea you record. Instead, you just write things down, perhaps on a “Daily notes” page, and connect each thing to other related things. If Roam can become and stay fast, I think it’s a promising platform for taking incremental notes. But Roam’s notion of time is weak at best – each day is treated as just another “thing” in a Roam graph of notes, rather than a first-class concept around which the tool is designed. In Roam, my thoughts don’t live “in time” – moments in time are just a special kind of idea. It doesn’t really make much sense, and I think this is a place where Roam has stayed too axiomatic for its own good.

Mira

The first tool I personally built that embodied the incremental note-taking principles is probably Mira, which I still use a year later as my primarily “people notes” app. I built it to replace my ever-growing mess of a note about everyone I knew and wanted to keep in touch with. Mira is fast – it often loads fully in the time it takes for Notion to start showing its loading spinner. After a conversation with someone, I always go back to Mira and add a few things I remember from the conversation, marked by the day’s date. When I open the app, Mira will show me people I’ve spoken to most recently, based on the conversations I’ve recorded. This means Mira is aware of time. In Mira, I rarely ever delete something from my past notes. Rather than removing “works at GFC” and replacing it with “works at Ideaflow”, I simply add a new entry: “Spoke at a tech dinner in New York, now working at Ideaflow”. In this way, Mira is a collection of notes grown incrementally over time. It describes a world changing through time, rather than a snapshot of it today. Lastly, Mira has some structured and free-form search. It’s lacking, but hopefully improving soon.

Ideaflow

Ideaflow, the note-taking software I help build at work, also embodies many of these principles. In fact, Ideaflow is my current “main” notes app. Ideaflow’s main interface is a timeline of notes, what we’ve internally called your “thought stream”. Imagine a long Twitter-style timeline, where each tweet is a note of some arbitrary length, potentially linking to many other notes. Many of these notes are short and simple, like random ideas or interesting links I read on the subway. There are even little facts I probably wouldn’t have put in my notes in most other apps, like transcripts of important emails and a list of publicly accessible bathrooms around the city. These are the 90%, underneath-the-surface thoughts that don’t really belong anywhere specific. Of course, there are also detailed, long-form notes like plans for conference talks, a folder of potential blog topics, records of conversations, and project ideas. Between my 980 notes today, there are around 1250 connections linking people to conversations, companies to investors, and ideas to their progenitors and other ideas inspired by them. So in a sense, incrementally, over time, Ideaflow helps me built up a sophisticated knowledge graph too. But in Ideaflow, time is a first-class citizen. Rather than a haphazard web of connections and words, notes go neatly into a timeline, grouped by days and weeks. This organization works together with my natural memory to help me remember things in units of time that I already use to understand my life. When I learn something new, I simply push another note onto the top of my timeline of notes, perhaps something connected to an older idea. Over time, this web grows denser and more populous like a forest of ideas growing around my life. I’m biased, of course, but I’ve found what I’ve been using so far to feel like a true extended memory, more than simply another database of facts.

Inc(remental)

Most recently, this week I began hacking on a tool called inc (short for “incremental”), a minimal notes app that delivers only the features promised by the principles above, and little else, in a small command-line driven package.

Inc is an experimental, append-only notes app. This means you grow your notebook by adding information to existing notes, or adding new notes; never modifying older ones. This approach to taking notes feels strange at first. Why would we want a notebook where we can never update our notes? What if something about the world changes?

Rich Hickey, in his talk about the design of the Datomic database, gives us the answer:

If my favorite color was red and now it’s blue, we don’t go back and change the fact that my favorite color was red to be blue – that’s wrong. Instead, we add a new, updated fact that my favorite color is now blue, but the old fact remains historically true.

In other words, this database (like our memory) doesn’t update information by forgetting what was once true, and overwriting it with the new fact; instead, it simply remembers that the fact changed at some point in time. Using this approach, we can have a notes app where we only add new information, and never delete old ones. (With this approach, it’s also obviously extra-important for our tools to understand time.)

Inc is currently just a command-line utility with a few commands:

+ Some note about #ink adds the note “Some note about #ink” to my notes. #ink is a tag I might use to search through my notes more effectively, but it has no special meaning beyond showing up in a different color in the app.
/some keyword searches my notes database using the keywords, and gives me a numbered list of the matching notes. Because the results are numbered here, I can then take another action:
@12 Goes to Stanford adds the information “Goes to Stanford” to note number 12 returned from my previous search.
Typing history shows us the full edit history of my notes database. In Inc, the way my notes came to be today is just as important as the information it currently holds. Using the history, I can rewind my notes back to any specific day, or just remember what I learned at any point in the past.

There are a few other shorthands and commands, but this is the core of Inc. Thought of something? Write it down in seconds. Trying to remember something? Search for it immediately. Want to review and understand your notes? Sift through time with a full history of your notes. Inc is focused on quickly capturing what’s on your mind, growing a knowledge base incrementally around your life, and helping you understand your notes with a first-class concept of time.

Above is how I use Inc in practice, to manage development of Inc itself, captured in the form of an inc history output. The most visible parts are all the notes I add, but in between them are the quick searches I do to remember and keep track of my ideas, and the occasional history lookup to help myself remember what I was doing, and place myself in the right mental context where I left off.

Truth be told, Inc is a new project (as are Ideaflow and many other projects in this space), so my hypotheses about incremental note-taking and the way these tools work are only so strong. But as a good investigator should, I want to hold myself to these principles laid out here, build tools around the incremental note-taking workflow, and see where the ideas take me. Perhaps I’ll come to believe them even more over time. There’s also a good chance I’ll correct myself, and look for a new thesis. Regardless, I’m excited by the vast design possibilities we’ve yet to explore in this space of building tools that embrace and extend the way our minds make sense of the world.

Thanks to Jared Pereira, Jacob Cole, and Jess Martin among others for the many conversations with me that have led, sometimes through long winding paths, to my thoughts in this post.

Liquidity of skill

Wed, 16 Jun 2021 19:11:35 -0400

Here is a mental model I share often with friends looking for job opportunities: your ability to get a job is the product of two quantities, the value of your skill and the liquidity of your skill.

The value of a skill is the fundamental value you add by applying your skill to a task. It is the difference between a task done with your skill, and a task done without. If I need to build an app to solve a problem, and I can use my programming skill to pull it off quickly and produce a good product, I’ve added some value to this task by being good at programming. That difference that I made is the value of my skill.

The liquidity of a skill is how easily and readily you can convert your skill into some other kind of useful resource, like money, social capital, or influence. If I’m only a mediocre programmer, but there are 10,000 people who know exactly how good I am, I’m going to be able to make more money and get hired quicker than a world-class programmer who anyone barely knows.

Some skills are inherently more liquid than others, but there are ways for everyone to make their skill more easily convertible to money or influence or whatever else you desire.

The ability to write software is of good (though not exceptional) value, but it is so unbelievably liquid right now. There’s a huge, growing, market with insatiable demand for good programmers, and the output of your skill can be scaled up so quickly and easily with just a good computer. This is why software engineers make lots of money. The ability to make great Korean food, by comparison, is not as liquid. Your work depends on real, physical ingredients that grow out of the ground, some well-equipped workspace, and real people visiting you to take the fruits of your labor from your hands. I don’t think this skill is any less valuable than my ability to write decent software. This skill is valuable, but illiquid. It’s like having a check for a million bucks that you can’t cash all at once.

It’s easy to make the mistake of conflating how much money you can make with how valuable your skill is. People think that being a doctor or a lawyer or an engineer is of fundamentally more value to society than being a chef or a musician, because they tend to make much more money. But the reality is that if one job makes more money than another, it’s generally not because that labor or skill is fundamentally more valuable, it’s just more liquid, more easily converted to money, or simply less replaceable.

Your ability to have a good career is the product of two things: the fundamental value and liquidity of the skills you have. So, when applied to job hunting, this means that there are really only two things that matter.

How good you are
How many people that influence hiring decisions know how good you are

All of the games people play to get an edge in hiring, like polishing resumes, practicing interviews, or going to networking events, are simply the popular ways of maximizing one of these two quantities. These small tactical pieces of advice can be useful, but I find it helpful to know what the ultimate goals are: to be good, and to have as many people know that as possible.

Most people focus too much on being good, and not enough on how liquid their skill is – how many people know how good they are. Many of my friends are better programmers than I am, no doubt. But I think I have a much less stressful time finding interesting opportunities because I sort of talk about it constantly online to tens of thousands of people and share what I make. Does this mean you should work off-hours or grind away on making a hundred side projects? No, obviously not, I’m just a pathological outlier. But think about how much time you spend improving your skill, going through school and honing your craft. Now think about how much time and energy you spend letting people know about what you do and why you do it. If that distribution of effort were more balanced, where would you be? What could you do?

Made of love

Thu, 10 Jun 2021 03:45:42 -0400

There are two ways to think about a multi-decade-long project, like building a company or growing a personal body of work.

The first way is to maximize value being created today. I describe this as building an earning machine. In this approach, you make decisions based on how much value you can pull out of your work. The most important thing is how much money or value your work can earn you. If you’re working on a side project, it’s important whether that side project can generate some revenue, and how much. If you’re building a company, you want to scale up and grow quickly. If you build something large enough that grows quickly enough, the thinking goes, it’ll probably last for a long time.

The second way is to invest in building something that can last the test of time, and continue to be valuable forever. In this approach, you are investing into an asset. The most important thing in this approach is not how much money or value you can pull out of the system now, but longevity – for how long, and how reliably, the value of whatever you create will hold.

In the first approach of building an earning machine, the value comes mostly from scale. An earning machine is valuable because it creates lots of value and earns lots of value in return, whether in money or popularity. In the second approach of building a long-term asset, the value comes from scarcity, and the fact that scarcity allows you to be unreasonably focused on quality.

Both approaches are valid ways of building something valuable and impactful.

We can compare the first “earning machine” approach to mass-market brands, and the second “asset” approach to luxury brands. Mass market brands used by hundreds of millions of people are optimized for scale and accessibility. They are valuable because they are big, and in reaching for that scale, they sacrifice some level of quality. This is partly because the price of quality is high (it costs a lot to make a luxury good), and partly because manufacturing hundreds of millions of things that all live up to a luxury standard of quality is extremely difficult. Luxury brands worry much less about scale. Their value comes from the quality of their craft. Ferrari and Chanel and Pagani aren’t as concerned about how many people can experience their work, and that scarcity affords them the money and time they need to reach for a much higher level of craftsmanship than mass-market products.

One interesting thing about these brands that define the upper echelon of luxury is that most of them trace their origins back to builders and craftspeople, rather than businesspeople. Very often, the brands bear the namesake of their progenitors – Enzo Ferrari, Coco Chanel, Horacio Pagani. It’s not hard to imagine that brands that began from the hands of expert builders (of cars, clothing, or whatever else) prioritize the quality of their craft more than brands built as enterprises. I think this different in priority for these brands comes partly from the attitude of their founders, and partly from the business culture that surrounds them.

I’ve wondered for a long time about what ultimately sets the products of these luxury brands apart. They aren’t always more aesthetically perfect, and are not always made of better materials – these are all matters of taste. But I think what sets these works apart, what defines craftsmanship, is the amount of love that went into producing the final product. Love is a fundamental ingredient of high-quality, well-crafted works and products. With it, we can build scarce assets that hold value over time.

When we make something we care about, and we put in the time to labor over every decision and detail, I think the end products come to bear unmistakable marks of that love. Creative process is full of uncertainty and setbacks and burdens that the maker must bear through. We often call the shadow of this shouldered burden “craftsmanship” or “attention to detail”. I think it’s these details, regardless of the particulars of fashion or taste, that define craftsmanship.

From the perspective of a maker, your ultimately scarce resource is the love and care you can put into your craft. There are only so many hours in the day, and so many hours I can put into making my work bear the marks of my love, and that makes these works of quality inherently valuable.

Love, I think, is also the necessary ingredient of any truly great long-term project. Whether you’re engineering a particular software feature, building a community, or designing a dress, over and over I’ve seen that there is no replacement for simply caring more than everybody else, and pouring your love into that process of creation.

There are two, related mindsets about creative craft here:

Treat a multi-decade project as a process of building a long-lasting asset rather than simply a way to get the most value at any given moment
Create something of unreasonable quality by investing your love into the craft

More and more, I find myself missing these two related mindsets in the world of technologists building with computers. I firmly believe, even in a world dominated by giant corporations of planet-spanning scale, there is a place in the market and in our hearts for things crafted with love poured into the labor of making something well-made and scarce, something unreasonably great. I think this approach ultimately leads us to build projects, communities, and companies that are not simply earning machines, but products of our care that last the test of time – things that are valuable because they are good, rather than good because they are valuable.

This is also how I’m choosing to think of the long-term project that is my portfolio of projects and writing. I don’t view any particular project as a way to build something that grows fast or sells for lots of money. I just want to build many, small, well-made things that bear the marks of my love, that serve as evidence that I cared about the quality of what came from my own hands. And in the end, I want to be able to look back on the whole thing and find it valuable because it is good, because of the hours I spent laboring over the particulars of my words and the pixels of my designs. Valuable, and lasting, because it is made of love.

Thanks to Chris Paik, Karina Nguyen, and Bat Manson for sharing some nuggets of ideas that inspired parts of this post.

Habitcrafting and tool-making

Tue, 08 Jun 2021 14:40:29 -0400

This is an excerpt from today’s issue of my weekly newsletter.

At work, we build a note-taking app called Ideaflow. As I talk to users and think a lot about how we use tools to organize information, I keep finding myself arriving at the same conclusion. A tool-making business is a habit-crafting business.

I’ve written before about how I think our tools are, and should be, simply reified workflows. They should be the concrete, touchable, visible versions of the workflows that our minds naturally tend towards. And what are the workflows that are essential to our lives but habits? The best tools become addictive and habitual in the best way. They become a part of the way we navigate life.

One of the things we’ve been talking about at Ideaflow recently is the idea of building a tool that becomes an “operating system for your life” – something that you can run your life on, so to speak. More than any domain-specific tool, a tool like this has to really become one with the habits and workflows that make up your waking hours. When we build Ideaflow, we aren’t so much building features, as we are crafting habits for our users to adopt and live by. When you need to remember something, we want you to open Ideaflow. When you need to think through a complex idea or take notes during a conversation, we want you to use Ideaflow as the scratchpad, a thinking-writing medium for thinking aloud on paper with your hands. Some part of this is building software, but I think a much more critical and interesting part of the challenge is designing these features so they could become habits.

Tools that support habits have a much higher bar to meet than simply offering useful features. They need to be fast, all the time – you shouldn’t have to wait for electrons to travel halfway around the planet for you to do something you do a dozen times every day. They need to be consistent – the buttons and gestures your muscle memory depends on can’t change all the time. They need to be easily within reach wherever you are – if not, they can’t become a meaningful part of your life.

If we do our job well, we won’t be selling an app, per se. We’ll be selling better habits for thinking and writing. It just happens to come in a software package.

In this way, building tools is a lot like building social media apps – the killer feature is that it becomes a part of your life, woven into your muscle memory and mental reflexes as intricately as the way you unlock your phone and scroll through tweets.

We’re certainly not there yet, but layer by layer, we’re laying the groundwork.

How to commit to the few right things

Mon, 24 May 2021 23:55:35 -0400

Since the start of the year, I’ve spent a lot of time speaking with friends and people I meet online about a common, difficult decision: in an ocean of interesting opportunities, how do I pick the few, right things that’ll take me the furthest? The good old “what do I commit to?” problem.

Some of these people are choosing whether to start their career at an established company or join an earlier-stage startup. Some people are wondering whether they should accept a generous job offer or continue looking for better opportunities. In these conversations and in my own experience, there are a few perspectives I’ve learned that seem to resonate with people.

If you’re any good at what you do, the opportunities you have around you are going to improve exponentially over time. There are a few feedback loops involved that make this so. If you’re good at what you do, there will always be people looking for people like you. Your network is going to grow, and as you get better opportunities and improve at your craft, your skill is going to grow, too. These all loop back on each other: being more practiced at your work attracts more people around you, and more people increases your chances of stumbling into the next great opportunity, where you’ll grow faster. In the end, you’re going to find yourself on a curve that bends upwards.

At any given moment in time, though, you won’t see the long arc that bends upwards in front of you or behind you. You’ll see a little tiny sliver of that curve that you sit on in that moment, and that small slice will feel relatively flat. So to predict what kinds of things you’ll do in the future, you’ll project that path forwards in time to months and years ahead in a straight line, not an upwards curve. Often people make commitment decisions based on whether the opportunities are better than where they expect to be in the future. And if they make the mistake of extrapolating their path linearly into the future, they’re always going to be faced with too many good opportunities.

One common approach against this problem is to jump around frequently. Do something for a short while, but be ready to jump ship when the next great opportunity comes along. Being flexible like this is a good way to remain open to options and get a wide breadth of experience, but it critically locks you out of the kind of long-term, compounding growth in experience, in skill, and in network that only comes when you can focus on a project or problem for years at a time. This is the paradox of choice: picking a pretty-good option and sticking with it long enough for the benefits to compound over time is better than picking the best option, if it means jumping around all the time.

So what’s the solution?

If you simply find good opportunities by projecting your current path forwards in time, most of the opportunities in front of you are going to seem great. But chances are, you’re not going to end up where you project. Most good people tend to underestimate their potential because they intuitively project their path forwards linearly, and if you follow an exponential curve upwards, you’re going to end up somewhere far above most of the options you’re presented with at any given moment.

The solution, then, is not to settle for opportunities that seem great, but wait for the ones that seem like true outliers – the once-in-five-year opportunities that are far above your obvious, linear path forward, but are probably on your exponential curve upwards into the future.

The catch about these outlier opportunities is that they are much harder to get than the ones on your linear path, and they’re much rarer. Despite this, in my experience, it’s worth the effort to be both patient and selective enough to commit deeply to these few right opportunities. In time, these experiences will be the ones that you’ll be able to stick with for many years to grow and change for the long term.