Introduction

Welcome to the documentation for Nolita! Nolita is a framework for building web-enabled agentic applications and workflows.

What do I need to use Nolita?

Before using Nolita you'll want to install Node.js, preferably version 20, and Google Chrome. If you want to work on the source code, you'll want pnpm too.

You don't need to install Nolita itself at all. You can run it directly with npx or import it into a Node.js project with npm i nolita. More information is available on the following pages on usage.

How does it work?

Nolita drives a Puppeteer installation using a local instance of Chrome and parses the accessiblity tree, preprocessing ARIA nodes to add additional accessibility information when necessary.

At the core of the framework is a state machine between Puppeteer and your model that enforces action steps with Zod.

In sum: "you tell the AI to do something on the internet and it does it."

In practice, Nolita allows for a large degree of granularity and extensibility, deploying agentic behaviour where necessary and stepping down into manual scripts or lower-level directives when you need to.

What do we mean by 'agentic'?

We consider agentic software to be software that can make decisions about its behaviour. For example, while a state machine normally restricts its inputs and outputs according to its state, it cannot reason about future inputs to give itself or outputs it should next derive in order to accomplish a higher-level goal.

In contrast, agentic software possesses the ability to evaluate its environment, consider multiple potential actions, and choose the most appropriate one based on predefined goals or learning algorithms. This decision-making capability allows agentic software to adapt and respond to dynamic conditions in a more flexible and intelligent manner than state machines.

How is Nolita different from [agentic framework]?

That being said, agents are an emerging art form. Agentic software based upon large language models to drive state machines are inherently probabilistic, and therefore not well-suited to production environments.

Nolita was written from the start to integrate with High Dimensional Research's Collective Memory Index. By working in concert, agentic actions are requested only for new and unfamiliar situations; if you imagine a webpage as a graph, then you can probably guess that most pages share structural similarities. We use a combination of accessibility tree preprocessing and graph comparison algorithms to then search for the most appropriate step in the browser state machine to execute, increasing the determinism and speed of the task, and leaving the agent to reason where it needs to.

What can I do with Nolita?

As a few examples, you can use Nolita to

gather structured data and chain it into other APIs, like sending an email or building an RSS feed;
quickly pipe information from the internet into your shell scripts, even from behind a log-in screen;
write and deploy Puppeteer scripts in natural language

Running tasks

If you want to test Nolita out quickly, you can do so on the command line by running

npx nolita

When running, you must provide a startUrl and objective. Before your first run, you will need to authenticate your model details and HDR keys with npx nolita auth.

If you don't include information, we will prompt you for it at runtime.

Flags

--startUrl dictates where we start the session.
--objective specifies what we want our agent to accomplish for us.
--headless specifies whether you want the browser to run in headless mode or not. We default to true, but you can set it to false to see the browser run.
--config takes a JSON file with the previous flags, if you want to provide them. You can also specify an inventory of personal data to use for the objective, like usernames and passwords. If you want to set per-task agent credentials, you can do so here and they will take precedence over nolita auth.

Optional features

--record will return the ID of the completed task session for later replay of the same actions, which you can use with the Page API.
--replay takes in a string of the ID above and currently just confirms that the route can be successfully followed. When using replay, objective and start URL are discarded.

Example configuration

{
  "agentProvider": "openai", // or process.env.HDR_AGENT_PROVIDER
  "agentModel": "gpt-4", // or process.env.HDR_AGENT_MODEL
  "agentApiKey": "sk-*********", // or process.env.HDR_AGENT_API_KEY
  "inventory": [
        {  
            "value": "student", 
            "name": "Username", 
            "type": "string" 
        },
        { 
            "value": "Password123",
            "name": "Password",
            "type": "string" 
        }
    ]
}

Using Nolita as a server

Since Nolita is written in TypeScript, it may not work for every pre-existing product. If you would like to use the same task-runner API provided by npx nolita for your own applications, you can run Nolita as a server.

npx nolita serve

This runs a local API for objective-first agentic navigation of a local Chrome instance.

After starting the server, you can see the /doc folder for the expected JSON payload.

Example payload

curl -X POST http://localhost:3000/browse \
      -H "Content-Type: application/json" \
      -d '{
        "browse_config": {
            "startUrl": "https://google.com",
            "objective": [
            "please tell me how many people edited wikipedia"
            ],
            "maxIterations": 10
        },
        "headless": true
        }
      '

Flags

--port will customize the port. By default, the server runs on port 3000.

Creating a new project

npx nolita create

Bootstraps a template application built on Express, React, TypeScript, and the core Nolita framework for making a user-facing, web-enabled, agentic product.

For reference, the template repository resides at hdresearch/create. Upon running the create command, the repository is cloned, its dependencies are installed and example files are instantiated.

Except one...

Please note that you will need to set up a .env file to use the application. You can copy and paste the example environment to modify it from there:

cp .env.example .env

What does it do?

The example application is extremely simple: it finds food from a specific location and outputs typed data, surfacing each step of the navigation in the console.

At the top of the App class in app/src/App.tsx you can start modifying the core objective:

  // You can change the URL to any website for the objective.
  const [url] = React.useState("https://www.google.com");


  const [objective] = React.useState(
    "where to find food in",
  );
  const [location, setLocation] = React.useState(
    "West Village"
  )

Tweaking these default values will then tweak what is requested from the server in app/src/lib/events.ts:

eventSource = new EventSource(
    `http://localhost:3040/api/browse?url=${encodeURIComponent(url)}&objective=${encodeURIComponent(objective)}%20${encodeURIComponent(location)}&maxIterations=10`,
  );

Which itself hits the server at /server/index.ts, running at port 3040 (in the dev environment) and taking in these parameters:

  const answer = await agentBrowser.browse(
    {
      startUrl: req.query.url as string,
      objective: [req.query.objective as string],
      maxIterations: parseInt(req.query.maxIterations as string) || 10,
    },
    CustomSchema,
  );

And returning each step back to the EventSource in as a callback to the browser's Logger:

  const logger = new Logger(["info"], (msg) => {
    return res.write(`data: ${msg}\n\n`);
  });

For more information, continue reading about the folder structure.

Using local models

Nolita can take both local and ollama providers for autonomous tasks. However, at present functionality is quite limited; we mark usage of local models as experimental, but available. We encourage you to experiment with your own workflows and see where delegating to local models makes sense.

Using Ollama

Ollama is supported as a provider and will connect to the default Ollama port when used. When you set ollama as a provider, the model name you provide to one that Ollama recognizes. For more information on available models for Ollama, see its documentation.

This means that we expect Ollama to be running when using Nolita with ollama as provider. You'll want to run Ollama's GUI or just ollama serve in a terminal window.

Using local model files

We use node-llama-cpp under the hood to run a model file. For information on node-llama-cpp and getting a model file for use, see its documentation.

Once a model file is downloaded, Nolita accepts the full path to the file as a model name when using a local provider.

You can set this, as usual, using npx nolita auth.

Usage and limitations

Smaller models (7B and below) are far less capable of holding enough context to navigate the web alone. They can, however, still process data well. In our current usage, we find that page.get() calls function consistently. Other calls, like autonomous browsing and navigation, can stumble on navigating the Nolita state machine. It is not uncommon to see errors about incorrect JSON from our generators.

You may, however, find better results with bigger local models; what models you can experiment with is a limitation of hardware.

Use Nolita as a scraper

If you're trying to just quickly gather data from the web, Nolita enables you to both dictate how much you want a model to autonomously search for data and what shape the data needs to come back in. These two features in tandem enable very powerful automation scripting. In this guide, which elaborates upon our findEmail example file, we'll walk you through what you need to do to get started.

Installation

Assuming you've installed Node.js (preferably the LTS), let's create a new folder for our script and put Nolita in it.

mkdir email-gathering && cd email-gathering
npm i nolita zod

NPM will make a package.json file and a node_modules folder for everything we need. Now let's just make our file and open it in your preferred code editor.

touch index.js

Necessary prerequisites

Nolita will want an applicable model set up before we start.

In your terminal, you may want to run npx nolita auth and input your model provider (and any applicable keys). If you prefer writing the configuration yourself, create .nolitarc in your home folder and fill in the blanks here:

{
    "agentModel": "", // use a model name from your provider
    "agentApiKey":"", // api key for anthropic / openai
    "agentProvider": "", //anthropic / openai
    "hdrApiKey":"" // optional
}

Imports and setup

We're going to need to import the Browser class and the makeAgent utility. The Browser class is the main navigation engine Nolita provides us; makeAgent is just a helper for making chat completion classes for our model without us having to think about it.

const { Browser, makeAgent } = require("nolita");
const { z } = require("zod");

Now we just write our main function, using those imports.

async function main() {
  const agent = makeAgent();
  const browser = await Browser.launch(true, agent);

Browser.launch() has two inputs: whether or not to run the browser headless, and the agent to attach to the Browser instance. If you want to see the browser open and autonomously navigate the page, change true to false.

Page actions

Now let's break down the next section. We'll use the browser constant here to make a Page class, corresponding to a browser tab, that we can manipulate and pilot:

  const page = await browser.newPage();
  await page.goto("https://hdr.is");
  await page.do("click on the company link");
  const answer = await page.get(
    "Find all the email addresses on the page",
    z.object({
      emails: z
        .array(z.string())
        .describe("The email addresses found on the page"),
    })
  );

We use .goto() to start us off at a URL, which instantiates our session. .do() gives a natural language directive to our model to navigate the page and issue a command to the browser. Please note that it's a single action, specific to the current page -- if you want to give a broader objective to the model, you'll want .browse() instead.

.get(), likewise, is specific to the current page. We provide it an objective and a schema. Our model will parse the page content and return the data in the shape we provide in the schema. This shape corresponds to a basic Zod object schema. You'll want to ensure you chain .describe() to let our model know exactly what you want.

In this case, we ask for an array of emails as strings inside a JSON object.

  console.log(answer);
  await browser.close();
}

main();

Adding it all together

Once it has the answer, it will write it to the console (making it pipeable to an arbitrary text file) and clean up the process.

In sum, our script looks like this:

const { Browser, makeAgent } = require("nolita");
const { z } = require("zod");

async function main() {
  const agent = makeAgent();
  const browser = await Browser.launch(true, agent);
  const page = await browser.newPage();
  await page.goto("https://hdr.is");
  await page.do("click on the company link");
  const answer = await page.get(
    "Find all the email addresses on the page",
    z.object({
      emails: z
        .array(z.string())
        .describe("The email addresses found on the page"),
    })
  );
  console.log(answer);
  await browser.close();
}

main();

We can run it by just running node index.js and even write the result with node index.js > emails.txt. If we try running the latter command, we'll see in our .txt file:

{
  emails: [
    'tynan.daly@hdr.is',
    'matilde.park@hdr.is',
    'SUPPORT@HDR.IS'
  ]
}

The folder structure

The default Nolita project uses the following folders to delineate each layer of an agentic application:

/agent includes the creation of a chat completion API, which is then passed to the Nolita Agent class for integration with the browser state machine.
/app includes all front-end code for the application.
/extensions includes the integration of inventories as well as defining custom types for your responses from the agent.
/server includes all back-end code for running the browse loop.

Inspiration

While working with other agentic companies, our research found the following separation of roles, which inspired the structure of the Nolita project.

LLM

What model is used on the 'bottom layer' of the stack?
Do you allow users to swap out underlying models?
System prompt is included here.

Agentic logic

What is the agent’s prompt on top of the underlying system prompt?
What’s the structure of the event loop we place an LLM in?
Does it self-iterate and improve? Is it set?
Is it a formal state machine, or is it entirely prompt driven?

Toolchain

How do we define what actions the LLM can perform?
Is the toolchain defined as part of prompt manipulation, or is it formally constrained (as in, proceeding along a graph of possible actions)?
- By going entirely prompt-driven, one can fall into “phantom actions” (reporting an action is undertaken, as opposed to making it such that saying an action is the same thing as the action itself.)
- Let me expound a little here: there’s a situation where by simply saying “I’ll invite this person” is itself part of a command to the toolchain, as opposed to a separate report to an observer. When writing entirely prompt-driven applications, one can hallucinate the structure of the action.
Do we write an underlying API to formalise all commands without directly hitting external APIs?

Event loop manipulation

Does the agent then “double check its work” before proceeding to presentation layer?
Do we GOTO 1 so to speak, if it performs incorrectly?

Presentation

How transparent is the stack to the user?
Is the agent abstracted as a “product” (with an agent’s artificial monologue puppeteering a conventional software stack), or anthropomorphized in its own right?
Is the user configuring tasks, agents, events?

/agent

The /agent folder of a Nolita project concerns itself with setting up a chat completion API and passing it to the Agent class.

Supported models

All OpenAI and Anthropic text-mode models are supported.

While we currently predominantly support OpenAI and Anthropic models, you can also write your own chat completion API for any model. For more information, see the Agent reference document.

/app

The /app folder bootstraps a starter front-end application built on TypeScript and React with Vite as its development server and build pipeline, and Tailwind as its CSS framework.

The sample application

searches a query on the internet ("where to get food in...");
returns each step of its navigation for your reference;
and finally returns with typed data, as defined in the /extensions folder.

Removing the application

If you want to bootstrap a Nolita application without a front-end, you can remove the app folder and the remaining calls to its build process.

In package.json, amend the scripts to

"scripts": {
    "server": "npx tsx ./server/index.ts",
    "start": "NODE_ENV=production npm run server"
  },

and in /server/index.ts, remove this remaining call to front-end code:

if (process.env.NODE_ENV === "production") {
   app.use("/", express.static(path.join(__dirname, "../app/dist")));
 }

/extensions

The extensions folder is intended for additions to the core agentic stack:

inventories, i.e. personal data outside the prompt context;
custom types, for specifying structured responses for the workflow

If you need to derive external information to pipe into either, it is also best done in this folder.

You could, for example, gather user session data to then derive which custom type to pass to the agent browser for the session. You could also, in addition, import user data from a database for use as an inventory while keeping all user data outside the prompt context itself.

For more information on inventories and how they work, see Inventory.

/server

The /server folder holds all back-end logic for the application.

We spin up all applicable classes in the Nolita framework, take the additions from the /extensions folder, and finally user input from the front-end application and pass an objective to the agent to perform on a local, sandboxed Chrome instance.

This section of the application is best-suited for including any additional API calls you need -- whether sending emails based upon the response from the agent, recording user objectives in your own database, or pre-processing input before incorporating the agent browse session.

Specifying types

We use Zod under the hood to enforce typed responses from the agent. You can use this to enforce predictable output for your application.

For example, in the example repository we include the following in extensions/schema.ts:

export const CustomSchema = z.object({
  restaurants: z.array(z.string().describe("The name of a restaurant")),
});

In this example, we inform the agent that we expect the response that reports a completed objective to also include an array of restaurants. When we finally get ObjectiveComplete back from the agent as an answer, we will also see restaurants as part of the response:

// inside the event response from the server`
export type AgentEvent = {
  command?: string[];
  done?: boolean;
  progressAssessment?: string;
  description?: string;
  objectiveComplete?: {
    kind: "ObjectiveComplete";
    restaurants: string[];
    result: string;
  }
  objectiveFailed: {
    kind: "ObjectiveFailed";
    result: string;
  }
};

Extending your application

You can most easily start tweaking your project by modifying the two pieces of state at the top of the App class in /app/src/App.tsx:

  // You can change the URL to any website for the objective.
  const [url] = React.useState("https://www.google.com");


  const [objective] = React.useState(
    "where to find food in",
  );
  const [location, setLocation] = React.useState(
    "West Village"
  )

and its appropriate code in the server, at ./server/index.ts:

// ...
app.get("/api/browse", async (req, res) => {
  res.setHeader("Content-Type", "text/event-stream");
  const browser = await Browser.create(true);
  const logger = new Logger(["info"], (msg) => {
    return res.write(`data: ${msg}\n\n`);
  });
  const agentBrowser = new AgentBrowser({ 
    agent, 
    browser, 
    // ...

You can then pass different objectives and start URLs to the backend, or pre-process incoming objectives with calls to APIs. You can pass steps from the Logger to external classes if necessary. You can also take the answer reported back from agentBrowser.browse and chain more actions onto it before returning to the user.

Extensions and inventories

Remember that you can always pass different pieces of personal data, i.e. inventories per user, whether stored in a database outside the template application or as another component of the application itself. By doing so, you can effectively customize agentic responses to the user as necessary.

Additional infrastructure

In production, you may find that one browser can't handle significant load. In such cases, an external API for spinning up multiple browsers on multiple machines may be necessary. Additional changes for Nolita are incoming for multi-tab browsing within one machine and for attaching white-label headless browsers in a future release.

Class: Agent

Constructors

new Agent()

new Agent(agentArgs: {objectgeneratorOptions: ModelConfig;providerConfig: ProviderConfig;systemPrompt: string; }): Agent

Parameters

Parameter	Type
`agentArgs`	`object`
`agentArgs.objectgeneratorOptions`?	`ModelConfig`
`agentArgs.providerConfig`	`ProviderConfig`
`agentArgs.systemPrompt`?	`string`

Parameter	Type	Description
`prompt`	`CoreMessage`[]	The prompt to send to the model
`commandSchema`	`T`	The schema to validate the response
`opts`	`object`	Options for actionCall function
`opts.autoSlice`	`boolean`	Whether to automatically slice the response
`opts.maxDelay`	`number`	Maximum delay
`opts.numOfAttempts`	`number`	Maximum number of retries
`opts.startingDelay`	`number`	Initial delay in milliseconds
`opts.timeMultiple`	`number`	Multiplier for the delay

Parameter	Type
`prompt`	`CoreMessage`[]
`schema`	`T`
`opts`	`object`
`opts.maxDelay`	`number`
`opts.numOfAttempts`	`number`
`opts.startingDelay`	`number`
`opts.timeMultiple`	`number`

Parameter	Type
`prompt`	`CoreMessage`[]
`responseSchema`	`T`
`opts`?	`object`
`opts.autoSlice`?	`boolean`

Parameter	Type
`currentState`	`object`
`currentState.ariaTree`	`string`
`currentState.kind`	`"ObjectiveState"`
`currentState.objective`	`string`
`currentState.progress`	`string`[]
`currentState.url`	`string`
`memories`	`object`
`memories.actionStep`	`object`
`memories.actionStep.command`?	`any`
`memories.actionStep.description`	`string`
`memories.actionStep.objectiveComplete`?	{`kind`: `"ObjectiveComplete"`;`result`: `string`; } \| {}
`memories.actionStep.progressAssessment`	`string`
`memories.objectiveState`	`object`
`memories.objectiveState.ariaTree`	`string`
`memories.objectiveState.kind`	`"ObjectiveState"`
`memories.objectiveState.objective`	`string`
`memories.objectiveState.progress`	`string`[]
`memories.objectiveState.url`	`string`
`responseSchema`	`T`

Documentation - Nolita