autopilot

The autopilot module provides a way to create Voice AI applications in Fonoster. It uses Fonoster internal Voice API and the power of Large Language Models to create a conversational experience with the calling party.

• Installation
• Example
• Adding Knowledge Base
• Adding Tools

Installation

Autopilot is an ordinary session app that AppBridge dials. Run the fonoster/autopilot image (built from mods/autopilot/Dockerfile), or from the repository:

$ cp config/autopilot-session.example.yaml config/autopilot-session.yaml
$ node mods/autopilot/dist/session/main.js --config config/autopilot-session.yaml

See docs/autopilot.md for the engine, pacing and the agent schema.

Example

To function, Autopilot needs a configuration file, an OpenAI API key, and an active Fonoster account or self-hosted Fonoster instance.

The configuration file has two major sections: conversationSettings and languageModel. The conversationSettings section contains the settings for the conversation, like the first message, the system template, and the transfer options. The languageModel section contains the settings for the language model, like the provider, the model, and the temperature, among others.

{
  "conversationSettings": {
    "firstMessage": "Hello, this is Olivia from Dr. Green's Family Medicine. How can I assist you today?",
    "systemPrompt": "You are a Customer Service Representative. You are here to help the caller with their needs.",
    "systemErrorMessage": "I'm sorry, but I seem to be having trouble. Please try again later.",
    "initialDtmf": "6589",
    "transferOptions": {
      "phoneNumber": "+15555555555",
      "message": "Please hold while I transfer you to a live agent.",
      "timeout": 30000
    },
    "idleOptions": {
      "message": "Are you still there?",
      "timeout": 10000,
      "maxTimeoutCount": 3
    }
  },
  "languageModel": {
    "provider": "openai",
    "model": "gpt-4o-mini",
    "maxTokens": 250,
    "temperature": 0.7,
    "knowledgeBase": [],
    "tools": []
  }
}

To run the Autopilot, you can use the following command:

$ AGENT=$(pwd)/agents.json OPEN_API_KEY=your-key autopilot

You will see a message like this:

2024-09-06 11:32:16.584 [info]: (voice) started voice server @ 0.0.0.0, port=50061 {}

This means that the Autopilot is running and ready to take calls.

Since you are running Autopilot locally, you must expose port 50061 to the Internet. To expose the port, you can use a service like ngrok

$ ngrok tcp 50061

Take the endpoint provided by ngrok and configure it in your Fonoster account. You can find more information about how to do this in the Fonoster documentation.

Language Model Providers

The Autopilot supports multiple language model providers. The following is a list of the supported providers:

We are constantly updating the list of supported providers and models. If you have a specific model you would like to use, please let us know.

Adding Knowledge Base

A knowledge base provides information to the language model, such as business hours, services, or products.

Currently, we support retrieving documents from an S3 bucket.

To add a knowledge base, include a new entry in the knowledgeBase array in the configuration file. Below is an example of a knowledge for a menu PDF.

{
  "conversationSettings": { ... },
  "languageModel": {
    "provider": "openai",
    "model": "gpt-4o-mini",
    "apiKey": "your-api-key",
    "maxTokens": 100,
    "temperature": 0.4,
    "knowledgeBase": [{
      "type": "s3",
      "title": "Menu PDF",
      "document": "sample.pdf"
    }],
    "tools": []
  }
}

Adding Tools

A tool in Autopilot is a way to interact with external services to get information or perform actions. For example, you can use a tool to get the weather, the latest news, or an SMS.

You can configure a new tool by adding a new entry in the tools array in the configuration file. The following is an example tool that gets the available appointment times for a specific date.

{
  "conversationSettings": { ... },
  "languageModel": {
    "provider": "openai",
    "model": "gpt-4o-mini",
    "apiKey": "your-api-key",
    "maxTokens": 100,
    "temperature": 0.4,
    "knowledgeBase": [],
    "tools": [
      {
        "name": "getAvailableTimes",
        "description": "Get available appointment times for a specific date.",
        "parameters": {
          "type": "object",
          "properties": {
            "date": {
              "type": "string",
              "format": "date"
            }
          },
          "required": [
            "date"
          ]
        },
        "operation": {
          "type": "get",
          "url": "https://api.example.com/appointment-times",
          "headers": {
            "x-api-key": "your-api-key"
          }
        }
      }
    ]
  }
}

In addition to the get operation type, you can use the post operation type. The post operation type is used when sending data to the tool. When sending a post, you can optionally set waitForResponse to false, which will "fire and forget" the request. The default behavior is to wait for the response.

If your tool needs the number of the caller or the number that received the call, you can use the reserved variables ingressNumber and callerNumber. Similarly, you can use the reserved variable callReceivedAt to get the date and time when the call was received in ISO 8601 format and the callDirection variable to check if the call was originated from the PSTN.

The expected format for the response is a JSON object with the following structure:

{ "result": "text to pass to the language model" }

For example:

{ "result": "The available appointment times are 9:00 AM, 10:00 AM, and 11:00 AM." }

Configuration

Autopilot reads autopilot-session.yaml, found via --config <path>, the AUTOPILOT_SESSION_CONFIG variable or ./config/autopilot-session.yaml. It names the signing secret AppBridge uses for Autopilot apps, where apps and agents come from, the registry catalog, platform keys by credential ref and the knowledge settings. See config/autopilot-session.example.yaml for every key.

Audio filters are part of the app's configuration (AppBridge runs them), not the agent's.