quickstart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quickstart\n",
    "This notebook contains a sample program to guide you through the features of the Palimpzest (PZ) library. \n",
    "PZ provides a high-level interface for composing and executing pipelines of semantic operators."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-requisites\n",
    "As Palimpzest uses LLM models for some operations, you need to set up **at least** one of the following\n",
    "API keys as environment variables:\n",
    "\n",
    "- `OPENAI_API_KEY` for using OPENAI's GPT-3.5 and GPT-4 models\n",
    "- `TOGETHER_API_KEY` for using TogetherAI's LLM models, including Mixtral\n",
    "\n",
    "Support for local model execution and other LLM APIs is underway!\n",
    "\n",
    "Edit the following snippet with your API key(s) in order to run the notebook. (Providing both keys enables PZ to perform more optimizations, but this is not necessary for the demo to work.)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"your-openai-api-key\"\n",
    "# os.environ[\"TOGETHER_API_KEY\"] = \"your-together-api-key\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: Enron Email Dataset\n",
    "In this demo we will work with a subset the [Enron Email Dataset](https://www.cs.cmu.edu/~enron/). We are going to use PZ to build a semantic pipeline that does the following:\n",
    "\n",
    "1. Load the text files that contain the emails. (Each `.txt` file contains a single email).\n",
    "2. Extract the sender, subject, and date of each email.\n",
    "3. Filter for the emails that (1) mention a vacation plan and (2) were sent in the month of July.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Create a `pz.Dataset`\n",
    "\n",
    "The first step in any Palimpzest program is to create a `pz.Dataset`, which represents a set of data that we can apply transformations to. In this example, we create the `pz.Dataset` by simply providing the path to our directory of text files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import palimpzest as pz\n",
    "\n",
    "# Dataset loading\n",
    "dataset = pz.Dataset(\"testdata/enron-tiny/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Extract Relevant Fields from Each Email\n",
    "Since we want to extract useful information from the input files, we need to define columns that specify which attributes we are interested in. We define each column with a dictionary that specifies:\n",
    "1. The column name,\n",
    "2. The column type, and\n",
    "3. A natural language description of what the column represents\n",
    "\n",
    "The names and natural language descriptions are used to help PZ properly extract the columns values. The types are used by PZ to type check the values generated for the column.\n",
    "\n",
    "In this demo, we will extract the sender, subject, and date of each email. We can do this by invoking PZ's `dataset.sem_add_columns()` function.\n",
    "\n",
    "**NOTE:** PZ uses [lazy evaluation](https://en.wikipedia.org/wiki/Lazy_evaluation), therefore the `dataset` returned by `dataset.sem_add_columns()` will not contain the computed values of the columns you specified. This computation will happen in Step 4 when we execute `dataset.run()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "email_cols = [\n",
    "    {\"name\": \"sender\", \"type\": str, \"desc\": \"The email address of the sender\"},\n",
    "    {\"name\": \"subject\", \"type\": str, \"desc\": \"The subject of the email\"},\n",
    "    {\"name\": \"date\", \"type\": str, \"desc\": \"The date the email was sent\"},\n",
    "]\n",
    "\n",
    "dataset = dataset.sem_add_columns(email_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Apply a Filter to the Emails\n",
    "Our next step is to filter for the emails that (1) mention a vacation plan and (2) were sent in the month of July.\n",
    "\n",
    "To do this, we will use the `dataset.sem_filter()` function. This function takes a string which describes the condition we are filtering for."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = dataset.sem_filter(\"The email was sent in July\")\n",
    "dataset = dataset.sem_filter(\"The email is about holidays\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Execute the Operations\n",
    "Finally, we can execute the operations we have defined on the `dataset` by calling `dataset.run()`.\n",
    "\n",
    "The `dataset.run()` function takes a `QueryProcessorConfig` as its sole argument. This config enables the user to control certain aspects of PZ's execution. For example, the `policy` config option allows the user to specify what PZ should optimize for when executing your program. Some policies include:\n",
    "- `MinCost`: minimize the cost of the program\n",
    "- `MinTime`: minimize the runtime of the program\n",
    "- `MaxQuality`: maximize the quality of the program output\n",
    "- `MaxQualityAtFixedCost`: maximize the output quality subject to an upper bound on the cost\n",
    "- etc.\n",
    "\n",
    "For a full list of policies please see our documentation.\n",
    "\n",
    "There are additional config parameters which control e.g. the parallelism used by PZ, the optimization strategy, and more. The details of these parameters can also be found in our documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NOTE: PZ supports many policies including pz.MaxQuality, pz.MaxQualityAtFixedCost, and more.\n",
    "#       See our documentation for more details: https://palimpzest.org/policy.html\n",
    "\n",
    "config = pz.QueryProcessorConfig(policy=pz.MinCost(), verbose=True)\n",
    "output = dataset.run(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Displaying the Output\n",
    "\n",
    "To print the results as a table, we can utilize the `to_df()` method of our `output` object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_df = output.to_df(cols=[\"date\", \"sender\", \"subject\"])\n",
    "display(output_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PZ also provides a detailed report of the execution sttatistics, including information about the runtime and cost of each operation.\n",
    "To access these statistics, you can use the `execution_stats` attribute of our `output` object:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "execution_stats = output.execution_stats\n",
    "print(\"Time to find an optimal plan:\", execution_stats.total_optimization_time,\"s\")\n",
    "print(\"Time to execute the plan:\", execution_stats.total_execution_time, \"s\")\n",
    "print(\"Total cost:\", execution_stats.total_execution_cost, \"USD\")\n",
    "\n",
    "print(\"Final plan executed:\")\n",
    "for plan, stats in execution_stats.plan_stats.items():\n",
    "    print(stats)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We hope this notebook is only the start of your Palimpzest journey! Feel free to reach out to us for more information!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}