docs: Document database-backed query analyzer (#2904)

kyleconroy · andrewmbenton · web-flow · commit 3a97e74711b6 · 2023-10-23T18:27:20.000-07:00
* docs: Document database-backed query analyzer

---------

Co-authored-by: Andrew Benton &lt;andrew@sqlc.dev&gt;
diff --git a/docs/howto/generate.md b/docs/howto/generate.md
@@ -0,0 +1,79 @@
+# `generate` - Generating code
+
+`sqlc generate` parses SQL, analyzes the results, and outputs code. Your schema and queries are stored in separate SQL files. The paths to these files live in a `sqlc.yaml` configuration file.
+
+```yaml
+version: "2"
+sql:
+  - engine: "postgresql"
+    queries: "query.sql"
+    schema: "schema.sql"
+    gen:
+      go:
+        package: "tutorial"
+        out: "tutorial"
+        sql_package: "pgx/v5"
+```
+
+We've written extensive docs on [retrieving](select.md), [inserting](insert.md),
+[updating](update.md), and [deleting](delete.md) rows. 
+
+By default, sqlc runs its analysis using a built-in query analysis engine. While fast, this engine can't handle some complex queries and type-inference.
+
+You can configure sqlc to use a database connection for enhanced analysis using metadata from that database.
+
+The database-backed analyzer currently supports PostgreSQL, with [MySQL](https://github.com/sqlc-dev/sqlc/issues/2902) and [SQLite](https://github.com/sqlc-dev/sqlc/issues/2903)
+support planned in the future.
+
+## Enhanced analysis with managed databases
+
+```{note}
+Managed databases are powered by [sqlc Cloud](https://dashboard.sqlc.dev). Sign up for [free](https://dashboard.sqlc.dev) today.
+```
+
+With [managed databases](managed-databases.md) configured, `generate` will automatically create a hosted ephemeral database with your
+schema and use that database to improve its query analysis. And sqlc will cache its analysis locally
+on a per-query basis to speed up future `generate` runs. This saves you the trouble of running and maintaining a database with
+an up-to-date schema. Here's a minimal working configuration:
+
+```yaml
+version: "2"
+cloud:
+  project: "<PROJECT_ID>"
+sql:
+  - engine: "postgresql"
+    queries: "query.sql"
+    schema: "schema.sql"
+    database:
+      managed: true
+    gen:
+      go:
+        out: "db"
+        sql_package: "pgx/v5"
+```
+
+## Enhanced analysis using your own database
+
+You can opt-in to database-backed analysis using your own database, by providing a `uri` in your sqlc
+[database](../reference/config.md#database) configuration.
+
+The `uri` string can contain references to environment variables using the `${...}`
+syntax. In the following example, the connection string will have the value of
+the `PG_PASSWORD` environment variable set as its password.
+
+```yaml
+version: "2"
+sql:
+  - engine: "postgresql"
+    queries: "query.sql"
+    schema: "schema.sql"
+    database:
+      uri: "postgres://postgres:${PG_PASSWORD}@localhost:5432/postgres"
+    gen:
+      go:
+        out: "db"
+        sql_package: "pgx/v5"
+```
+
+Databases configured with a `uri` must have an up-to-date schema for query analysis to work correctly, and `sqlc` does not apply schema migrations your database. Use your migration tool of choice to create the necessary
+tables and objects before running `sqlc generate`.
diff --git a/docs/howto/upload.md b/docs/howto/upload.md
@@ -1,4 +1,4 @@
-# Uploading projects
+# `upload` - Uploading projects
 
 ```{note}
 Project uploads are powered by [sqlc Cloud](https://dashboard.sqlc.dev). Sign up for [free](https://dashboard.sqlc.dev) today.
@@ -18,7 +18,7 @@ After creating a project, add the project ID to your sqlc configuration file.
 ```yaml
 version: "2"
 cloud:
-  project: "<PROJECT-ID>"
+  project: "<PROJECT_ID>"
 ```
 
 You'll also need to create an auth token and make it available via the
diff --git a/docs/howto/vet.md b/docs/howto/vet.md
@@ -1,4 +1,4 @@
-# Linting queries
+# `vet` - Linting queries
 
 *Added in v1.19.0*
 
@@ -43,7 +43,7 @@ message Parameter
 ```
 
 In addition to this basic information, when you have a PostgreSQL or MySQL
-[database connection configured](../reference/config.html#database)
+[database connection configured](../reference/config.md#database)
 each CEL expression has access to the output from running `EXPLAIN ...` on your query
 via the `postgresql.explain` and `mysql.explain` variables.
 This output is quite complex and depends on the structure of your query but sqlc attempts
@@ -95,7 +95,7 @@ rules:
 The CEL expression environment has two variables containing `EXPLAIN ...` output,
 `postgresql.explain` and `mysql.explain`. `sqlc` only populates the variable associated with
 your configured database engine, and only when you have a
-[database connection configured](../reference/config.html#database).
+[database connection configured](../reference/config.md#database).
 
 For the `postgresql` engine, `sqlc` runs
 
@@ -163,22 +163,27 @@ rules:
   rule: "!has(postgresql.explain)" # A dummy rule to trigger explain
 ```
 
-Please note that `sqlc` does not manage or migrate your database. Use your
-migration tool of choice to create the necessary database tables and objects
-before running `sqlc vet` with rules that depend on `EXPLAIN ...` output.
+Please note that databases configured with a `uri` must have an up-to-date
+schema for `vet` to work correctly, and `sqlc` does not apply schema migrations
+to your database. Use your migration tool of choice to create the necessary
+tables and objects before running `sqlc vet` with rules that depend on
+`EXPLAIN ...` output.
+
+Alternatively, configure [managed databases](managed-databases.md) to have
+`sqlc` create hosted ephemeral databases with the correct schema automatically.
 
 ## Built-in rules
 
 ### sqlc/db-prepare
 
-When a [database](../reference/config.html#database) connection is configured, you can
+When a [database](../reference/config.md#database) connection is configured, you can
 run the built-in `sqlc/db-prepare` rule. This rule will attempt to prepare
 each of your queries against the connected database and report any failures.
 
 ```yaml
 version: 2
 sql:
-  - schema: "query.sql"
+  - schema: "schema.sql"
     queries: "query.sql"
     engine: "postgresql"
     gen:
@@ -191,12 +196,20 @@ sql:
       - sqlc/db-prepare
 ```
 
-Databases configured with a `uri` must have an up-to-date schema, and `sqlc` does not apply schema migrations your database. You can configure [managed databases](managed-databases.md) instead to have `sqlc` create and migrate databases automatically.
+Please note that databases configured with a `uri` must have an up-to-date
+schema for `vet` to work correctly, and `sqlc` does not apply schema migrations
+to your database. Use your migration tool of choice to create the necessary
+tables and objects before running `sqlc vet` with the `sqlc/db-prepare` rule.
+
+Alternatively, configure [managed databases](managed-databases.md) to have
+`sqlc` create hosted ephemeral databases with the correct schema automatically.
 
 ```yaml
 version: 2
+cloud:
+  project: "<PROJECT_ID>"
 sql:
-  - schema: "query.sql"
+  - schema: "schema.sql"
     queries: "query.sql"
     engine: "postgresql"
     gen:
@@ -215,7 +228,7 @@ example](https://github.com/sqlc-dev/sqlc/blob/main/examples/authors/sqlc.yaml).
 ## Running lint rules
 
 When you add the name of a defined rule to the rules list
-for a [sql package](https://docs.sqlc.dev/en/stable/reference/config.html#sql),
+for a [sql package](../reference/config.md#sql),
 `sqlc vet` will evaluate that rule against every query in the package.
 
 In the example below, two rules are defined but only one is enabled.
diff --git a/docs/index.rst b/docs/index.rst
@@ -36,6 +36,15 @@ code ever again.
    tutorials/getting-started-postgresql.md
    tutorials/getting-started-sqlite.md
 
+.. toctree::
+   :maxdepth: 2
+   :caption: Commands
+   :hidden:
+
+   howto/generate.md
+   howto/vet.md
+   howto/upload.md
+
 .. toctree::
    :maxdepth: 2
    :caption: How-to Guides
@@ -57,10 +66,12 @@ code ever again.
    howto/overrides.md
    howto/rename.md
 
-   howto/vet.md
+.. toctree::
+   :maxdepth: 3
+   :caption: sqlc Cloud
+   :hidden:
+
    howto/managed-databases.md
-   howto/ci-cd.md
-   howto/upload.md
 
 .. toctree::
    :maxdepth: 3
@@ -81,7 +92,8 @@ code ever again.
    :caption: Conceptual Guides
    :hidden:
 
+   howto/ci-cd.md
    guides/using-go-and-pgx.rst
-   guides/development.md
    guides/plugins.md
+   guides/development.md
    guides/privacy.md
diff --git a/docs/reference/config.md b/docs/reference/config.md
@@ -7,6 +7,8 @@ file must be in the directory where the `sqlc` command is run.
 
 ```yaml
 version: "2"
+cloud:
+  project: "<PROJECT_ID>"
 sql:
 - schema: "postgresql/schema.sql"
   queries: "postgresql/query.sql"
@@ -16,7 +18,7 @@ sql:
       package: "authors"
       out: "postgresql"
   database:
-    uri: "postgresql://postgres:postgres@localhost:5432/postgres"
+    managed: true
   rules:
     - sqlc/db-prepare
 - schema: "mysql/schema.sql"
@@ -46,6 +48,8 @@ Each mapping in the `sql` collection has the following keys:
   - A mapping to configure database connections. See [database](#database) for the supported keys.
 - `rules`:
   - A collection of rule names to run via `sqlc vet`. See [rules](#rules) for configuration options.
+- `analzyer`:
+  - A mapping to configure query analysis. See [analyzer](#analyzer) for the supported keys.
 - `strict_function_checks`
   - If true, return an error if a called SQL function does not exist. Defaults to `false`.
 
@@ -85,6 +89,8 @@ sql:
 
 The `database` mapping supports the following keys:
 
+- `managed`:
+  - If true, connect to a [managed database](../howto/managed-databases.md). Defaults to `false`.
 - `uri`:
   - Database connection URI
 
@@ -105,7 +111,14 @@ sql:
       package: authors
       out: postgresql
 ```
- 
+
+### analyzer
+
+The `analyzer` mapping supports the following keys:
+
+- `database`:
+  -  If false, do not use the configured database for query analysis. Defaults to `true`.
+  
 ### gen
 
 The `gen` mapping supports the following keys:
diff --git a/docs/tutorials/getting-started-postgresql.md b/docs/tutorials/getting-started-postgresql.md
@@ -32,7 +32,7 @@ following contents:
 ```yaml
 version: "2"
 cloud:
-  project: "<your project id>"
+  project: "<PROJECT_ID>"
 sql:
   - engine: "postgresql"
     queries: "query.sql"
