Kat Batuigas | CrunchyData Blog

Querying Spatial Data with PostGIS and ogr_fdw

Kat.Batuigas@crunchydata.com (Kat Batuigas) — Fri, 03 Sep 2021 05:00:00 EDT

In my last post, I did a simple intro to foreign data wrappers in PostgreSQL. postgres_fdw is an extension available in Postgres core that allows you to issue queries against another Postgres database. It's just one of many foreign data wrappers that you can use in Postgres, so for today's post we'll look at another that works especially well with spatial data formats: ogr_fdw.

I had also previously talked about some different ways to get spatial data into a Postgres/PostGIS database, but the cool thing about foreign data wrappers is that they're an alternative to needing to have everything in the same data store. With spatial data being stored and shared in so many different formats, imagine being able to abstract that conversion away and just focus on analysis. Read on for a couple of quick demos.

Get started with ogr_fdw

Ogr_fdw is a Postgres extension. In order to install it, you have to download your Linux distro package, or use StackBuilder to add it to your Postgres install on Windows. This is what I ran in Ubuntu 20.04:

sudo apt update
sudo apt install postgresql-13-ogr-fdw

You'll also want to make sure you've enabled the PostGIS extension to take advantage of spatial functions and filtering. Otherwise, spatial data will be in bytea format (representing Well Known Binary [WKB] geometries).

Like most extensions, you can enable ogr_fdw with the CREATE EXTENSION command:

test=# CREATE EXTENSION ogr_fdw;

Then, you can add the foreign server, then create the foreign schema. Let's follow along with the instructions from the ogr_fdw GitHub repo, then look at a more involved example afterwards.

Example: Shapefile saved locally

test=# CREATE SERVER myserver FOREIGN DATA WRAPPER ogr_fdw OPTIONS (datasource '/tmp/data', format 'ESRI Shapefile');
test=# IMPORT FOREIGN SCHEMA ogr_all
LIMIT TO (pt_two)
FROM SERVER myserver INTO public;

First, we set our data source to the directory that contains the shapefile to query. ogr_fdw works with the full list of GDAL/OGR formats.

IMPORT FOREIGN SCHEMA is available starting from Postgres 9.5 and allows you to automatically create foreign tables for the foreign server. ogr_all is a "special" schema - ogr_fdw will by default create foreign tables for all layers detected in the data source. Here, we're saying that we only want the layer named pt_two to be created as a foreign table. You can use the LIMIT TO / EXCLUDE Postgres clauses if you want only specific layers. Or, you can use quotes around the remote schema string like so:

IMPORT FOREIGN SCHEMA "pt"
FROM SERVER myserver INTO public;

This will create tables for matching layers only (ie names prefixed with pt).

Alternatively, you could use CREATE FOREIGN TABLE to manually create individual tables:

test=# CREATE FOREIGN TABLE pt_two (
    fid int,
    geom geometry(Point, 4326),
    name varchar,
    age int,
    height real,
    birthdate date) SERVER myserver OPTIONS (layer 'pt_two');

The OPTIONS clause accepts a layer parameter for the name of the layer to query.

Wait, what if we don't know the table definition? Not a problem! ogr_fdw comes with the ogr_fdw_info utility - you can use it like so:

ogr_fdw_info -s /tmp/data/ -l pt_two

And it'll generate the CREATE SERVER and CREATE FOREIGN TABLE statements for that layer. You might prefer to go this route instead.

And now, you can use SQL to query the shapefile!

test=# SELECT * FROM pt_two;
 fid |                        geom                        | name  | age | height | birthdate
-----+----------------------------------------------------+-------+-----+--------+------------
   0 | 0101000020E6100000C00497D1162CB93F8CBAEF08A080E63F | Peter |  45 |    5.6 | 1965-04-12
   1 | 0101000020E610000054E943ACD697E2BFC0895EE54A46CF3F | Paul  |  33 |   5.84 | 1971-03-25

Neat. Now let's try a different example. What if we want to connect to a non-local file?

Example: Zipped shapefile hosted on a private S3 bucket

In this example, I'll query a zip file that contains a shapefile of neighborhood association boundaries within the City of Tampa (Neighborhoods.zip), hosted in a private S3 bucket named kbatu-example.

If you take a look at the GDAL docs for AWS S3, you'll see that the second authentication method requires a couple of config options to be set. How do you set config options with ogr_fdw? That's done by passing in a config_options parameter when creating the foreign server. Here's how I've set up the foreign server and table:

tampa=# CREATE SERVER s3_tampa
  FOREIGN DATA WRAPPER ogr_fdw
  OPTIONS (
    datasource '/vsizip/vsis3/kbatu-example/Neighborhoods.zip',
    format 'ESRI Shapefile',
    -- Separate multiple config options with space
    config_options 'AWS_ACCESS_KEY_ID=access_key_id AWS_SECRET_ACCESS_KEY=my_access_key');
tampa=# IMPORT FOREIGN SCHEMA ogr_all FROM SERVER s3_tampa INTO public;

In psql, I can use these backslash commands to do a quick check on my foreign server and tables:

tampa=# \des
              List of foreign servers
       Name       |  Owner   | Foreign-data wrapper
------------------+----------+----------------------
 s3_tampa         | postgres | ogr_fdw
(1 row)

tampa=# \det
      List of foreign tables
 Schema |     Table     |  Server
--------+---------------+----------
 public | neighborhoods | s3_tampa
(1 row)

Query the shapefile: spatial joins and SQL filters

Some months ago, I had started to play around with open data from the City of Tampa. I thought it would be a great way to become more comfortable with PostGIS while getting to know my city a little better. I have Public Art data stored in my database, so why not try a spatial query on it with my new Neighborhoods table?

For instance, in the time I've lived in Tampa, I've seen that many of the public art installations are found in the central downtown area, which itself consists of a few different neighborhoods, depending on whom you ask. Which art installations are located outside of that central downtown area? (I'm using the neighborhood associations dataset, and the AssocLabel field in particular as a proxy.)

Here's my version of a query that answers that question:

tampa=# SELECT
    a.title AS artwork_name,
    n.assoclabel AS neighborhood_name,
    n.zipcode AS neighborhood_zip
FROM tampa_public_art a
JOIN neighborhoods n
ON ST_Contains(n.geom, ST_Transform(a.geom, 4326))
WHERE n.assoclabel NOT IN ('Tampa Downtown Partnership', 'Channel District', 'Downtown River Arts Neighborhood Association', 'Port of Tampa');

I've used a spatial join to form a table of geometries where the public artwork points are found within the neighborhood polygons. Note that I apply ST_Transform to reproject the artwork geometries since the dataset uses the Pseudo-Mercator/EPSG:3857 projection, whereas the neighborhoods data uses WGS84/EPSG:4326. Pseudo-Mercator is more appropriate for web maps or visualizing spatial data, which isn't the crux of what I'm trying to do so I prefer to use WGS84 for this analysis. I'm also using a NOT IN operator to exclude my own (arbitrary) list of neighborhoods that count for the downtown area.

The following image is taken from DBeaver's Spatial Viewer (adding in the neighborhood and artwork geometries to the SELECT statement):

There are 39 (out of 89) works that are a little bit more dispersed throughout the city, in neighborhoods such as Macfarlane Park in West Tampa, and Sulphur Springs between the zoo and the University of South Florida campus. Now I know to keep an eye out for them if I'm in those areas.

I will also point out that this is a relatively simple query involving small tables, and since we're wrapping around a file and not a database, my query doesn't use any indexes to gain efficiencies. So, ogr_fdw may not be the first tool I reach for when it comes to complex queries on larger datasets. With that said, I'd be happy to have it as an option for more adhoc analyses. The OGR SQL documentation also mentions some query limitations and I'd recommend looking those over as well.

Tell me you're using Postgres, without telling me you're using Postgres

In the above example, we're able to query data in a shapefile without having to:

Download the zip file
Extract its contents
Then load the spatial data into Postgres/PostGIS with ogr2ogr or shp2pgsql

Foreign data wrappers really can open up some new possibilities for data analysis with Postgres. I've only scratched the surface here with ogr_fdw, but don't forget to browse the project README for even more examples and tips (such as how to view the logic pushed down from OGR to the source data). And, if you enjoyed this post, you'll probably want to check out some of my colleague Paul Ramsey's ogr_fdw musings as well as appearances on the Crunchy Data blog.

Understanding Foreign Data Wrappers in Postgres and postgres_fdw

Kat.Batuigas@crunchydata.com (Kat Batuigas) — Wed, 18 Aug 2021 05:00:00 EDT

The idea of writing a database query that can then go out to an external sourcemay not occur to someone who is not a DBA early on. That is: instead of figuring out how to grab then load multiple data sets into the same store, or configuring your application backend to connect to a bunch of disparate sources, why not use query JOINs like you usually would across tables within one database?

In case you're not familiar, the dblink module in PostgreSQL, along with the concept of database links or linked servers in other DBMSs, has been around for a while. Foreign data wrappers are newer, having been introduced with PG 9+. Postgres now has a lot of foreign data wrappers available and they work with plenty of different source types: NoSQL databases, platforms like Twitter and Facebook, geospatial data formats, etc. My colleague Craig Kerstiens has shared his thoughts on Postgres being a "batteries included" database, and it's so easy to see why.

That said, why might you still need foreign data wrappers between Postgres servers? The default implementation doesn't support cross-database queries, even on the same Postgres server. So you still need the wrapper to handle the connection and fetch foreign data. postgres_fdw is more or less the dblink equivalent for access between Postgres servers, with the main difference being that postgres_fdw conforms to SQL standards. They do provide a lot of the same functionality but postgres_fdw is more recommended and more widely used at this point.

postgres_fdw basics

If you haven't tried postgres_fdw, setting it up is pretty simple. Say I have a contacts table on a local Postgres install, and I want to be able to query a remote Crunchy Bridge database that stores sales information. I can do the following:

Load the extension in my local database (postgres_fdw is included in Postgres contrib, and you do need CREATE privileges on the local database):
```
CREATE EXTENSION postgres_fdw;
```

Create a foreign server:

CREATE SERVER salesinfo_bridge
	FOREIGN DATA WRAPPER postgres_fdw
    OPTIONS (host 'p.2gdmzr2pcbadzcstrkuolxvtpq.db.postgresbridge.com', dbname 'sales');

Set up a user mapping to authenticate:

CREATE USER MAPPING FOR postgres
	SERVER salesinfo_bridge
    OPTIONS (user 'fdw_user', password 'password');

Then I can set up foreign tables that correspond to the tables I want to query on the foreign server. This is done in two different ways:
- Run CREATE FOREIGN TABLE, which is pretty similar to CREATE TABLE in that you have to define column names, data types, constraints etc.
- Run IMPORT FOREIGN SCHEMA, which imports tables and views from a schema, and creates foreign tables that match the definitions for the external tables. You even have the option to include/exclude specific tables only, which makes it even more convenient:
```
test=# IMPORT FOREIGN SCHEMA public LIMIT TO (payment_methods, accounts)
FROM SERVER salesinfo_bridge INTO public;
```

And I can carry on querying as if these tables were all on the same database!

test=# SELECT c.id, pm.type, acct.balance
FROM contacts c
LEFT JOIN accounts acct ON c.id = acct.contact_id
LEFT JOIN payment_methods pm ON acct.id = pm.acct_id
WHERE acct.balance > 0;
id  |    type    | balance
----+------------+----------
  1 | mastercard | 2742.62
  2 | mastercard |  464.76
  3 | mastercard |  116.67

Even though I've recreated up the foreign schema on the local server, these tables don't actually store data locally. This means that the local server doesn't know anything about statistics from the external server (we'll look at the implications a bit later).

Don't forget about the psql commands that provide information related to foreign data wrappers:

\des - list foreign servers
\deu - list uses mappings
\det - list foreign tables
\dtE - list both local and foreign tables
\d <name of foreign table> - show columns, data types, and other table metadata

Some things to consider for query optimization

postgres_fdw will try to optimize queries on its own. With that said, the local server doesn't automatically gather statistics from the foreign server or tables either. That makes sense, but can also make query planning and optimization still somewhat involved. As a user, you might consider these approaches (the first two are described in the official postgres_fdw docs):

Tell the foreign data wrapper that you want the foreign server to perform the cost estimate, by setting the use_remote_estimate option to true on the server or table level.
So, every time you query the foreign server, you're asking it to perform additional EXPLAIN commands. You're effectively requesting more across the network each time, which may add to a longer total time for your query to return results depending on how complex the query is. If you leave it to the default false value, the local server performs the cost estimation itself.
Run ANALYZE on the foreign tables, which updates those table statistics on the local server. But if the foreign tables are updated pretty frequently, the local statistics can quickly become stale as well. So you'd also need to consider how often you may have to schedule ANALYZE.
Use a materialized view on the foreign table, which you can also refresh on a desired basis. You'd be working with a local snapshot of the external data, which should help with speed. This could work well when you're not required to always use live, up-to-the-minute data.

Foreign data wrappers as an alternative to ETL?

From my non-DBA perspective, the main takeaway is that foreign data wrappers can simplify data querying and analysis when you need data from disparate sources. And the biggest drawback may be that in many cases you can't "set it and forget it," if you don't want to risk poor query performance. With that said, if you're dealing with massive amounts of data anyway, then there might be more suitable approaches. But either way, it's nice that there are other options aside from the standard ETL/ELT pattern.

For those of you out there using foreign data wrappers, what have been your most important considerations? I'm also curious to hear of other use cases for FDWs. We're all ears at @crunchydata.

Postgres Full-Text Search: A Search Engine in a Database

Kat.Batuigas@crunchydata.com (Kat Batuigas) — Tue, 27 Jul 2021 05:00:00 EDT

Early in on my SQL journey, I thought that searching for a piece of text in the database mostly involved querying like this:

SELECT col FROM table WHERE col LIKE '%some_value%';

Then I would throw in some wildcard operators or regular expressions if I wanted to get more specific.

Later on, I worked with a client who wanted search functionality in an app, so LIKE and regex weren't going to cut it. What I had known all along was just pattern matching. It works perfectly fine for certain purposes, but what happens when it's not just a matter of checking for a straightforward pattern in a single text field?

For example, what if you wanted to search across multiple fields? How about returning possible matches even if the search term happens to be misspelled? Also, what if you have very large amounts of data to search on? Sure, you can create indexes for columns on which you want to query for pattern matches, but that will have limitations (for instance, the B-tree index doesn't work for col LIKE '%substring%').

So when we say PostgreSQL is the "batteries included database," this is just one reason why. With Postgres, you don't need to immediately look farther than your own database management system for a full-text search solution. If you haven't yet given Postgres' built-in full-text search a try, read on for a simple intro.

Postgres Full-Text Search Basics for the Uninitiated

Core Postgres includes the following full-text search capabilities. To name a few:

Ignore stop words (common words such as "the" or "an").
Stemming, where search matches can be based on a "root" form, or stem, of a word (“run” matches “runs” and “running” and even “ran”).
Weight and rank search matches (so best matches can be sorted to the top of a result list).

Before we go further, let's also get ourselves familiarized with the following concepts:

A document is a set of data on which you want to carry out your full-text search. In Postgres, this could be built from a single column, or a combination of columns, even from multiple tables.
The document is parsed into tokens, which are small fragments (e.g. words, phrases, etc) from the document's text. Tokens are then converted to more meaningful units of text called lexemes.
In Postgres, this conversion is done with dictionaries -- there are built-in ones, but custom dictionaries can be created if necessary. These dictionaries help determine stop words that should get ignored, and whether differently-derived words have the same stem. Most dictionaries are for a specific language (English, German, etc) but you could also have ones that are for a specific domain.
The sorted list of lexemes from the document is stored in the tsvector data type.

Example: Searching Storm Event Details

I have a table that contains storm events data gathered by the U.S. National Weather Service. For simplicity's sake I won't include all possible fields in the statement below, but there's a copy of the data and some further information available in this repository.

CREATE TABLE se_details (
    episode_id int,
    event_id int primary key,
    state text,
    event_type text,
    begin_date_time timestamp,
    episode_narrative text,
    event_narrative text,
    ...
);

Let's also say that we want to carry out a full-text search on the data on the event_narrative column. We could add a new column to the table to store the preprocessed search document (i.e. the list of lexemes):

ALTER TABLE se_details ADD COLUMN ts tsvector
    GENERATED ALWAYS AS (to_tsvector('english', event_narrative)) STORED;

ts is a generated column (new as of Postgres 12), and it's automatically synced with the source data.

We can then create a GIN index on ts:

CREATE INDEX ts_idx ON se_details USING GIN (ts);

And then we can query like so:

SELECT state, begin_date_time, event_type, event_narrative
FROM se_details
WHERE ts @@ to_tsquery('english', 'tornado');

tsquery is the other full-text search data type in Postgres. It represents search terms that have also been processed as lexemes, so we'll pass in our input term to the to_tsquery function in order to optimize our query for full-text search. (@@ is a match operator.)

What we get with this query are records where "tornado" is somewhere in the text string, but in addition to that, here are a couple of records in the result set where there are also matches for "tornado" as lexeme ("tornado-like" and "tornadoes"):

state           | KENTUCKY
begin_date_time | 2018-04-03 18:08:00
event_type      | Thunderstorm Wind
event_narrative | A 1.5 mile wide swath of winds gusting to around 95 mph created **tornado-like** damage along Kentucky Highway 259 in Edmons
on County. The winds, extending 3/4 of a mile north and south of Bee Spring, destroyed or heavily damaged several small outbuildings, tore
part of the roof off of one home, uprooted and snapped the trunks of numerous trees, and snapped around a dozen power poles. Several othe
r homes sustained roof damage, and wind-driven hail shredded vinyl siding on a number of buildings.

state           | WISCONSIN
begin_date_time | 2018-08-28 15:30:00
event_type      | Thunderstorm Wind
event_narrative | A swath of widespread tree and crop damage across the southern portion of the county. Sections of trees and crops compl
etely flattened, and some structural damage from fallen trees or due to the strong downburst winds. Various roads closed due to fallen tre
es. Two semi-trucks were overturned on highway 57 in Waldo. The widespread wind damage and tornadoes caused structural damage to many home
s with 70 homes sustaining affected damage, 3 homes with minor damage, 2 homes with major damage, one home destroyed, and 2 businesses wit
h minor damage.

Searching for Phrases

One way to handle phrases as search terms is to use the & (AND) or <-> (FOLLOWED BY) Boolean operators with the tsquery.

For example, if we want to search for the phrase "rain of debris":

SELECT state, begin_date_time, event_type, event_narrative
FROM se_details
WHERE ts @@ to_tsquery('english', **'rain & of & debris'**);

The search phrase gets normalized to 'rain' & 'debri'. The order doesn't matter as long as both 'rain' and 'debri' have matches in the document, such as this example:

A debris flow caused by heavy rain on a saturated hillside blocked the Omak River Road one mile south of the intersection with State Route 97.

If we do to_tsquery('english', 'rain <-> of <-> debris') the tsquery value is 'rain' <2> 'debri', meaning it will only match where 'rain' is followed by 'debri' precisely two positions away, such as here:

Heavy rain caused debris flows on the Coal Hollow Fire and Tank Hollow Fire burn scars.

(This was actually the only match, so using the <-> operator is a little bit more restrictive.)

The phraseto_tsquery function can also parse the phrase itself, and inserts <N> between lexemes where N is the integer position of the next lexeme when counting from the preceding one. This function doesn't recognize operators unlike to_tsquery; for example, we can just pass in the entire phrase like so:

phraseto_tsquery('english', 'rain of debris')

The tsquery value is 'rain' <2> 'debri' like above, so phraseto_tsquery also accounts for positioning.

Functions for Weighting and Ranking Search Results

One very common use case for assigning different weights and ranking is searching on articles. For example, you may want to merge the article title and abstract or content together for search, but want matches on title to be considered more relevant and thus rank higher.

Going back to our storm events example, our data table also has an episode_narrative column in addition to event_narrative. For storm data, an event is an individual type of storm event (e.g. flood, hail), while an episode is an entire storm system and could contain many different types of events.

Let's say we want to be able to carry out a full-text search on event as well as episode narratives, but have decided that the event narrative should weigh more than the episode narratives. We could define the ts column like this instead:

ALTER TABLE se_details ADD COLUMN ts tsvector
    GENERATED ALWAYS AS
     **(setweight(to_tsvector('english', coalesce(event_narrative, '')), 'A') ||**
     **setweight(to_tsvector('english', coalesce(episode_narrative, '')), 'B'))** STORED;

setweight is a full-text function that assigns a weight to the components of a document. The function takes the characters 'A', 'B', 'C', or 'D' (most weight to least, in that order). We're also using a coalesce here so that the concatenation doesn't result in nulls if either episode_narrative or event_narrative contain null values.

You could then use the ts_rank function in an ORDER BY clause to return results from most relevant to less.

SELECT …
ORDER BY ts_rank(ts, to_tsquery('english', 'tornado')) DESC;

So, this record is ranked higher in the search results:

state             | MISSISSIPPI
begin_date_time   | 2018-04-06 22:18:00
event_type        | Tornado
event_narrative   | This tornado touched down near the Jefferson Davis-Covington County line along Lucas Hollow Road. It continued southeast, crossing the
 county line. Some large limbs and trees were snapped and uprooted at this location. It then crossed Lucas Hollow Road again before crossing Leonard Road.
 A tornado debris signature was indicated on radar in these locations. The tornado uprooted and snapped many trees in this region. It also overturned a sm
all tractor trailer on Oakvale Road and caused some minor shingle damage to a home. After crossing Oakvale Road twice, the tornado lifted before crossing
Highway 35. The maximum winds in this tornado was 105mph and total path length was 2.91 miles. The maximum path width was 440 yards.
episode_narrative | A warm front was stretched across the region on April 6th. As a disturbance rode along this stalled front, it brought copious amounts
of rain to the region thanks to ample moisture in place. As daytime heating occurred, some storms developed which brought severe weather to the region.

Compared to this, where there is a match for "tornado" in episode_narrative but not event_narrative:

state             | NEBRASKA
begin_date_time   | 2018-06-06 18:10:00
event_type        | Hail
event_narrative   | Hail predominately penny size with some quarter size hail mixed in.
episode_narrative | Severe storms developed in the Nebraska Panhandle during the early evening hours of Jun
e 6th. As this activity tracked east, a broken line of strong to severe thunderstorms developed. Hail up to
 the size of ping pong balls, thunderstorm wind gusts to 70 MPH and a brief tornado touchdown were reported
. Heavy rain also fell leading to flash flooding in western Keith county.

Tip: ts_rank returns a floating-point value, so you could include the expression in your SELECT to see how these matches score. In my case I get around a 0.890 for the Mississippi event, and 0.243 for the Nebraska event.

Yes, You Can Keep Full-Text Search in Postgres

You can get even deeper and make your Postgres full-text search even more robust, by implementing features such as highlighting results, or writing your own custom dictionaries or functions. You could also look into enabling extensions such as unaccent (remove diacritic signs from lexemes) or pg_trgm (for fuzzy search). Speaking of extensions, those were just two of the extensions supported in Crunchy Bridge. We've built our managed cloud Postgres service such that you can dive right in and take advantage of all these Postgres features.

With all that said: as you can see, you don't need a very involved setup to get started. It's a good idea to try out whether you are just beginning to explore a full-text search solution, or even just reevaluating whether you need to go all out for a dedicated full-text search service, especially if you already have Postgres in your stack.

To be fair, Postgres doesn't have some search features that are available with platforms such as Elasticsearch. But a major advantage is that you won't have to maintain and sync a separate data store. If you don't quite need search at super scale, there might be more for you to gain by minimizing dependencies. Plus, the Postgres query syntax that you already know with the addition of some new functions and operators, can get you pretty far. Got any other questions or thoughts about full-text search with Postgres? We're happy to hear them on @crunchydata.

Logging Tips for Postgres, Featuring Your Slow Queries

Kat.Batuigas@crunchydata.com (Kat Batuigas) — Tue, 22 Jun 2021 05:00:00 EDT

In the last several months, we've featured simple yet powerful tools for optimizing PostgreSQL queries. We've walked through how the pg_stat_statements extension can show which queries are taking up the most time to run system-wide. We've also looked at how to use the EXPLAIN command to uncover query plans for individual queries.

You can get a lot out of those two, but you may have also wondered, "What about logs? Surely I can use Postgres' logs to help me find and track slow queries too?" Today we're going to take a look at a useful setting for your Postgres logs to help identify performance issues. We'll take a walk through integrating a third-party logging service such as LogDNA with Crunchy Bridge PostgreSQL and setting up logging so you're ready to start monitoring and watching for performance issues.

Logging your slow queries with log_min_duration_statement

With pg_stat_statements, you have to query the view periodically, while logging works a bit more behind the scenes. Pg_stat_statements also works well for analyzing queries in the aggregate, but you may want to see the exact queries that took a long time to run individually.

The log_min_duration_statement configuration parameter allows Postgres to do some of the work in finding slow queries. You decide the threshold, and the server logs the SQL statements that take at least that much time to run. In this example, I'll say 0.1 seconds, although figuring out that threshold is going to be specific to your case.

So if I have a database named us in a Crunchy Bridge cluster, I can run the following as a super user:

ALTER DATABASE us SET log_min_duration_statement = '100ms';

Your minimum duration might be higher based on the types of queries running. If you set it at too low of a number, you could end up with too much logging information than is actually helpful. Logs also take up space, so that's another thing to consider.

Crunchy Bridge clusters have syslog enabled as a log destination, so server logs can be exported to a third party logging service. Crunchy Bridge in particular can integrate with most providers that support syslog-ng.

When you use a service such as LogDNA, your account is typically associated with an access key for integrations. You may see this being referred to as an ingestion key, API key or token. LogDNA uses two types of API keys: ingestion keys for sending data to LogDNA, and service keys to send data from LogDNA.

LogDNA integration for Crunchy Bridge

The following steps assume that you've already set up a LogDNA account and are logged in via their web app. (They have a Quick Start guide for your reference. You only need to get to the step where you set up an organization and get an automatically-generated ingestion key.)

In your LogDNA account, open the Manage Organization page and retrieve your Ingestion Key under "API Keys."
On your Crunchy Bridge dashboard, select a cluster, and navigate to the Logging tab. Select "Add Log Destination."

In the "Create log destination" window, fill out the fields with these values:

Host: syslog-a.logdna.com
Port: 6514

Message template (includes your Ingestion Key):

<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} [logdna@48950 key=\"INSERT-YOUR-INGESTION-KEY-HERE\"] $MSG\n

For example, my message template looks like this:

<$PRI>1 $ISODATE $HOST $PROGRAM $PID $MSGID [logdna@48950 key=\"17dcbad54b86a51585fdfa403d801668\"] $MSG\n

Description: A brief description for your reference.

Once you save this log destination, your Crunchy Bridge Postgres cluster is added as an ingestion source for your organization in LogDNA. There's nothing else you need to do in Bridge nor LogDNA to enable the integration.

Spotting slow queries in your Bridge logs

Head over to the official Crunchy Bridge docs for more details on what the data will look like in LogDNA. For this example, I have queries running against United States placenames data in a geonames table. Here, we'll take a look at an extract from my LogDNA viewer that shows the SQL statements that were logged:

Apr 13 21:19:25 c6p3j5s7evattiqewf7al52hzu postgres info [16-1] 2021-04-14 01:19:25.389 GMT [3086911][client backend][4/0][0] [user=postgres,db=us,app=psql] LOG: duration: 226.904 ms statement: SELECT name, type, lon, lat FROM geonames WHERE name LIKE 'Spring%';
Apr 13 21:19:42 c6p3j5s7evattiqewf7al52hzu postgres info [17-1] 2021-04-14 01:19:42.418 GMT [3086911][client backend][4/0][0] [user=postgres,db=us,app=psql] LOG: duration: 120.019 ms statement: SELECT name, type, lon, lat FROM geonames WHERE name LIKE 'Middletown%';
Apr 13 21:19:55 c6p3j5s7evattiqewf7al52hzu postgres info [18-1] 2021-04-14 01:19:55.433 GMT [3086911][client backend][4/0][0] [user=postgres,db=us,app=psql] LOG: duration: 134.442 ms statement: SELECT name, type, lon, lat FROM geonames WHERE name LIKE 'Spring%';
Apr 13 21:20:05 c6p3j5s7evattiqewf7al52hzu postgres info [19-1] 2021-04-14 01:20:05.110 GMT [3086911][client backend][4/0][0] [user=postgres,db=us,app=psql] LOG: duration: 136.167 ms statement: SELECT name, type, lon, lat FROM geonames WHERE name LIKE 'Tampa Bay%';

c6p3j5s7evattiqewf7al52hzu is the Crunchy Bridge host, the app/process is postgres, and queries that ran for at least 100ms are logged at the info level. The Bridge logs are updated in LogDNA in pretty much real time.

Log messages from Crunchy Bridge Postgres also include a log line prefix based on this format:

%m [%p][%b][%v][%x] %q[user=%u,db=%d,app=%a]

Let's break that down a little bit:

%m: Time stamp with milliseconds - 2021-04-14 01:19:25.389 GMT
[%p]: Process ID - [3086911]
[%b]: Backend type - [client backend]
[%v]: Virtual transaction ID (backendID/localXID) - [4/0]
[%x]: Transaction ID (0 if none is assigned) - [0]
%q: Produces no output, but tells non-session processes to stop at this point in the string; ignored by session processes
%u, %d, %a: User name, database name, application name - [user=postgres,db=fakedb,app=[unknown]]

The default log line prefix for Postgres includes just the time stamp and process ID. But with Bridge log messages, you immediately see more information about that particular process. Your logs are more readable right off the bat.

I also just ran these queries in psql but you could have them running via a different client or from your own application. The log messages would indicate as such too.

Filtering and views in LogDNA

There's plenty you can do in the LogDNA app in terms of searching, parsing, and even visualizing logs, but for now let's try a couple of simple log management features in the web app.

Filtering

By default, the LogDNA viewer displays all logs from all your organization's ingestion sources. At the top of the page, you can quickly filter on:

Hosts (Ingestion source)
Apps (App or program), e.g. postgres, syslog-ng
Levels (e.g. INFO, NOTICE, DEBUG, WARN, ERR)

On the bottom of the page is a search field - any search terms you enter here can be saved together with the page filters in a new view, for your convenience.

Views

As an example, I've created a basic view to show slow queries:

I've filtered on the Crunchy Bridge host, postgres as the app, and the info level. Let's also say I only want SELECT statements to be included in this view. "LOG: duration: " SELECT as my search query is sufficient to capture those logged SQL queries.

No, I don't plan on watching my views like this guy… maybe.

You can get even more granular with your views. The official docs on search in LogDNA are helpful, and for a deeper dive into LogDNA's approach to search, check out this blog post on regex vs. search terms. To learn more about working with and viewing logs, visit the official LogDNA docs.

OK, where to from here?

I've only run pretty basic queries against a test database for this example. A more complex, production-level system has more requirements and so your logging and monitoring setup might be much more involved. But hopefully this shows you how it takes not much work at all to get this kind of integration up and running with Crunchy Bridge.

So, I appear to be seeing some slowness with a particular statement. Great, now what? This could be a point where I check pg_stat_statements to see how the parameterized version of this query (SELECT name, type, lon, lat FROM geonames WHERE name LIKE $1) has performed in terms of total or even average execution time. Or, I could dive into EXPLAIN to see the query plans and have a better idea for how the query could be optimized.

Figuring out why queries are slow and what should be done to fix them is a whole other layer, so we'll fast forward a bit. Let's say I decide to add just a standard B-tree index for the geonames.name column.

CREATE INDEX on geonames(name);

Adding this index dropped those same queries down from >100ms to no more than several milliseconds, according to EXPLAIN ANALYZE. I'd probably keep an eye on the logs as well as pg_stat_statements to see if this optimization holds up well over time. And even if it doesn't, or if new trends start to come up, a) I'll be able to see that pretty easily, and b) I know I have the tools as well as the data to dig into. If nothing else, it's always possible that I just need to be writing better queries...

Something I didn't do in this example but I learned about from my colleague Craig Kerstiens is log_temp_files. There is a work_mem configuration that sets how much memory Postgres can use on a given process for a query. When that amount isn't enough, Postgres begins to spill to disk, i.e. write temp files. When log_temp_files is set to 0, the server logs each time a temp file is written. If you start to see a lot of that in your logs, that might indicate that work_mem needs to be increased.

The folks at LogDNA have done a great job of spelling out why you may want to manage your logs in a central place, and how logging can be part of a more proactive approach to incident prevention. Logging is obviously for more than just fixing slow queries. But, this could be part of a query optimization strategy that works well for you. Thoughts? Let us know on Twitter @crunchydata.

Explaining Your Postgres Query Performance

Kat.Batuigas@crunchydata.com (Kat Batuigas) — Mon, 22 Mar 2021 05:00:00 EDT

In a previous post, I talked about pg_stat_statements as a tool for helping direct your query optimization efforts. Now let's say you've identified some queries you want to look into. The EXPLAIN command helps you look even closer into an individual query. If you're already proficient in EXPLAIN, great! Read on for an easy refresher. If you're less familiar with it, this will be a (hopefully) gentle introduction on what insights it might help provide.

I'm going to demonstrate simple EXPLAIN usage with a Geonames country dataset for the United States. There are over 2 million records in this set, and I'm hoping that's a good table size to clearly demonstrate how EXPLAIN is useful.

For those newer to SQL, it's considered a declarative language (although modern DBMS's have enhanced SQL to accommodate procedural programming too). You tell the database server what you want, and it does the job of determining the best way to return you those results. In PostgreSQL, the planner puts together a plan for executing a query based on "query structure and the properties of the data," among other factors. EXPLAIN lets you see that plan. While EXPLAIN is a Postgres-specific command, other RDBMS's will have similar tools.

Let's get to some examples to better see what EXPLAIN does.

Geonames for United States

If you want to follow along with the example, I'm using Postgres 13 and connecting to the database via psql on WSL2. We'll only be using simple query examples, so even if you're still starting out with Postgres you'll be able to easily understand what the queries are meant to return.

You can grab the US.zip archive from their download server.

Here's the list of data attributes, but I've grabbed only a few of them to load into my database. (If you need to work with a smaller dataset, there are more countries available in the Geonames server.) I've imported the data into a table defined like so:

CREATE TABLE us_geonames (
	id integer,
	name text,
	lat float8,
	lon float8,
	type text
);

How to use EXPLAIN

To use the EXPLAIN command, you tack on EXPLAIN before the statement you want to run:

EXPLAIN SELECT name FROM customers WHERE id = 1;

This will return the estimated plan and cost, in plain text by default.

You can also add some options that return slightly different output. A common one is ANALYZE (which I'll use throughout this post). In addition to the estimated plan and statistics, it will go ahead and run the query and give you the actual run statistics. Or, you can also change the output format to, say, JSON.

(Tip: Other Postgres clients such as pgAdmin can also show you the query plan in a graphical format.)

EXPLAIN ANALYZE will actually run the query, so be careful with updates or deletes! In those cases, consider not using ANALYZE, or you might perhaps wrap the entire statement in a transaction that you can roll back.

Querying with `text` columns

I'm starting out with a query that has a WHERE clause (in this case I know this should return exactly one row):

us=# EXPLAIN ANALYZE SELECT * from us_geonames WHERE name = 'Tampa International Airport';

														QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------
 Gather  (cost=1000.00..30062.35 rows=7888 width=84) (actual time=37.164..192.388 rows=1 loops=1)
   Workers Planned: 3
   Workers Launched: 3
   ->  Parallel Seq Scan on us_geonames  (cost=0.00..28273.55 rows=2545 width=84) (actual time=93.085..130.232 rows=0 loops=4)
         Filter: (name = 'Tampa International Airport'::text)
         Rows Removed by Filter: 559758
 Planning Time: 0.052 ms
 Execution Time: 192.413 ms
(8 rows)

I won't go into the details of every specific thing we see in the EXPLAIN output, but let's walk through some general concepts:

The tree structure. The indented statements that start with an → mean that the following operation is a child node of the outer/parent node. For my query, the topmost plan node is a Gather, and Parallel Sequential Scan is a child node of Gather.
The output from the child node is fed as input to the outer node. For our purposes, don't worry too much about how that works exactly. The point is that the tree structure helps you immediately tell that there are multiple steps or levels to the execution plan.
Planning and execution times. I used the ANALYZE option, so we also see how much time it took to generate the query plan, and how much time was spent executing the query (this doesn't include the planning time).
Estimated statistics. The first set of parentheses on each node line or summary provides estimates made from the query plan.
The topmost or root node gives totals that include those from the child nodes. Each child node or step will have its own statistics, including any child nodes it may also have.
The cost estimate isn't associated with actual units, so don't think of it in terms of time or any specific resources. The plan also provides the estimated number of rows that will be returned or match the condition.
Actual run statistics. With ANALYZE, I get a second set of parentheses. This provides the actual time spent (in milliseconds) in that node, as well as the actual rows returned. Additionally, loops will show you if that particular node was executed more than once.
In our example, you may have noticed that the estimated 7888 rows is a little off from the actual rows returned. This is something you might look out for or might be a flag.

If you were curious about the "Workers" part of the output: this is when the server is able to carry out parallel queries to help speed things up. There are settings that can be controlled that affect parallel queries (for example, I have max_parallel_workers set to 4). You'll see Gather or Gather Merge plan nodes with parallel queries.

So the gist of the above plan is, there are sequential scans (i.e. scan the table, row by row from first to last) that are happening in parallel, searching for matches on the filter criteria name = 'Tampa International Airport'. The work to scan the entire table is essentially divvied up between 3 asynchronous threads. The output from the parallel scans is gathered to produce the result set.

There's only one Tampa International Airport, so it seems like a waste of time to scan each row to find the matching record (and we might not know if that record is towards the top of the table, or nearer the end). The planner at least determined that parallel queries would help. I didn't try it "nonparalleled", though I'd imagine that would go even slower.

We'll try a few more examples. How about pattern matching?

us=# EXPLAIN ANALYZE SELECT * FROM us_geonames WHERE name like '%Tampa%';

													QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------
 Gather  (cost=1000.00..31962.05 rows=217 width=44) (actual time=22.871..209.757 rows=467 loops=1)
   Workers Planned: 3
   Workers Launched: 3
   ->  Parallel Seq Scan on us_geonames  (cost=0.00..30940.35 rows=70 width=44) (actual time=56.135..164.358 rows=117 loops=4)
         Filter: (name ~~ '%Tampa%'::text)
         Rows Removed by Filter: 559641
 Planning Time: 0.141 ms
 Execution Time: 209.810 ms

This turned out largely the same as the above, except this time the Filter criteria is for name values that contain the string "Tampa". It also looks like the estimated vs. actual rows is less skewed.

Let's see a sort:

us=# EXPLAIN ANALYZE SELECT * FROM us_geonames ORDER BY type;

													QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------
 Gather Merge  (cost=122640.75..379045.89 rows=2166804 width=44) (actual time=669.975..1752.805 rows=2239032 loops=1)
   Workers Planned: 3
   Workers Launched: 3
   ->  Sort  (cost=121640.71..123446.38 rows=722268 width=44) (actual time=597.224..747.053 rows=559758 loops=4)
         Sort Key: type
         Sort Method: external merge  Disk: 37016kB
         Worker 0:  Sort Method: external merge  Disk: 34768kB
         Worker 1:  Sort Method: external merge  Disk: 28584kB
         Worker 2:  Sort Method: external merge  Disk: 26688kB
         ->  Parallel Seq Scan on us_geonames  (cost=0.00..29134.68 rows=722268 width=44) (actual time=0.025..80.148 rows=559758 loops=4)
 Planning Time: 0.049 ms
 Execution Time: 1883.467 ms
(12 rows)

Looks similar to the first plan, except there is now a Sort node with the sequential scans as child node. Gather Merge is used with sorted data.

And let's try changing things up a bit so that we're grouping, using an aggregate function, and sorting:

us=# EXPLAIN ANALYZE SELECT type, COUNT(*) FROM us_geonames GROUP BY 1 ORDER BY 2;

													QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=33818.27..33818.66 rows=156 width=12) (actual time=529.629..532.138 rows=416 loops=1)
   Sort Key: (count(*))
   Sort Method: quicksort  Memory: 44kB
   ->  Finalize GroupAggregate  (cost=33753.31..33812.59 rows=156 width=12) (actual time=528.715..531.991 rows=416 loops=1)
         Group Key: type
         ->  Gather Merge  (cost=33753.31..33808.69 rows=468 width=12) (actual time=528.705..531.654 rows=1250 loops=1)
               Workers Planned: 3
               Workers Launched: 3
               ->  Sort  (cost=32753.27..32753.66 rows=156 width=12) (actual time=464.698..464.732 rows=312 loops=4)
                     Sort Key: type
                     Sort Method: quicksort  Memory: 40kB
                     Worker 0:  Sort Method: quicksort  Memory: 39kB
                     Worker 1:  Sort Method: quicksort  Memory: 39kB
                     Worker 2:  Sort Method: quicksort  Memory: 39kB
                     ->  Partial HashAggregate  (cost=32746.02..32747.58 rows=156 width=12) (actual time=464.298..464.383 rows=312 loops=4)
                           Group Key: type
                           Batches: 1  Memory Usage: 77kB
                           Worker 0:  Batches: 1  Memory Usage: 77kB
                           Worker 1:  Batches: 1  Memory Usage: 77kB
                           Worker 2:  Batches: 1  Memory Usage: 77kB
                           ->  Parallel Seq Scan on us_geonames  (cost=0.00..29134.68 rows=722268 width=4) (actual time=0.004..117.760 rows=559758 loops=4)
 Planning Time: 0.066 ms
 Execution Time: 532.207 ms
(23 rows)

As you might have guessed, this plan is more complex than the previous ones. Basically, the query has to aggregate and sort rows multiple times to return the desired results. We won't worry about HashAggregate or GroupAggregate in this post, but check out this glossary with some quick explanations for operations or other terms you might see with EXPLAIN. This is just another demonstration of the tree structure and how each step relates to the next.

Use EXPLAIN ANALYZE to see query performance after indexing

So there were at least a couple of things that we saw which indicate that the table might benefit from an index or two: 1) that sequential scan to return just one row, and 2) estimated number of rows being quite a bit off from actual returned rows.

Figuring out when to index, and which things to index is an entire discussion on its own. For this post let's just go ahead and add two indexes, and then check to see if EXPLAIN ANALYZE tells us anything new.

CREATE INDEX us_geonames_name_idx on us_geonames (name);

CREATE INDEX us_geonames_type_idx on us_geonames (type);

us=# EXPLAIN ANALYZE SELECT * from us_geonames WHERE name = 'Tampa International Airport';

													QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------

 Bitmap Heap Scan on us_geonames  (cost=4.52..47.91 rows=11 width=44) (actual time=0.035..0.036 rows=1 loops=1)
   Recheck Cond: (name = 'Tampa International Airport'::text)
   Heap Blocks: exact=1
   ->  Bitmap Index Scan on us_geonames_name_idx  (cost=0.00..4.51 rows=11 width=0) (actual time=0.030..0.030 rows=1 loops=1)
         Index Cond: (name = 'Tampa International Airport'::text)
 Planning Time: 0.214 ms
 Execution Time: 0.074 ms
(7 rows)

We can see right away that after adding the index, the planner pretty much went "Oh, let's actually not scan the table from top to bottom." With the Bitmap Index Scan, Postgres uses the index we added for the name column to first find out the location of any matching rows, and then retrieves those rows in the Bitmap Heap Scan.

You might notice that the planning time is longer than the execution time. Still, it's that many times faster than the original plan.

us=# EXPLAIN ANALYZE SELECT * FROM us_geonames WHERE name like '%Tampa%';

													QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------
 Gather  (cost=1000.00..31962.05 rows=217 width=44) (actual time=25.249..229.884 rows=467 loops=1)
   Workers Planned: 3
   Workers Launched: 3
   ->  Parallel Seq Scan on us_geonames  (cost=0.00..30940.35 rows=70 width=44) (actual time=33.100..183.968 rows=117 loops=4)
         Filter: (name ~~ '%Tampa%'::text)
         Rows Removed by Filter: 559641
 Planning Time: 0.123 ms
 Execution Time: 229.944 ms
(8 rows)

Isn't that interesting? This plan is essentially the same, meaning that the planner chose not to use the index. Just because we added an index doesn't mean it'll be used! In this scenario, it has to do with the pattern we're trying to search on -- we just created a B-Tree index (the default kind), but it's not actually helpful for this kind of query.

us=# EXPLAIN ANALYZE SELECT * FROM us_geonames ORDER BY type;

													QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------
 Index Scan using us_geonames_type_idx on us_geonames  (cost=0.43..128729.96 rows=2239032 width=44) (actual time=0.054..898.211 rows=2239032 loops=1)
 Planning Time: 0.137 ms
 Execution Time: 1007.902 ms
(3 rows)

This plan doesn't contain any child nodes: it straight up uses the index (whew) for the type column to do the sorting.

Finally:

us=# EXPLAIN ANALYZE SELECT type, COUNT(*) FROM us_geonames GROUP BY 1 ORDER BY 2;


														QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=28286.27..28286.66 rows=156 width=12) (actual time=269.117..269.259 rows=416 loops=1)
   Sort Key: (count(*))
   Sort Method: quicksort  Memory: 44kB
   ->  Finalize GroupAggregate  (cost=1000.49..28280.58 rows=156 width=12) (actual time=59.070..269.029 rows=416 loops=1)
         Group Key: type
         ->  Gather Merge  (cost=1000.49..28275.90 rows=624 width=12) (actual time=59.057..268.624 rows=619 loops=1)
               Workers Planned: 4
               Workers Launched: 4
               ->  Partial GroupAggregate  (cost=0.43..27201.52 rows=156 width=12) (actual time=1.189..181.774 rows=124 loops=5)
                     Group Key: type
                     ->  Parallel Index Only Scan using us_geonames_type_idx on us_geonames  (cost=0.43..24401.17 rows=559758 width=4) (actual time=0.036..90.309 rows=447806 loops=5)
                           Heap Fetches: 0
 Planning Time: 0.080 ms
 Execution Time: 269.316 ms
(14 rows)

This one also looks similar to the pre-index plan, although it does at least use the index, and we see that the execution time is basically cut in half.

EXPLAIN (ANALYZE) is a friend that tells it like it is

EXPLAIN can get kind of intimidating, especially if you're like me (not a DBA nor an advanced Postgres user). But if you stick to some core ideas, you'll eventually become more adept at processing this information to understand the potential prickly bits in your queries:

The query plan is a tree structure that shows you each inner or child node that feeds into an outer or root node.
EXPLAIN alone shows estimates. With ANALYZE, you'll actually run the query and see the time it took to create the query plan, plus the time it took to execute the query according to that plan.
Easy things to look out for to try and diagnose a query are estimated vs actual rows, and perhaps table scans. Sequential scans are not always bad nor slow (it depends on the query, e.g. how restrictive a WHERE clause is), but there might be an opportunity there to optimize.
I should of course note that the query planner relies on statistics collected by Postgres about the system. Those might need looking at and updating if you're not really seeing what you'd expect.

This post showed you how to jump into the EXPLAIN command with some simple examples. For an even more hands-on experience, we have a 15-minute lesson in our Learning Portal--this time with geospatial data and indexes. And finally, I also recommend checking out my colleague Jonathan Katz's post on BRIN indexes for another look into how EXPLAIN helps shine a light on query performance.