Martin Davis | CrunchyData Blog

SVG Images from Postgres

Martin.Davis@crunchydata.com (Martin Davis) — Tue, 30 May 2023 09:00:00 EDT

PostGIS excels at storing, manipulating and analyzing geospatial data. At some point it's usually desired to convert raw spatial data into a two-dimensional representation to utilize the integrative capabilities of the human visual cortex. In other words, to see things on a map.

PostGIS is a popular backend for mapping technology, so there are many options to choose from to create maps. Data can be rendered to a raster image using a web map server like GeoServer or MapServer; it can be converted to GeoJSON or vector tiles via servers such as pg_featureserv and pg_tileserv and then shipped to a Web browser for rendering by a library such as OpenLayers, MapLibre or Leaflet; or a GIS application such as QGIS can connect to the database and create richly-styled maps from spatial queries.

What these options have in common is that they require external tools which need to be installed, configured and maintained in a separate environment. This can introduce unwanted complexity to a geospatial architecture.

This post presents a simple way to generate maps entirely within the database, with no external infrastructure required.

SVG for the win

A great way to display vector data is to use the Scalable Vector Graphic (SVG) format. It provides rich functionality for displaying and styling geometric shapes. SVG is widely supported by web browsers and other tools.

By including CSS and Javascript it's possible to add advanced styling, custom popups, dynamic behaviour and interaction with other web page elements.

Introducing `pg-svg`

Generating SVG "by hand" is difficult. It requires detailed knowledge of the SVG specification, and constructing a complex text format in SQL is highly error-prone. While PostGIS has had the function ST_AsSVG for years, it only produces the SVG path data attribute value. Much more is required to create a fully-styled SVG document.

The PL/pgSQL library pg-svg solves this problem! It makes it easy to convert PostGIS data into styled SVG documents. The library provides a simple API as a set of PL/pgSQL functions which allow creating an SVG document in a single SQL query. Best of all, this installs with a set of functions, nothing else required!

Example map of US high points

The best way to understand how pg-svg works is to see an example. We'll create an SVG map of the United States showing the highest point in each state. The map has the following features:

All 50 states are shown, with Alaska and Hawaii transformed to better fit the map
States are labeled, and filled with a gradient
High points are shown at their location by triangles whose color and size indicate the height of the high point.
Tooltips provide more information about states and highpoints.

The resulting map looks like this (to see tooltips open the raw image):

The SQL query to generate the map is here. It can be downloaded and run using psql:

psql -A -t -o us-highpt.svg  < us-highpt-svg.sql

The SVG output us-highpt.svg can be viewed in any web browser.

How it Works

Let's break the query down to see how the data is prepared and then rendered to SVG. A dataset of US states in geodetic coordinate system (WGS84, SRID = 4326) is required. We used the Natural Earth states and provinces data available here. It is loaded into a table ne.admin_1_state_prov with the following command:

shp2pgsql -c -D -s 4326 -i -I ne_10m_admin_1_states_provinces.shp ne.admin_1_state_prov | psql

The query uses the SQL WITH construct to organize processing into simple, modular steps. We'll describe each one in turn.

Select US state features

First, the US state features are selected from the Natural Earth boundaries table ne.admin_1_state_prov.

us_state AS (SELECT name, abbrev, postal, geom
  FROM ne.admin_1_state_prov
  WHERE adm0_a3 = 'USA')

Make a US state map

Next, the map is made more compact by realigning the far-flung states of Alaska and Hawaii.
This is done using PostGIS affine transformation functions. The states are made more proportionate using ST_Scale, and moved closer to the lower 48 using ST_Translate. The scaling is done around the location of the state high point, to make it easy to apply the same transformation to the high point feature.

,us_map AS (SELECT name, abbrev, postal,
    -- transform AK and HI to make them fit map
    CASE WHEN name = 'Alaska' THEN
      ST_Translate(ST_Scale(
        ST_Intersection( ST_GeometryN(geom,1), 'SRID=4326;POLYGON ((-141 80, -141 50, -170 50, -170 80, -141 80))'),
        'POINT(0.5 0.75)', 'POINT(-151.007222 63.069444)'::geometry), 18, -17)
    WHEN name = 'Hawaii' THEN
      ST_Translate(ST_Scale(
        ST_Intersection(geom, 'SRID=4326;POLYGON ((-161 23, -154 23, -154 18, -161 18, -161 23))'),
        'POINT(3 3)', 'POINT(-155.468333 19.821028)'::geometry), 32, 10)
    ELSE geom END AS geom
  FROM us_state)

High Points of US states

Data for the highest point in each state is provided as an inline table of values:

,high_pt(name, state, hgt_m, hgt_ft, lon, lat) AS (VALUES
 ('Denali',              'AK', 6198, 20320,  -151.007222,63.069444)
,('Mount Whitney',       'CA', 4421, 14505,  -118.292,36.578583)
...
,('Britton Hill',        'FL',  105,   345,  -86.281944,30.988333)
)

Prepare High Point symbols

The next query does several things:

translates the lon and lat location for Alaska and Hawaii high points to match the transformation applied to the state geometry
computes the symHeight attribute for the height of the high point triangle symbol
assigns a fill color value to each high point based on the height
uses ORDER BY to sort the high points by latitude, so that their symbols overlap correctly when rendered

,highpt_shape AS (SELECT name, state, hgt_ft,
    -- translate high points to match shifted states
    CASE WHEN state = 'AK' THEN lon + 18
      WHEN state = 'HI' THEN lon + 32
      ELSE lon END AS lon,
    CASE WHEN state = 'AK' THEN lat - 17
      WHEN state = 'HI' THEN lat + 10
      ELSE lat END AS lat,
    (2.0 * hgt_ft) / 15000.0 + 0.5 AS symHeight,
    CASE WHEN hgt_ft > 14000 THEN '#ffffff'
         WHEN hgt_ft >  7000 THEN '#aaaaaa'
         WHEN hgt_ft >  5000 THEN '#ff8800'
         WHEN hgt_ft >  2000 THEN '#ffff44'
         WHEN hgt_ft >  1000 THEN '#aaffaa'
                             ELSE '#558800'
    END AS clr
  FROM high_pt ORDER BY lat DESC)

Generate SVG elements

The previous queries transformed the raw data into a form suitable for rendering.
Now we get to see pg-svg in action! The next query generates the SVG text for each output image element, as separate records in a result set called shapes.

The SVG elements are generated in the order in which they are drawn - states and labels first, with high-point symbols on top. Let's break it down:

SVG for states

The first subquery produces SVG shapes from the state geometries. The svgShape function produces an SVG shape element for any PostGIS geometry. It also provides optional parameters supporting other capabilities of SVG. Here title specifies that the state name is displayed as a tooltip, and style specifies the styling of the shape. Styling in SVG can be provided using properties defined in the Cascaded Style Sheets (CSS) specification. pg-svg provides the svgStyle function to make it easy to specify the names and values of CSS styling properties.

Note that the fill property value is a URL instead of a color specifier. This refers to an SVG gradient fill which is defined later.

The state geometry is also included in the subquery result set, for reasons which will be discussed below.

,shapes AS (
  -- State shapes
  SELECT geom, svgShape( geom,
    title => name,
    style => svgStyle(  'stroke', '#ffffff',
                        'stroke-width', 0.1::text,
                        'fill', 'url(#state)',
                        'stroke-linejoin', 'round' ) )
    svg FROM us_map

SVG for state labels

Labels for state abbreviations are positioned at the point produced by the ST_PointOnSurface function. (Alternatively, ST_MaximumInscribedCircle could be used.) The SVG is generated by the svgText function, using the specified styling.

  UNION ALL
  -- State names
  SELECT NULL, svgText( ST_PointOnSurface( geom ), abbrev,
    style => svgStyle(  'fill', '#6666ff', 'text-anchor', 'middle', 'font', '0.8px sans-serif' ) )
    svg FROM us_map

SVG for high point symbols

The high point features are displayed as triangular symbols. We use the convenient svgPolygon function with a simple array of ordinates specifying a triangle based at the high point location, with height given by the previously computed svgHeight column. The title is provided for a tooltip, and the styling uses the computed clr attribute as the fill.

  UNION ALL
  -- High point triangles
  SELECT NULL, svgPolygon( ARRAY[ lon-0.5, -lat, lon+0.5, -lat, lon, -lat-symHeight ],
    title => name || ' ' || state || ' - ' || hgt_ft || ' ft',
    style => svgStyle(  'stroke', '#000000',
                        'stroke-width', 0.1::text,
                        'fill', clr  ) )
    svg FROM highpt_shape
)

Produce final SVG image

The generated shape elements need to be wrapped in an <svg> document element. This is handled by the svgDoc function.

The viewable extent of the SVG data needs to be provided by the viewbox parameter. The most common case is to display all of the rendered data. An easy way to determine this is to apply the PostGIS ST_Exrtent aggregate function to the input data (this is why we included the geom column as well as the svg text column). We can include a border by enlarging the extent using the ST_Expand function. The function svgViewBox converts the PostGIS geometry for the extent into SVG format.

We also include a definition for an SVG linear gradient to be used as the fill style for the state features.

SELECT svgDoc( array_agg( svg ),
    viewbox => svgViewbox( ST_Expand( ST_Extent(geom), 2)),
    def => svgLinearGradient('state', '#8080ff', '#c0c0ff')
  ) AS svg FROM shapes;

The output from svgDoc is a text value which can be used anywhere that SVG is supported.

More to Explore

We've shown how the pg-svg SQL function library lets you easily generate map images from PostGIS data right in the database. This can be used as a simple ad-hoc way of visualizing spatial data. Or, it could be embedded in a larger system to automate repetitive map generation workflows.

Although SVG is a natural fit for vector data, there may be situations where producing a map as a bitmap (raster) image makes sense.
For a way of generating raster maps right in the database see this PostGIS Day 2022 presentation. This would be especially appealing where the map is displaying data stored using PostGIS raster data. It would also be possible to combine vector and raster data into a hybrid SVG/image output.

Although we've focussed on creating maps of geospatial data, SVG is often used for creating other kinds of graphics. For examples of using it to create geometric and mathematical designs see the pg-svg demo folder. Here's an image of a Lissajous knot generated by this SQL.

You could even use pg-svg to generate charts of non-spatial data (although this would be better handled by a more task-specific API).

Let us know if you find pg-svg useful, or if you have ideas for improving it!

Temporal Filtering in pg_featureserv with CQL

Martin.Davis@crunchydata.com (Martin Davis) — Fri, 10 Feb 2023 10:00:00 EST

In a previous post we announced the CQL filtering capability in pg_featureserv. It provides powerful functionality for attribute and spatial querying of data in PostgreSQL and PostGIS.

Another important datatype which is often present in datasets is temporal. Temporal datasets contain attributes which are dates or timestamps. The CQL standard defines some special-purpose syntax to support temporal filtering. This allows pg_featureserv to take advantage of the extensive capabilities of PostgreSQL for specifying queries against time-valued attributes. This post in the CQL series will show some examples of temporal filtering in pg_featureserv.

CQL Temporal filters

Temporal filtering in CQL is provided using temporal literals and conditions.

Temporal literal values may be dates or timestamps:

2001-01-01
2010-04-23T01:23:45

Note: The temporal literal syntax is based on an early version of the OGC API Filter and CQL standard. The current draft CQL standard has a different syntax: DATE('1969-07-20') and TIMESTAMP('1969-07-20T20:17:40Z'). It also supports intervals: INTERVAL('1969-07-16', '1969-07-24'). A subsequent version of pg_featureserv will support this syntax as well.

Temporal conditions allow time-valued properties and literals to be compared via the standard boolean comparison operators <,>,<=,>=,=,<> and the BETWEEN..AND operator:

start_date >= 2001-01-01
event_time BETWEEN 2010-04-22T06:00 AND 2010-04-23T12:00

The draft CQL standard provides dedicated temporal operators, such as T_AFTER, T_BEFORE, T_DURING, etc. A future version of pg_featureserv will likely provide these operators.

Publishing Historical Tropical Storm tracks

We'll demonstrate temporal filters using a dataset with a strong time linkage: tracks of tropical storms (or hurricanes). There is a dataset of Historical Tropical Storm Tracks available here.

The data requires some preparation. It is stored as a set of records of line segments representing 6-hour long sections of storm tracks. To provide simpler querying we will model the data using a single record for each storm, with a line geometry showing the entire track and attributes for the start and end time for the track.

The data is provided in Shapefile format. As expected for a worldwide dataset, it is in the WGS84 geodetic coordinate system (lat/long). In PostGIS this common Spatial Reference System is assigned an identifier (SRID) of 4326.

The PostGIS shp2pgsql utility can be used to load the dataset into a spatial table called trop_storm_raw. The trop_storm_raw table is a temporary staging table allowing the raw data to be loaded and made available for the transformation phase of data preparation.

shp2pgsql -c -D -s 4326 -i -I -W LATIN1 "Historical Tropical Storm Tracks.shp" public.trop_storm_raw | psql -d database

The options used are:

-c - create a new table
-D - use PostgreSQL dump format to load the data
-s - specify the SRID of 4326
-i - use 32-bit integers
-I - create a GIST index on the geometry column (this is not strictly necessary, since this is just a temporary staging table)
-W - specifies the encoding of the input attribute data in the DBF file

Next, create the table having the desired data model:

CREATE TABLE public.trop_storm (
    btid int PRIMARY KEY,
    name text,
    wind_kts numeric,
    pressure float8,
    basin text,
    time_start timestamp,
    time_end timestamp,
    geom geometry(MultiLineString, 4326)
);

It's good practice to add comments to the table and columns. These will be displayed in the pg_featureserv Web UI.

COMMENT ON TABLE public.trop_storm IS 'This is my spatial table';
COMMENT ON COLUMN public.trop_storm.geom IS 'Storm track LineString';
COMMENT ON COLUMN public.trop_storm.name IS 'Name assigned to storm';
COMMENT ON COLUMN public.trop_storm.btid IS 'Id of storm';
COMMENT ON COLUMN public.trop_storm.wind_kts IS 'Maximum wind speed in knots';
COMMENT ON COLUMN public.trop_storm.pressure IS 'Minumum pressure in in millibars';
COMMENT ON COLUMN public.trop_storm.basin IS 'Basin in which storm occured';
COMMENT ON COLUMN public.trop_storm.time_start IS 'Timestamp of storm start';
COMMENT ON COLUMN public.trop_storm.time_end IS 'Timestamp of storm end';

Now the power of SQL can be used to transform the raw data into the simpler data model. The track sections can be combined into single tracks with a start and end time using the following query.

The original data represents the track sections as MultiLineStrings with single elements. The element is extracted using ST_GeometryN so that the result of aggregating them using ST_Collect is a MultiLineString, not a GeometryCollection. (An alternative is to aggregate into a GeometryCollection and use ST_CollectionHomogenize to reduce it to a MultiLineString.)
The final ST_Multi ensures that all tracks are stored as MultiLineStrings, as required by the type constraint on the geom column.
the filter condition time_end - time_start < '1 year'::interval removes tracks spanning the International Date Line.

WITH data AS (
 SELECT btid, name, wind_kts, pressure, basin, geom,
    make_date(year::int, month::int, day::int) + ad_time::time AS obs_time
 FROM trop_storm_raw ORDER BY obs_time
),
tracks AS (
  SELECT btid,
    MAX(name) AS name,
    MAX(wind_kts) AS wind_kts,
    MAX(pressure) AS pressure,
    MAX(basin) AS basin,
    MIN(obs_time) AS time_start,
    MAX(obs_time) AS time_end,
    ST_Multi( ST_LineMerge( ST_Collect( ST_GeometryN(geom, 1)))) AS geom
  FROM data GROUP BY btid
)
INSERT INTO trop_storm
SELECT * FROM tracks WHERE time_end - time_start < '1 year'::interval;

This is a small dataset, and pg_featureserv does not require one, but as per best practice we can create a spatial index on the geometry column:

CREATE INDEX trop_storm_gix ON public.trop_storm USING GIST ( geom );

Once the trop_storm table is created and populated, it can be published in pg_featureserv. Issuing the following request in a browser shows the feature collection in the Web UI:

http://localhost:9000/collections.html

http://localhost:9000/collections/public.trop_storm.html

The dataset can be viewed using pg_featureserv's built-in map viewer (note that to see all 567 records displayed it is probably necessary to increase the limit on the number of response features):

http://localhost:9000/collections/public.trop_storm/items.html?limit=1000

Querying by Time Range

That's a lot of storm tracks. It would be easier to visualize a smaller number of tracks. A natural way to subset the data is by querying over a time range. Let's retrieve the storms between the start of 2005 and the end of 2009. This is done by adding a filter parameter with a CQL expression against the dataset temporal property time_start (storms typically do not span the start of years). To query values lying between a range of times it is convenient to use the BETWEEN operator. The filter condition is time_start BETWEEN 2005-01-01 AND 2009-12-31. The full request is:

http://localhost:9000/collections/public.trop_storm/items.html?filter=time_start BETWEEN 2005-01-01 AND 2009-12-31&limit=100

Submitting this query produces a result with 68 tracks:

Querying by Time and Space

Temporal conditions can be combined with other kinds of filters. For instance, we can execute a spatio-temporal query by using a temporal condition along with a spatial condition. In this example, we query the storms which occurred in 2005 and after in Florida. The temporal condition is expressed as time_start > 2005-01-01.

The spatial condition uses the INTERSECTS predicate to test whether the line geometry of a storm track intersects a polygon representing the (simplified) coastline of Florida. The polygon is provided as a geometry literal using WKT. (For more information about spatial filtering with CQL in pg_featureserv see this blog post.)

POLYGON ((-81.4067 30.8422, -79.6862 25.3781, -81.1609 24.7731, -83.9591 30.0292, -85.2258 29.6511, -87.5892 29.9914, -87.5514 31.0123, -81.4067 30.8422))

Putting these conditions together in a boolean expression using AND, the request to retrieve the desired tracks from pg_featureserv is:

http://localhost:9000/collections/public.trop_storm/items.html?filter=time_start > 2005-01-01 AND INTERSECTS(geom, POLYGON ((-81.4067 30.8422, -79.6862 25.3781, -81.1609 24.7731, -83.9591 30.0292, -85.2258 29.6511, -87.5892 29.9914, -87.5514 31.0123, -81.4067 30.8422)) )&limit=100

This query produces a result with only 9 tracks, all of which cross Florida:

Try it yourself!

CQL temporal filtering is included in the forthcoming pg_featureserv Version 1.3. But you can try it out now by downloading the latest build. Let us know what use cases you find for CQL temporal filtering! Crunchy Data offers full managed PostGIS in the Cloud, with Container apps to run pg_featureserv. Try it today.

Spatial Filters in pg_featureserv with CQL

Martin.Davis@crunchydata.com (Martin Davis) — Tue, 15 Mar 2022 05:00:00 EDT

pg_featureserv provides access to the powerful spatial database capabilities of PostGIS and PostgreSQL via a lightweight web service. To do this, it implements the OGC API for Features (OAPIF) RESTful protocol. OAPIF is part of the Open Geospatial Consortium (OGC) OGC API suite of standards.

In a previous post, we announced an exciting new capability for pg_featureserv : support for CQL filters. CQL (Common Query Language) is another OGC standard that provides the equivalent of SQL WHERE clauses for web queries.

As you would expect given the OGC focus on promoting ease-of-access to spatial information, CQL supports spatial filtering. This lets us take advantage of PostGIS' ability to query spatial data very efficiently. In this post we'll show some examples of spatial filtering using CQL with pg_featureserv.

The companion vector tile service pg_tileserv also supports CQL, and spatial filtering works there as well.

CQL Spatial Filters

Spatial filtering in CQL involves using spatial predicates to test a condition on the geometry property of features. Spatial predicates include the standard OGC Simple Features predicates for spatial relationships:

INTERSECTS - tests whether two geometries intersect
DISJOINT - tests whether two geometries have no points in common
CONTAINS - tests whether a geometry contains another
WITHIN - tests whether a geometry is within another
EQUALS - tests whether two geometries are topologically equal
CROSSES - tests whether the geometries cross
OVERLAPS - tests whether the geometries overlap
TOUCHES - tests whether the geometries touch

pg_featureserv also implements the distance predicate DWITHIN.

Spatial predicates are typically used to compare the feature geometry property against a geometry value. Geometry values are expressed in Well-Known Text (WKT):

POINT (1 2)
LINESTRING (0 0, 1 1)
POLYGON ((0 0, 0 9, 9 0, 0 0))
POLYGON ((0 0, 0 9, 9 0, 0 0),(1 1, 1 8, 8 1, 1 1))
MULTIPOINT ((0 0), (0 9))
MULTILINESTRING ((0 0, 1 1),(1 1, 2 2))
MULTIPOLYGON (((1 4, 4 1, 1 1, 1 4)), ((1 9, 4 9, 1 6, 1 9)))
GEOMETRYCOLLECTION(POLYGON ((1 4, 4 1, 1 1, 1 4)), LINESTRING (3 3, 5 5), POINT (1 5))
ENVELOPE (1, 2, 3, 4)

By default, the coordinate reference system (CRS) of geometry values is geodetic (longitude and latitude). If needed a different CRS can be specified by using the filter-crs parameter. (PostGIS supports a large number of standard coordinate reference systems.)

Here are some examples of spatial filter conditions:

INTERSECTS(geom, ENVELOPE(-100, 49, -90, 50) )
CONTAINS(geom, POINT(-100 49) )
DWITHIN(geom, POINT(-100 49), 0.1)

Of course, these can be combined with attribute conditions to express real-world queries.

Publishing Geographic Names

For these examples we'll use the U.S. Geographic Names Information System (GNIS) dataset. It contains more than 2 million points for named geographical features. We load this data into a PostGIS spatial table called us.geonames. The point location values are stored in a column called geom of type geography. (PostGIS allows storing spatial data as either geometry or geography. We'll explain later why for this case it is better to use geography).

We create a spatial index on this column to provide fast spatial querying.

CREATE TABLE us.geonames (
    id integer PRIMARY KEY,
    name text,
    lat double precision,
    lon double precision,
    type text,
    state text,
    geom geography(Point),
    ts tsvector
);

CREATE INDEX us_geonames_gix ON us.geonames USING GIST ( geom );

We can now publish the dataset with pg_featureserv, and view query results on the web UI.

The service collections page shows the published tables and views (here we have used the configuration Database.TableIncludes setting to publish just the us schema):

http://localhost:9000/collections.html

The collections\us.geonames page shows the collection metadata:

http://localhost:9000/collections/us.geonames.html

Querying with `INTERSECTS`

For this example we'll query water features on the San Juan Islands in the state of Washington, USA. Because there is no GNIS attribute providing region information, we have to use a spatial filter to specify the area we want to query. We used QGIS to create a polygon enclosing the islands.

We can convert the polygon to WKT and use it in an INTERSECTS spatial predicate (since we are querying points, WITHIN could be used as well - it produces the same result). To retrieve only water features (Lakes and Reservoirs) we add the condition type IN ('LK','RSV'). The query URL is:

http://localhost:9000/collections/public.geonames/items.html?filter=type IN ('LK','RSV') AND INTERSECTS(geom,POLYGON ((-122.722 48.7054, -122.715 48.6347, -122.7641 48.6046, -122.7027 48.3885, -123.213 48.4536, -123.2638 48.6949, -123.0061 48.7666, -122.722 48.7054)))

The result of the query is a dataset containing 33 GNIS points:

Querying with `DWITHIN`

Now we'll show an example of using a distance-based spatial filter, using the DWITHIN predicate. This is the reason we loaded the GNIS data as geography.
DWITHIN tests whether a feature geometry is within a given distance of a geometry value. By using the geography type, we can specify the distance in metres, which are the units of measure of the geodetic EPSG:4326 coordinate system. If we had loaded the dataset using the geometry type, the units would have been degrees, which is awkward to use. Also, geography computes the distance correctly on the surface of the earth (using the great-circle distance).

Let's query for mountains (type = 'MT') within 100 kilometers of Seattle (lat/long of about 47.6,-122.34 - note that WKT requires this as long-lat). The query URL is:

http://localhost:9000/collections/us.geonames_geo/items.html?filter=type = 'MT' AND DWITHIN(geom,Point(-122.34 47.6),100000)&limit=1000

This gives a result showing 695 mountains near Seattle. It's a hilly place!

Try it yourself!

CQL filtering will be included in the forthcoming pg_featureserv Version 1.3. But you can try it out now by downloading the latest build. Let us know what use cases you find for CQL spatial filtering!

More to come

pg_featureserv with CQL attribute and spatial filtering provides a highly effective platform for accessing data over the web. And we're continuing to enhance it with new capabilities to unlock even more of the power of PostGIS and PostgreSQL. Stay tuned for posts on advanced query functionality including temporal filtering, geometry transformations, and aggregation.

CQL Filtering in pg_featureserv

Martin.Davis@crunchydata.com (Martin Davis) — Mon, 07 Mar 2022 04:00:00 EST

The goal of pg_featureserv is to provide easy and efficient access to PostGIS from web clients.

To do this, it uses the emerging OGC API for Features (OAPIF) RESTful protocol. This is a natural fit for systems which need to query and communicate spatial data. The core OAPIF specification provides a basic framework for querying spatial datasets, but it has only limited capability to express filtering subsets of spatial tables.

In particular, it only allows filtering on single attribute values, and it only supports limited spatial filtering via the bbox query parameter (in PostGIS terms, this is equivalent to using the && operator with a box2d).

Of course, PostGIS and PostgresQL provide much more powerful capabilities to filter data using SQL WHERE clauses. It would be very nice to be able to use these via pg_featureserv. Luckily, the OGC is defining a way to allow filtering via the Common Query Language (CQL) which, as the name suggests, is a close match to SQL filtering capabilities. This is being issued under the OGC API suite as CQL2 (currently in draft).

Recently we added pg_featureserv support for most of CQL2. Here we'll describe the powerful new capability it provides.

CQL Overview

CQL is a simple language to describe logical expressions. A CQL expression applies to values provided by feature properties and constants including numbers, booleans and text values. Values can be combined using the arithmetic operators +,-,*, / and % (modulo). Conditions on values are expressed using simple comparisons (<,>,<=,>=,=,<>). Other predicates include:

prop IN (val1, val2, ...)
prop BETWEEN val1 AND val2
prop IS [NOT] NULL
prop LIKE | ILIKE pattern

Conditions can be combined with the boolean operators AND,OR and NOT.

You will notice that this is very similar to SQL (probably not a coincidence!). That makes it straightforward to implement, and easy to use for us database people.

CQL also defines syntax for spatial and temporal filters. We'll discuss those in a future blog post.

Filtering with CQL

A CQL expression can be used in a pg_featureserv request in the filter parameter.
This is converted to SQL and included in the WHERE clause of the underlying database query. This allows the database to use its query planner and any defined indexes to execute the query efficiently.

Here's an example. We'll query the Natural Earth admin boundaries dataset, which we've loaded into PostGIS as a spatial table. (See this post for details on how to do this.) We're going to retrieve information about European countries where the population is 5,000,000 or less. The CQL expression for this is continent = 'Europe' AND pop_est <= 5000000.

Here's the query to get this result:

http://localhost:9000/collections/ne.countries/items.json?properties=name,pop_est&filter=continent%20=%20%27Europe%27%20AND%20pop_est%20%3C=%205000000&limit=100

Note: it's a good idea to to URL-encode spaces and special characters.

This returns a GeoJSON response with 25 features:

By using the extension html instead of json in the request we can visualize the result in the pg_featureserv UI:

More power, fewer functions

One of the cool things about pg_featureserv and its companion pg_tileserv is the ability to serve data provided by PostgreSQL functions. In a previous post we showed how to use a function to find countries where the name matches a search string . Now we can do this more easily and flexibly by using a CQL filter:

http://localhost:9000/collections/ne.countries/items.html?properties=name,pop_est&filter=name%20ILIKE%20%27Mo%25%27

Note that the ILIKE wildcard must be URL-encoded as %25.

And the filter can easily include more complex conditions, which is harder to do with a function. But function serving is still powerful for things like generating spatial data and routing.

More to come...

We'll publish a blog post on the spatial filtering capabilities of CQL soon, along with some other interesting spatial capabilities in pg_featureserv. CQL support will be rolled out in pg_tileserv soon as well. This brings some exciting possibilities for large-scale data visualization!

PostgreSQL provides even more powerful expression capabilities than are available in CQL. There's things like string concatenation and functions, the CASE construct for "computed if", and others. What kinds of things would you like to see pg_featureserv support?

Try it!

CQL filtering will be included in the forthcoming pg_featureserv Version 1.3.
But you can try it out now by simply downloading the latest build. Let us know what use cases you find for CQL filtering!

Using PostGIS and pg_featureserv with QGIS

Martin.Davis@crunchydata.com (Martin Davis) — Wed, 21 Jul 2021 05:00:00 EDT

My colleague Kat Batuigas recently wrote about using the powerful open-source QGIS desktop GIS to import data into PostGIS from an ArcGIS Feature Service. This is a great first step toward moving your geospatial stack onto the performant, open source platform provided by PostGIS. And there's no need to stop there! Crunchy Data has developed a suite of spatial web services that work natively with PostGIS to expose your data to the web, using industry-standard protocols. These include:

pg_tileserv - allows mapping spatial data using the MVT vector tile format
pg_featureserv - publishes spatial data using the OGC API for Features protocol

Recent versions of QGIS support using the OGC API for Features (previously known as WFS3) as a vector data source. So it should be able to source data from pg_featureserv.

Let's see how it works.

Load Data into PostGIS

To keep things simple we are using a Crunchy Bridge cloud-hosted Postgres/PostGIS instance. For demo purposes we'll load a dataset of British Columbia wildfire perimeter polygons (available for download here). The data is provided as a shapefile, so we can use the PostGIS shp2pgsql utility to load it into a table. (If the data was in another format then we could load it using ogr2ogr, or use QGIS itself as Kat described).

We use the -c option to have the loader create a table appropriate for the dataset, and the -I option to create a spatial index on it (always a good idea). The data is in the BC-Albers coordinate system, so we specify the SRID using as -s 3005. Here we are doing the load in two steps using an intermediate SQL file, or it can be done in a single command by piping the shp2pgsql output to psql.

shp2pgsql -c -D -s 3005 -i -I prot_current_fire_polys.shp bc.wildfire_poly > bc_wf.sql
psql -h p.asdfghjklqwertyuiop12345.db.postgresbridge.com -U postgres < bc_wf.sql

Using psql we can connect to the database and verify that the table has been created and loaded:

postgres=# \d bc.wildfire_poly
                                            Table "bc.wildfire_poly"
   Column   |            Type             | Collation | Nullable |                    Default
------------+-----------------------------+-----------+----------+-----------------------------------------------
 gid        | integer                     |           | not null | nextval('bc.wildfire_poly_gid_seq'::regclass)
 objectid   | double precision            |           |          |
 fire_year  | integer                     |           |          |
 fire_numbe | character varying(6)        |           |          |
 version_nu | double precision            |           |          |
 fire_size_ | numeric                     |           |          |
 source     | character varying(50)       |           |          |
 track_date | date                        |           |          |
 load_date  | date                        |           |          |
 feature_co | character varying(10)       |           |          |
 fire_stat  | character varying(30)       |           |          |
 fire_nt_id | integer                     |           |          |
 fire_nt_nm | character varying(50)       |           |          |
 fire_nt_lk | character varying(254)      |           |          |
 geom       | geometry(MultiPolygon,3005) |           |          |
Indexes:
    "wildfire_poly_pkey" PRIMARY KEY, btree (gid)
    "wildfire_poly_geom_idx" gist (geom)

 postgres=# select count(*) from bc.wildfire_poly;
 count
-------
   133

Serve PostGIS Data with pg_featureserv

The Crunchy spatial services are lightweight native applications (written in Go), so it's easy to install them on a local or hosted platform. They can be downloaded as applications, a Docker container, or built from source. See the installation guide for details.

We're running pg_featureserv in a local environment. To connect it to the Bridge Postgres instance, we specify the database connection information in the config/pg_featureserv.toml file. The connection string can also be specified in an environment variable. See the documentation for more information.

[Database]
# Database connection
# postgresql://username:password@host/dbname
DbConnection = "postgres://postgres:password@p.asdfghjklqwertyuiop12345.db.postgresbridge.com:5432/postgres"

For ease-of-use pg_featureserv provides a browser-based Admin UI. Using this we can see the data table published as a collection:

http://localhost:9000/collections.html

We can display the the collection metadata:

http://localhost:9000/collections/bc.wildfire_poly.html

The Admin UI also lets us see the data on a map:

http://localhost:9000/collections/bc.wildfire_poly/items.html?limit=200

The main use of pg_featureserv is to serve feature data via the OGC API for Features (OAPIF), which is a RESTful HTTP protocol returning GeoJSON. We can verify that a OAPIF data query works by issuing the following request URL. The response is a GeoJSON document (shown here using the handy JSON display in the Firefox browser):

http://localhost:9000/collections/bc.wildfire_poly/items.json

Display pg_featureserv Collection as a QGIS Layer

In QGIS, we can create a layer that displays the data coming from the pg_featureserv instance.

To do this, under the Layer menu choose Add Layer > Add WFS Layer**...**. This displays the Data Source Manager window at the WFS/OGC API-Features tab. Click New to define the connection to the pg_featureserv service. The Connection Details dialog lets us enter the following information:

We'll use pg_fs as the name of the connection
The connection URL is the service home endpoint http://localhost:9000/
The WFS Version is OGC API - Features
We'll specify the Max. number of features as 200, since that will allow loading the entire dataset without paging

When we click Connect we see the collections published by the service (in this demo there is only one):

Now we can select the bc.wildfire_poly collection, and click Add to add it as a layer to the QGIS map layout. The screenshot also shows the result of using the Identify Features tool on the map, showing that all attribute data is loaded as well.

Of course, QGIS is able to connect to PostGIS directly, and provides full query and update capability when doing that. But it can be simpler, more flexible and more secure to expose PostGIS data via a web service. That way, it can be easily accessed by many different tools which might not be able to or allowed to connect to PostGIS directly.

We're also exploring ways to be able to run pg_tileserv and pg_featureserv directly in Crunchy Bridge, to provide a turn-key solution for exposing spatial data on the web. If this sounds interesting to you, get in touch!