Jason O'Donnell | CrunchyData Blog https://www.crunchydata.com/blog/author/jason-odonnell https://www.crunchydata.com/build/_assets/default.png-W4XGD4DB.webp Jason O'Donnell | CrunchyData Blog https://www.crunchydata.com/blog/author/jason-odonnell 256 256 PostgreSQL experts from Crunchy Data share advice, performance tips, and guides on successfully running PostgreSQL and Kubernetes solutions en-us Tue, 27 Mar 2018 05:00:00 EDT 2018-03-27T09:00:00.000Z en-us hourly 1 <![CDATA[ An Easy Recipe for Creating a PostgreSQL Cluster with Docker Swarm ]]> https://www.crunchydata.com/blog/an-easy-recipe-for-creating-a-postgresql-cluster-with-docker-swarm One of the biggest benefits of running PostgreSQL is running your cluster in primary-replica setup for the purposes of high-availability or load balancing your read-only queries. It is not necessarily simple to deploy a primary-replica setup out of the box, but by using modern containerization technology, we can greatly simplify the process.

In this article, I will demonstrate how to easily setup and deploy a PostgreSQL primary-replica cluster using Docker and Docker Swarm.

Background

Streaming replication, or pushing changes from a primary PostgreSQL instance to its replicas in "real-time," was introduced into PostgreSQL in version 9.0 and has continued to receive numerous enhancements in all subsequent releases, including:

The PostgreSQL documentation also provides an overview and comparison of the different replication methods.

The primary-replica methodology of deploying a PostgreSQL replica is an essential tool in creating high-availability environments for your database cluster: a proper deployment should ensure that your data is stored on different disks in different data centers.

Replication is not a "setup and forget" operation: in a production system, you will want to ensure you have appropriate monitoring in place, for instance, to know that all of your replicas are online or to know how much data a replica may need to be in sync with the primary.

While the primary-replica configuration with PostgreSQL is great, it can take a little bit of work to setup. Fortunately, there is a way to deploy this setup more easily with Docker.

Environment SETUP

In order to deploy this recipe, you will need at least Docker 1.12, which was released in July 2016, installed in order to successfully deploy this recipe.

To begin, provision a Docker cluster. For this example a development cluster, you can load Docker on each of the machines you will use in your Swarm. This recipe is using the following architecture:

PostgreSQL Docker Swarm Architecture

Each host should have Docker enabled and started.

Swarm Setup

Since version 1.12, Docker has included the Swarm clustering technology directly within the Docker Engine.

Creating a Docker Swarm cluster is out of scope for this tutorial, however, documentation on setting up a Swarm cluster can be found here.

Container Placement

Two or more hosts are required for a high-availability PostgreSQL cluster configuration. The primary and replica(s) should run on different worker nodes to maximize availability.

In order to deploy the Crunchy PostgreSQL containers to multiple hosts, you will need to use node labels. Labeling hosts has a few advantages when it comes to PostgreSQL containers:

  • Spreading services out on many worker nodes improves availability
  • Hosts can be optimized (such as using high-performance disks) for reads (replicas) or writes (primary)

Remember: as of PostgreSQL 10, a primary database can support both reads and writes, but a replica only allows read queries.

To allow for container placement on specific worker nodes, add a metadata label to the Swarm nodes:

node1_id=$(docker node list | grep worker1 | awk '{print $1}')
docker node update --label-add type=primary ${node1_id?}

In the above example, a label called primary has been added to the worker1. Using this label we can apply constraints to the Docker Swarm deployment of the PostgreSQL stack.

Note*: We did not apply a constraint for replicas as we can just use the inverse constraint:

node.labels.type != primary

PostgreSQL Stack Definition

With the Swarm deployed and the worker node properly labeled, we can deploy the PostgreSQL stack.

The PostgreSQL stack is comprised of a primary and replica services. The following is the service definition:

docker-compose.yml

---
version: '3.3'

services:
  primary:
    hostname: 'primary'
    image: crunchydata/crunchy-postgres:centos7-10.3-1.8.2
    environment:
      - PGHOST=/tmp
      - MAX_CONNECTIONS=10
      - MAX_WAL_SENDERS=5
      - PG_MODE=primary
      - PG_PRIMARY_USER=primaryuser
      - PG_PRIMARY_PASSWORD=password
      - PG_DATABASE=testdb
      - PG_USER=testuser
      - PG_PASSWORD=password
      - PG_ROOT_PASSWORD=password
      - PG_PRIMARY_PORT=5432
    volumes:
      - pg-primary-vol:/pgdata
    ports:
      - '5432'
    networks:
      - crunchynet
    deploy:
      placement:
        constraints:
          - node.labels.type == primary
          - node.role == worker
  replica:
    image: crunchydata/crunchy-postgres:centos7-10.3-1.8.2
    environment:
      - PGHOST=/tmp
      - MAX_CONNECTIONS=10
      - MAX_WAL_SENDERS=5
      - PG_MODE=replica
      - PG_PRIMARY_HOST=primary
      - PG_PRIMARY_PORT=5432
      - PG_PRIMARY_USER=primaryuser
      - PG_PRIMARY_PASSWORD=password
      - PG_DATABASE=testdb
      - PG_USER=testuser
      - PG_PASSWORD=password
      - PG_ROOT_PASSWORD=password
    volumes:
      - pg-replica-vol:/pgdata
    ports:
      - '5432'
    networks:
      - crunchynet
    deploy:
      placement:
        constraints:
          - node.labels.type != primary
          - node.role == worker
networks:
  crunchynet:

volumes:
  pg-primary-vol:
  pg-replica-vol:

Notice that the primary service defines a hostname but the replica service does not. Replicas require a hostname to start replication. By providing a static hostname to the primary the replicas can connect without having to discover the primary container.

The replicas, however, do not have a hostname. This allows the replica service to be scaled beyond a single replica (as shown later).

The only real difference between the primary and replica services is the PG_MODE environment variable. This configures the containers to either be a primary or replica.

Deploying the Stack

After saving this file to docker-compose.yml, we can deploy the stack with Docker:

docker stack deploy --compose-file=./docker-compose.yml pg-stack

This stack deployment will create a PostgreSQL cluster similar to this diagram:

PostgreSQL Docker Swarm Primary Replica Cluster

Testing the Cluster

To check if services are running, run the following commands:

docker service ls
docker service ps pg-stack_primary
docker service ps pg-stack_replica

To increase the number of replicas, run the following:

docker service scale pg-stack_replica=2
docker service ps pg-stack_replica

To verify the replicas are streaming, query the PostgreSQL primary on the worker1 host using the following command:

docker exec -it $(docker ps -q) psql -U postgres -x -c 'table pg_stat_replication' postgres

You should see a row for each replica along with its replication status.

Example Code

The example described above is provided in the Crunchy Containers Suite GitHub in the following location:

https://github.com/CrunchyData/crunchy-containers/tree/master/examples/docker/swarm-service

Conclusion

Docker and Docker Swarm provide tools to take container deployments to the next level. We hope that this demonstration proves how easy it is to get up and running with Crunchy PostgreSQL containers. ]]> https://blog.crunchydata.com/blog/an-easy-recipe-for-creating-a-postgresql-cluster-with-docker-swarm Tue, 27 Mar 2018 05:00:00 EDT 2018-03-27T09:00:00.000Z 2018-03-27T09:00:00.000Z <![CDATA[ pgAudit: Auditing Database Operations Part 2 ]]> https://www.crunchydata.com/blog/pgaudit-auditing-database-operations-part-2 In the last blog post, pgAudit was configured to audit entire classes of statements (session auditing). Session auditing works great, but it can generate a lot of logs and not every administrator needs all that information. In this blog post pgAudit will be configured to use an auditing role to watch only specific objects.

Getting Started

This guide assumes pgAudit has already been installed on the target DB server. For more instructions on installing pgAudit, see the official documentation here.

pgAudit auditor role only supports the following commands against objects:

  • SELECT
  • INSERT
  • UPDATE
  • DELETE

Setup

First, create a role to designate as the auditing role:

CREATE ROLE auditor NOLOGIN;

Next, configure pgAudit to use that role for auditing:

ALTER SYSTEM SET pgaudit.role TO 'auditor';
SELECT pg_reload_conf();

That's it! pgAudit is now configured to use an auditing role.

Auditor Example

With pgAudit configured to use an auditing role, auditor can be assigned objects to audit.

First, lets create a test table and insert some data:

CREATE TABLE public.pgauditExample(id SERIAL, name TEXT, secret TEXT, age INT);
INSERT INTO public.pgauditExample(name, secret, age) VALUES ('crunchy', 'my-secret', 30);

Next, grant the operations to the auditing role that should be monitored:

GRANT SELECT (name, secret), UPDATE (secret) ON public.pgauditExample TO auditor;

Notice how SELECT is granted to name and secret, however, auditor is also granted UPDATE on secret.

Next, trigger some audit logging on the objects that were just configured:

SELECT name FROM public.pgauditExample;
SELECT secret FROM public.pgauditExample;
UPDATE public.pgauditExample SET secret = 'new-secret' WHERE 'name' = 'crunchy';
SELECT age FROM public.pgauditExample;

Finally, check pg_log for the audit entries:

$ grep AUDIT postgresql-Fri.log | grep OBJECT
2016-10-12 13:54:54.836 UTC postgres postgres LOG: AUDIT: OBJECT,5,1,READ,SELECT,TABLE,public.pgauditexample,SELECT name FROM pgauditExample;,<none>
2016-10-12 13:54:54.837 UTC postgres postgres LOG: AUDIT: OBJECT,6,1,READ,SELECT,TABLE,public.pgauditexample,SELECT secret FROM pgauditExample;,<none>
2016-10-12 13:54:54.838 UTC postgres postgres LOG: AUDIT: OBJECT,7,1,WRITE,UPDATE,TABLE,public.pgauditexample,UPDATE pgauditExample SET secret = 'new-secret' WHERE 'name' = 'crunchy';,<none>

Notice how there is no audit log for the SELECT on age. The auditor role has not been granted privileges on that column, thus it does not watch the object.

Multiple Auditor Roles

In the last example pgAudit was configured to use the auditor role to watch specific objects. Although pgAudit can only be assigned one master auditor role, multiple roles can be configured to audit objects by granting them to the master role.

First, create new roles to also be used for auditing:

CREATE ROLE read_auditor NOLOGIN;
CREATE ROLE write_auditor NOLOGIN;
CREATE ROLE delete_auditor NOLOGIN;

Next, grant these roles to the auditor role:

GRANT read_auditor TO auditor;
GRANT write_auditor TO auditor;
GRANT delete_auditor TO auditor;

With the auditor role configured to use multiple roles, create a test table:

CREATE TABLE public.pgauditAuditorExample(id SERIAL, name TEXT, secret TEXT, age INT);
INSERT INTO public.pgauditAuditorExample(name, secret, age) VALUES ('crunchy', 'my-secret', 30);

Next, grant the new roles access to various operations on the test table:

GRANT SELECT (name, secret, age) ON public.pgauditAuditorExample TO read_auditor;
GRANT INSERT(secret), UPDATE (secret) ON public.pgauditAuditorExample TO write_auditor;
GRANT DELETE ON public.pgauditAuditorExample TO delete_auditor;

With the new roles assigned to their objects to audit, trigger the audit logging:

SELECT secret FROM public.pgauditAuditorExample;
INSERT INTO public.pgauditAuditorExample(name, secret, age) VALUES ('postgres', 'new-secret', 100);
UPDATE public.pgauditAuditorExample SET secret = 'new-secret' WHERE name = 'crunchy';
DELETE FROM public.pgauditAuditorExample WHERE age = 100;

Finally, check pg_log for the audit entries:

$ grep AUDIT postgresql-Wed.log | grep OBJECT
2016-10-12 14:22:01.416 UTC postgres postgres LOG: AUDIT: OBJECT,4,1,READ,SELECT,TABLE,public.pgauditauditorexample,SELECT secret FROM public.pgauditAuditorExample;,<none>
2016-10-12 14:22:05.447 UTC postgres postgres LOG: AUDIT: OBJECT,5,1,WRITE,INSERT,TABLE,public.pgauditauditorexample,"INSERT INTO public.pgauditAuditorExample(name, secret, age) VALUES ('postgres', 'new-secret', 100);",<none>
2016-10-12 14:22:12.299 UTC postgres postgres LOG: AUDIT: OBJECT,6,1,WRITE,UPDATE,TABLE,public.pgauditauditorexample,UPDATE public.pgauditAuditorExample SET secret = 'new-secret' WHERE name = 'crunchy';,<none>
2016-10-12 14:22:17.198 UTC postgres postgres LOG: AUDIT: OBJECT,7,1,WRITE,DELETE,TABLE,public.pgauditauditorexample,DELETE FROM public.pgauditAuditorExample WHERE age = 100;,<none>

Wrap up

These examples show how pgAudit can be tuned to watch specific objects using auditor role(s). This gives administrators the ability to finely tune what is audited in their database by either using session auditing or object auditing.

pgAudit is shipped with the Crunchy Certified PostgreSQL distribution and can also be obtained from the project repository. ]]> https://blog.crunchydata.com/blog/pgaudit-auditing-database-operations-part-2 Fri, 14 Oct 2016 05:00:00 EDT 2016-10-14T09:00:00.000Z 2016-10-14T09:00:00.000Z <![CDATA[ pgAudit: Auditing Database Operations Part 1 ]]> https://www.crunchydata.com/blog/pgaudit-auditing-database-operations-part-1 The PostgreSQL Audit extension (pgaudit) provides detailed session and/or object audit logging via the standard PostgreSQL logging facility.

Basic statement logging can be provided by the standard logging facility in PostgreSQL. Out of the box logging provided by PostgreSQL is acceptable for monitoring and other usages but does not provide the level of detail generally required for an audit.

pgAudit enhances PostgreSQL's logging abilities by allowing administrators to audit specific classes of operations or choosing specific objects to monitor.

Getting Started

This guide assumes pgAudit has already been installed on the target DB server. For more instructions on installing pgAudit, see the official documentation here.

Session Auditing

Session auditing allows administrators to choose classes of statements to log:

  • READ (SELECT and COPY when the source is a relation or a query)
  • WRITE (INSERT, UPDATE, DELETE, TRUNCATE, and COPY when the destination is a relation)
  • FUNCTION (Functions and DO blocks)
  • ROLE (GRANT, REVOKE, CREATE/ALTER/DROP ROLE)
  • DDL (All DDL not included in ROLE)
  • MISC (DISCARD, FETCH, CHECKPOINT, VACUUM)

READ Example

First, create a test table and insert some data:

CREATE TABLE pgauditExample(id SERIAL, name TEXT);
INSERT INTO pgauditExample(name) VALUES ('crunchy');

Next, configure pgAudit to audit the read class by altering the pgaudit.log parameter:

ALTER SYSTEM SET pgaudit.log TO 'read';
SELECT pg_reload_conf();

With pgAudit set to audit the read class, SELECT from our test table:

SELECT name FROM pgauditExample;

Finally, check pg_log for an audit entry:

$ grep AUDIT postgresql-Fri.log | grep READ
2016-09-30 00:16:24.688 UTC postgres postgres LOG: AUDIT: SESSION,1,1,READ,SELECT,,,SELECT name FROM pgauditExample;,<none>

WRITE Example

In the last example we configured pgAudit to audit the READ class of statements. Building on the previous example, add WRITE:

ALTER SYSTEM SET pgaudit.log TO 'read, write';
SELECT pg_reload_conf();

With pgAudit set to audit the read and write classes, INSERT, UPDATE and DELETE from our test table:

INSERT INTO pgauditExample(name) VALUES ('postgres');
UPDATE pgauditExample SET name = 'awesome' WHERE name = 'postgres';
DELETE FROM pgauditExample WHERE name = 'awesome';

Finally, check pg_log for the audit entries:

$ grep AUDIT postgresql-Fri.log | grep WRITE

2016-09-30 00:25:05.785 UTC postgres postgres LOG: AUDIT: SESSION,2,1,WRITE,INSERT,,,INSERT INTO pgauditExample(name) VALUES ('postgres');,<none>
2016-09-30 00:25:05.787 UTC postgres postgres LOG: AUDIT: SESSION,3,1,WRITE,UPDATE,,,UPDATE pgauditExample SET name = 'awesome' WHERE name = 'postgres';,<none>
2016-09-30 00:25:06.476 UTC postgres postgres LOG: AUDIT: SESSION,4,1,WRITE,DELETE,,,DELETE FROM pgauditExample WHERE name = 'awesome';,<none>

Function Example

So far we've configured pgAudit to audit READ and WRITE. Next, add FUNCTION to the watch list.

ALTER SYSTEM SET pgaudit.log TO 'read, write, function';
SELECT pg_reload_conf();

With pgAudit set to audit the function class, execute an anonymous function:

DO $$
BEGIN
 RAISE NOTICE 'pgAudit rocks!';
END
$$;

Finally, check pg_log for the audit entries:

$ tail -5 postgresql-Fri.log

2016-09-30 14:51:19.036 UTC postgres postgres LOG: AUDIT: SESSION,1,1,FUNCTION,DO,,,"DO $$
 BEGIN
 RAISE NOTICE 'pgAudit rocks!';
 END
 $$;",<none>

Role Example

In the last example we configured pgAudit to audit the READ, WRITE and FUNCTION classes of statements. Building on the previous example, add ROLE. Instead of adding role to the pgaudit.log parameter, notice the configuration is different this time:

ALTER SYSTEM SET pgaudit.log TO 'all, -misc, -ddl';
SELECT pg_reload_conf();

This time the configuration specifies all classes except misc and ddl (read, write, function, role).

With pgAudit set to audit the role class, create, alter and drop some roles:

CREATE ROLE bob;
CREATE ROLE alice;
ALTER ROLE bob LOGIN;
ALTER ROLE alice LOGIN CONNECTION LIMIT 1;
GRANT ALL ON TABLE pgauditExample TO bob;
GRANT SELECT ON TABLE pgauditExample TO alice;
REVOKE ALL ON TABLE pgauditExample FROM bob;
REVOKE ALL ON TABLE pgauditExample FROM alice;
DROP ROLE bob;
DROP ROLE alice;

Finally, check pg_log for the audit entries:

2016-09-30 15:03:11.522 UTC postgres postgres LOG: AUDIT: SESSION,2,1,ROLE,CREATE ROLE,,,CREATE ROLE bob;,<none>
2016-09-30 15:03:11.523 UTC postgres postgres LOG: AUDIT: SESSION,3,1,ROLE,CREATE ROLE,,,CREATE ROLE alice;,<none>
2016-09-30 15:03:11.524 UTC postgres postgres LOG: AUDIT: SESSION,4,1,ROLE,ALTER ROLE,,,ALTER ROLE bob LOGIN;,<none>
2016-09-30 15:03:11.526 UTC postgres postgres LOG: AUDIT: SESSION,5,1,ROLE,ALTER ROLE,,,ALTER ROLE alice LOGIN CONNECTION LIMIT 1;,<none>
2016-09-30 15:03:11.528 UTC postgres postgres LOG: AUDIT: SESSION,6,1,ROLE,GRANT,,,GRANT ALL ON TABLE pgauditExample TO bob;,<none>
2016-09-30 15:03:11.529 UTC postgres postgres LOG: AUDIT: SESSION,7,1,ROLE,GRANT,,,GRANT SELECT ON TABLE pgauditExample TO alice;,<none>
2016-09-30 15:03:11.531 UTC postgres postgres LOG: AUDIT: SESSION,8,1,ROLE,REVOKE,,,REVOKE ALL ON TABLE pgauditExample FROM bob;,<none>
2016-09-30 15:03:11.532 UTC postgres postgres LOG: AUDIT: SESSION,9,1,ROLE,REVOKE,,,REVOKE ALL ON TABLE pgauditExample FROM alice;,<none>
2016-09-30 15:03:11.534 UTC postgres postgres LOG: AUDIT: SESSION,10,1,ROLE,DROP ROLE,,,DROP ROLE bob;,<none>
2016-09-30 15:03:11.876 UTC postgres postgres LOG: AUDIT: SESSION,11,1,ROLE,DROP ROLE,,,DROP ROLE alice;,<none>

DDL Example

In the last example we configured pgAudit to audit all classes except misc and ddl. Building on the previous example, add ddl:

ALTER SYSTEM SET pgaudit.log TO 'all, -misc';
SELECT pg_reload_conf();

With pgAudit set to audit the ddl class, have some fun with tables:

CREATE TABLE pgauditDDLExample(id SERIAL);
ALTER TABLE pgauditDDLExample ADD COLUMN name text;
CREATE POLICY namePolicy ON pgauditDDLExample FOR ALL USING (current_user = 'postgres');
DROP POLICY namePolicy on pgauditDDLExample;
DROP TABLE pgauditDDLExample;

Finally, check pg_log for the audit entries:

2016-09-30 15:07:02.776 UTC postgres postgres LOG: AUDIT: SESSION,2,1,DDL,CREATE TABLE,,,CREATE TABLE pgauditDDLExample(id SERIAL);,<none>
2016-09-30 15:08:18.054 UTC postgres postgres LOG: AUDIT: SESSION,3,1,DDL,ALTER TABLE,,,ALTER TABLE pgauditDDLExample ADD COLUMN name text;,<none>
2016-09-30 15:09:18.095 UTC postgres postgres LOG: AUDIT: SESSION,4,1,DDL,CREATE POLICY,,,CREATE POLICY namePolicy ON pgauditDDLExample FOR ALL USING (current_user = 'postgres');,<none>
2016-09-30 15:09:37.562 UTC postgres postgres LOG: AUDIT: SESSION,5,1,DDL,DROP POLICY,,,DROP POLICY namePolicy on pgauditDDLExample;,<none>
2016-09-30 15:09:45.378 UTC postgres postgres LOG: AUDIT: SESSION,6,1,DDL,DROP TABLE,,,DROP TABLE pgauditDDLExample;,<none>

MISC Example

The last class is MISC, configure pgAudit to audit all classes:

ALTER SYSTEM SET pgaudit.log TO 'all';
SELECT pg_reload_conf();

With pgAudit set to audit all classes, here's a demonstration of the misc class:

CHECKPOINT;
VACUUM pgauditExample;

Finally, check pg_log for the audit entries:

$ grep AUDIT postgresql-Fri.log | grep MISC
2016-09-30 15:17:45.214 UTC postgres postgres LOG: AUDIT: SESSION,3,1,MISC,CHECKPOINT,,,CHECKPOINT;,<none>
2016-09-30 15:17:47.474 UTC postgres postgres LOG: AUDIT: SESSION,4,1,MISC,VACUUM,,,VACUUM pgauditExample;,<none>

Wrap Up

From the examples above, pgAudit was configured to session audit entire classes of SQL using pgAudit. Session auditing is a great feature, however, a lot of logs can be generated using session auditing. In part 2 of this series on pgAudit, an auditor role will be configured to watch specific objects instead of classes. Stay tuned! ]]> https://blog.crunchydata.com/blog/pgaudit-auditing-database-operations-part-1 Mon, 03 Oct 2016 05:00:00 EDT 2016-10-03T09:00:00.000Z 2016-10-03T09:00:00.000Z