Paul Laurence | CrunchyData Blog

Using Kubernetes? Chances Are You Need a Database

Paul.Laurence@crunchydata.com (Paul Laurence) — Mon, 03 May 2021 05:00:00 EDT

Whether you are starting a new development project, launching an application modernization effort, or engaging in digital transformation, chances are you are evaluating Kubernetes. If you selected Kubernetes, chances are you will ultimately need a database.

Kubernetes provides many benefits for running applications including efficiency, automation, or infrastructure abstraction. These features allow you to deploy highly availability databases and scale, making it easier to manage hardware for databases as they grow.

More users are adopting databases on Kubernetes. The Cloud Native Computing Foundation (CNCF) provides great data on Kubernetes adoption. The Cloud Native Survey 2020 that 55% of respondents are using stateful applications in containers in production. Crunchy Data has many customers who successfully run Postgres on Kubernetes.

So how do you start with databases on Kubernetes?

Selecting Your Database

As users select new databases for Kubernetes, they generally start with open source. Typically users adopt a combination of SQL and NoSQL databases for their data toolbox.

At Crunchy Data, we are fans of Postgres and believe it is a great fit for SQL workloads on Kubernetes. We are happy to see the CNCF agree with this assessment. The CNCF End User Technology Radar for Database Storage places Postgres in the "Adopt" category. This status indicates Postgres is among "widely adopted and recommended by the respondents."

With the right tools (like PGO, the Postgres Operator from Crunchy Data) and expertise, Postgres is Kubernetes friendly.

You Probably Want an Operator

The term “operator” was a term coined by CoreOS. Technically, a Kubernetes Operator is an application-specific controller for Kubernetes. Simply put, an operator is a Kubernetes extension to automate actions of a human "operator".

The operator pattern works especially well for databases, which can be admin intensive. Tasks like backups or restore typically need human intervention. A database operator can automate these administrative tasks, reducing your admin burden. Application specific operators go a step further to provide application specific automation which can accelerate scaling and standardization.

For instance, PGO automates Postgres specific capability through automating Postgres tools and expertise. PGO provides GitOps friendly workflows for provisioning standardized Postgres clusters and automates high availability, disaster recovery, and monitoring.

You can use an operator in a range of use cases. Users can deploy Kubernetes native applications alongside Postgres using PGO. Operators provide centralized administrations and automation at scale for users building a database-as-a-service.

Storage Options

Databases are different than stateless apps in that they need persistent storage.

Databases running on Kubernetes can make use of a range of storage types. Options include HostPath, Network File System (NFS), and Dynamic storage. PGO is storage agnostic: it works with any supported Kubernetes storage system.

You must provision persistent volumes for both Hostpath and NFS, though automated storage provisioners do exist. Dynamic storage classes allow users to request persistent volume claims and have the persistent volume created for you. There are many providers of dynamic storage classes to choose from. You will need to configure what works for your environment and size the Physical Volumes, Persistent Volumes (PVs).

Public cloud environments with multiple availability zones raise additional issues. If you are planning to use a public cloud with multiple zones, you will want to become familiar with the topology aware storage class configurations.

Getting Started

The Kubernetes ecosystem has made it easy to deploy self-managed databases. Tools such as Helm and Kustomize provide automated installation and initial configuration. Marketplaces like OperatorHub.io, the OpenShift Lifecycle Manager and Google Kubernetes Marketplace also provide easy getting started workflows.

If you have a default storage class setup, the below commands enables you to deploy PGO:

kubectl create namespace pgo>
kubectl apply -f https://raw.githubusercontent.com/CrunchyData/postgres-operator/v4.6.2/installers/kubectl/postgres-operator.yml

Within a few minutes, you should be up and running and ready to deploy a database to your Kubernetes environment.

Ready for a Deeper Dive?

The CNCF research is important validation of what we are seeing in the market. Users are adopting databases on Kubernetes. Operators, Helm, Kustomize and improvements in persistent storage have made this a much easier option. That said, there are of course areas where you may want to do a deeper dive include:

Deploying Postgres on Kubernetes is one of the many ways to benefit from the power and efficiency of Postgres. At Crunchy Data, we are seeing customers have success with this model and further grow their adoption of trusted open source Postgres. Now is a great time to get started running Postgres on Kubernetes.

Is Postgres Secure?

Paul.Laurence@crunchydata.com (Paul Laurence) — Mon, 29 Mar 2021 05:00:00 EDT

With the rise of Postgres, new organizations are evaluating how to benefit from its power and flexibility. As that evaluation progresses, Postgres advocates must address the question, "Is Postgres secure?"

There are a variety of ways to answer this question, but the short answer is a confident "Yes!"

At Crunchy Data, we often collaborate with organizational stakeholders to address this question. Many organizations have used the same collection of databases for years (maybe decades). The idea of adopting a new database often requires first brushing off the security criteria, and then evaluating whether Postgres meets these requirements.

This post provides a framework for thinking through how you can confront "security questions" as you embark on your Postgres journey.

Does Postgres Have the Security Functionality We Require?

Chances are your organization is using an alternative database today. As a result, your organization has some expectation of the security functionality that a database must provide. An initial question is often "Does Postgres have the necessary security functionality that we rely on from $OTHER_DATABASE?"

Short answer is YES.

Common Criteria for Information Technology Security is an international standard (ISO/IEC 15408) for computer security certification. Common Criteria (CC) provides database specific security guidance through the Protection Profile for Database Management Systems (DBMS PP). The DBMS PP presents the "security functional requirements" and "security assurance requirements" that representatives from industry, government, and CC Test Laboratories have determined to be applicable to databases. According to DBMS PP, in order for a database to achieve Common Criteria certification, it is necessary for the database to provide:

Discretionary Access Control (DAC) limits access to objects based on the identity of the subjects or groups to which the subjects and objects belong, and which allows authorized users to specify how the objects that they control are protected.
Audit Capture for creation of information on all auditable events.
Authorized administration role to allow authorized administrators to configure the policies for discretionary access control, identification and authentication, and auditing. The TOE must enforce the authorized administration role.
Limitation of the number of concurrent sessions and restrictions on establishing sessions.

Our team went through the initial Common Criteria certification of PostgreSQL 9.5 in 2016 and we've just completed recertification on PostgreSQL 12. We validated that PostgreSQL and several open source extensions, including PostGIS, pgaudit and the JDBC driver, support the CC certification capabilities.This is a powerful statement: while the Common Criteria certification does have some specifics to how Crunchy Data distributes PostgreSQL, all of the components are just the open source projects!

CC certification provides lab based validation that open source Postgres, with these extensions, provides the full set of security functionality required by the DBMS PP and that users can expect from a database.

How Do I Securely Deploy and Administer my Postgres Database?

The existence of security functionality alone is of course just the beginning. Using these capabilities consistent with best practices are necessary for secure use of Postgres.

In working with the Center for Internet Security (CIS) and the United States Defense Information Security Agency (DISA), we've developed security guidance for the deployment, configuration and administration of Postgres. The most recent versions of these standards bring them current with PostgreSQL 13, the most recent major release. The Security Technical Implementation Guide (STIG) in particular provides guidance on the configuration of PostgreSQL to address requirements associated with:

Auditing
Logging
Data Encryption at Rest
Data Encryption in Transit
Access Controls
Administration
Authentication
Protecting against SQL Injection

There are many things that you must consider when you want to securely manage a PostgreSQL database, but these guides provide ways you can tick the standard "checkboxes" when to securely deploy your data.

How Does Postgres Address SQL Injection?

Despite the widespread awareness of this attack vector, the risk of SQL injection attacks remains a common question. Through SQL injection attacks, bad actors “inject” data into your SQL query using destructive phrases or unescaped parameters. Hackers use this technique to gain access to business data and personal information, as well as modify or delete the content within your database.

Most modern ORMs keep you well protected and are actively tested against these vulnerabilities. A basic security tool will also tell you if you're susceptible.

For Postgres users, writing customer SQL as part of their application, specific guidance on the topic is also applicable.

The best practices from the CIS Benchmark and the DISA STIG will also provide the necessary protection.

How Can I Address My Organization's Encryption Requirements?

Data encryption is a common organizational requirement and security best practice. The general requirement typically includes both "encryption of data at rest" and "encryption of data in transit". Each has its own range of solutions, including full disk encryption, volume encryption, and folder encryption.

Postgres enables data encryption in conjunction with the broader enterprise IT environment through a number of features, including the "pgcrypto" extension, and support for FIPS validated encryption modules.

While this post will not attempt to do the subject justice, a few initial observations based on how we see our customers address these requirements:

For "Encryption At Rest", our customers often address these requirements through the storage layer or by your infrastructure provider (such is the case on Crunchy Bridge).
For "Encryption In Transit", Postgres enables users to connect over TLS and setting explicitly in your connection string.

Again, the CIS Benchmark and the DISA STIG also provide specific configuration guidance.

The Hot Topic of the Day: Supply Chain

Recent news has highlighted the risks and importance of understanding the software supply chain. Postgres is of course an open source software project maintained by an active global development community with a high degree of transparency. This transparency provides an important mitigation against potential security risks. That said, freely available source code is a mitigation against supply chain risk, not a cure all.

The core Postgres source code is only one consideration. Production Postgres deployments often make use of a broad range of Postgres tools and extensions. It is important to consider how you will maintain a secure supply chain for this full ecosystem of software. It is also necessary to think through the various "stops" along the software supply chain path between the Postgres source code and the binaries running on your server.

One of the values provided by open source software vendors is maintenance and certification of the software supply chain. This is where the Common Criteria come in: the CC EAL 2+ certification provides third party validation of the software supply chain and support to ensure you are installing only trusted components.

Yes, Postgres is Secure

There is no shortage of good reasons to adopt Postgres in your data management toolbox. Postgres' security capabilities are just one of them.

It is a testament to the global Postgres community that open source Postgres can stand up to the security standards developed by leading proprietary database vendors and imposed by security conscious enterprise users, and enable security conscious users to adopt Postgres consistent with industry best practices.

The Answer is Postgres; The Question is How?

Paul.Laurence@crunchydata.com (Paul Laurence) — Mon, 08 Feb 2021 04:00:00 EST

There is increasing consensus that Postgres is a great choice of database for a broad range of use cases. As our friends at RedMonk have said:

the answer is postgres, now what's the question again? ;-)
— Elon Mook (@monkchips) April 29, 2017

You have a number of good options for how to run Postgres: run it in VMs, as a managed service or bare metal. Benjamin Good, a Google Cloud Solutions Architect, wrote a helpful blog post of when to run databases on Kubernetes; a common question and increasingly popular and successful option.

So which deployment model is appropriate for you? As usual, it depends.

At Crunchy Data, we support customers who have made the choice for all of the above. That vantage point gives us some perspective on when you should choose which approach. Building on the Google Cloud framework, let's review the options and considerations.

Postgres on Virtual Machines: The "Old" Standard

The Google Cloud analysis describes this as the "Full Ops" option, "Where you take full responsibility for building your database, scaling it, managing reliability, setting up backups, and more," with a core trade-off between the level of effort to maintain the database against the flexibility associated with your choice of location for deployment (on-premise, public cloud, etc.).

Most Postgres users view running on VMs as a tried and true approach (though this wasn’t always the case). There are many tools for automating Postgres on VMs, enabling users to reduce the administrative level of effort. Automated or not, you are of course responsible for the database administration.

Often the choice of Postgres on VMs is most appropriate when you already have the necessary internal infrastructure and expertise to run databases and you want control over both your database and infrastructure. The decision is commonly driven by economies of scale. In the public cloud you may prefer VM-based Postgres deployments due to flexibility in version availability, configuration options and availability of extensions.

In these cases, Postgres on VMs provides the right balance of flexibility, control and investment.

Postgres on Kubernetes: Is Kubernetes Ready for Databases?

PostgreSQL on Kubernetes is a new option by database standards. Kubernetes provides many benefits for running applications, including efficiency, automation, or infrastructure abstraction. But what about running databases?

It is a common question of whether Kubernetes has matured to the point where it is ready for stateful workloads. My colleague Greg Smith recounts that it took VMs about five years of maturation before users could reliably run Postgres. By that time, following appropriate testing of their configuration, users could prove that their VM platform was sufficiently reliable to run a database. Has Kubernetes reached the maturity stage where you can reliably run databases on that platform?

The Cloud Native Computing Foundation (CNCF) Cloud Native Survey 2020 provides interesting data. The CNCF Survey reports that 55% of respondents are using stateful applications in containers in production.

Today, it isn’t just early adopters running Postgres on Kubernetes. Many Crunchy Data customers have successfully deployed Postgres on Kubernetes, and have been kind enough to talk about it.

While there were initially some rough edges in running Postgres on Kubernetes, it has come a long way since the early days. The Kubernetes ecosystem has evolved considerably since the days of "Petsets" and Third Party Resources with the evolution of the Operator model from CoreOS and the variety of storage options currently available to users.

Similar to VMs, with time Postgres admins are getting more comfortable running Postgres on Kubernetes. To help build that confidence, our team recently worked with Red Hat to validate Postgres containers on Kubernetes.

Postgres on Kubernetes: But Postgres?

The Google Cloud post describes databases on Kubernetes as "closer to the full-ops option," while indicating that "You do get some benefits in terms of the automation Kubernetes provides to keep the database application running." In the Google Cloud's analysis, the key decision point is whether the database has "Kubernetes-friendly features" or "Is there an operator project to help?"

We of course believe Postgres is a great database. Its core functionality is extended through a robust open source software ecosystem. With the right tools (like the Postgres Operator) and expertise, Postgres is Kubernetes friendly. Was it designed from scratch to work with cloud native primitives? No.? Does it work with cloud native primitives from continued development and evolution? Absolutely.

Kubernetes provides many interesting capabilities for running databases, particularly when coupled with an Operator. The ability to scale up nodes uniformly makes it easier to manage hardware for databases as they grow. Kubernetes features like node affinity and tolerations allow admins to make decisions about where Postgres instances are deployed. These tools combine to enable database workloads to benefit from high availability or specific hardware.

That said, running Kubernetes, and applications on Kubernetes, comes with its share of administrative requirements. Operators and tools such as Helm and Kustomize are all helpful in easing the administrative burden, but automation and orchestration associated with Kubernetes does not come for free.

In the context of Postgres, the question seems to boil down to whether a user values the benefits of Kubernetes sufficiently to sustain the incremental administration.

Fully Managed Postgres

What about fully managed database options for Postgres? As database workloads move to the cloud, managed services are an attractive option for deploying databases. Google Cloud describes this as the "low-ops choice" as the ‘managed service’ handles a number of the database administration tasks for you - including backups, patching and scaling.

Crunchy Data launched Crunchy Bridge to provide users with the best managed Postgres experience. By combining industry leading Postgres expertise, with extensive operational expertise running Postgres as a service, Crunchy Bridge provides users with the flexibility of Postgres control, Postgres extensions and dedicated Postgres support.

Managed Postgres also enables you to benefit from hands-on expertise from Crunchy Data Postgres engineers. As my colleague Craig likes to say, “Fully managed Postgres from Crunchy provides you with the benefit of the Crunchy Data engineers with decades of experience running large scale Postgres databases and database as a service. “ While many days you may not need this level of expertise to run Postgres, when you do, you can have confidence the scale we operate Postgres at has given us experience to effectively troubleshoot any problem with Postgres.

For users who are not interested in maintaining a database server, managed Postgres is a great option. And you still get the best part: Postgres

So what should you do?

Good news. Deciding to use Postgres is a great start, we like to think it’s the right start. As to how best to deploy it, you have a number of options.

To ensure users have a variety of Postgres options, Crunchy Data provides Crunchy High Availability PostgreSQL for VM based deployments, Crunchy PostgreSQL for Kubernetes complete with a mature Postgres Operator, and Crunchy Bridge, a managed Postgres service.

The analysis that we have seen is less of an either/or decision and more of a question of when and where. Similar to the hybrid cloud reality that enterprises of all sizes tend to use, Postgres users often choose some combination of these deployment models based upon their team requirements and organizational standards. The choice is less about deciding which model to use for all applications, and more about which choice to use for a given project.

For many projects, using a managed service works well if the use cases require a "set and forget" Postgres deployment. The choice between deploying Postgres on VMs and Kubernetes is less about a decision for more or less management or automation but rather based on whether a group is standardizing on Kubernetes.

Are you still wondering how you should deploy PostgreSQL? Let me offer a decision tree to help you:

As with everything, each decision has trade offs but as long as you are starting with "The answer is Postgres," it is unlikely you'll get too far off the correct path. If you need help, Crunchy Data is here to assist.

Postgres: The Friendly Relational Member of Your Data Management Toolbox

Paul.Laurence@crunchydata.com (Paul Laurence) — Fri, 15 Jan 2021 04:00:00 EST

While every year feels like the year of Postgres these days, 2012 did not. For most observers, 2012 was the year of "Big Data" as NoSQL technologies like Hadoop and MongoDB were demonstrating powerful new data management use cases.

At the same time, Crunchy Data was still just an idea and was beginning to engage with various consumers of database technology on how this wave of new open source tools were impacting their data strategy. During these early discussions - and many since - we heard how organizations were building a modern data management toolbox. The tools were being selected to support the next generation of application development. Organizations were including a NoSQL tool like Hadoop, one or two legacy databases, a data caching or message broker technology, and a modern relational tool as the new SQL standard. And the relational database tool of choice that we heard about time and time again, was Postgres.

Despite the enthusiasm for Big Data, the macro trend driving change in how organizations think about databases, the more nuanced trend was toward open source software as both a viable and innovative solution to data problems. While Linux was an increasingly well-established operating system of choice, prior to the Big Data movement, open source data tooling was still relegated to the early adopters and enterprise "exception" servers. For Big Data in 2012, open source was not only a credible option, it was the only option for addressing these new challenges.

No more. As organizations began to get comfortable with open source for their "NoSQL" data stores, the next logical question was,"Why not use open source for our relational databases?" The death of relational and SQL was grossly overstated, and the forward thinking users were betting on Postgres as their relational database for the future.

As organizations re-evaluated their long held data management perspective, Postgres was ready and waiting. What began as a well thought out codebase at UC Berkeley had developed into a rock solid and full featured relational database ready to take on all sorts of workloads.

Postgres was steadily innovating as a community-driven open source database, adding new capabilities with each release. Users who had looked at Postgres long ago were surprised to learn that it now supported native replication, JSONB, query parallelism, and partitioning. This was on top of many features that had Postgres at parity with other relational databases, such as multi-version concurrency control, native procedural language support (PL/SQL, Python, Perl, R), advanced authentication mechanisms (certificates, Kerberos, ActiveDirectory) and more. With each release, the Postgres community brought it closer to readiness for the wide range of applications backed by relational databases, along with the wide range of Postgres extensions like PostGIS that enable even more rapid innovation and workload specific capabilities.

We at Crunchy Data have continued to be impressed by the Postgres community and its steady innovation. Year after year, the project gets better. The rise of the cloud and new platforms such as Kubernetes have only accelerated the trend toward open source infrastructure, and open source data management by extension.

The recent selection of Postgres as the "Database of the Year" for the third time in four years is by no means an overnight success story, but is well deserved recognition for a database decades in the making. As organizations look for the relational database of the future, Postgres is ready and waiting. We are proud to be among the leading contributors and supporters of this important movement.