Scaling Postgres

In this episode of Scaling Postgres, we discuss WAL growing to 11 Terabytes, pgec supporting writes, Patroni 3.0 & Citus for high availability, and setting up PostgREST. Visit https://www.scalingpostgres.com/episodes/255-integer-overflow-user-friendly-permissions-dump-logical-replication-worker-config/ to get the show notes as well as subscribe to get notified of new episodes.

In this episode of Scaling Postgres, we discuss how to detect & handle integer overflows, a wish for user-friendly permissions, using a dump to start logical replication and configuring background workers. Visit https://www.scalingpostgres.com/episodes/255-integer-overflow-user-friendly-permissions-dump-logical-replication-worker-config/ to get the show notes as well as subscribe to get notified of new episodes.

In this episode of Scaling Postgres, we discuss PG Edge Cache, using Postgres & OpenAI, migrating citext to case-insenstive collations and PG16 compression options. Visit https://www.scalingpostgres.com to get the show notes as well as subscribe to get notified of new episodes.  

In this episode of Scaling Postgres, we discuss working with unlogged tables, the new pg_stat_io feature, handling complex type constraints and choosing the best text type. Visit https://www.scalingpostgres.com to get the show notes as well as subscribe to get notified of new episodes.

In this episode of Scaling Postgres, we discuss new Postgres releases, performance secrets, things not to do and filter vs. case. Visit https://www.scalingpostgres.com to get the show notes as well as subscribe to get notified of new episodes.  

In this episode of Scaling Postgres, we discuss how to use the filter clause, create hypothetical indexes, store non-relational data and manage TOAST. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.crunchydata.com/blog/using-postgres-filter https://www.crunchydata.com/blog/will-postgres-use-my-index-hypothetical-indexing-with-postgres https://blog.rustprooflabs.com/2023/02/postgres-relational-plus https://hdombrovskaya.wordpress.com/2023/02/03/pgsql-phriday-005-relational-and-non-relational-data/ https://www.pgmustard.com/blog/storing-blobs-in-postgres https://andreas.scherbaum.la/blog/archives/1132-Relational-and-Non-relational-Data-PGSQL-Phriday-005.html https://www.percona.com/blog/unlocking-the-secrets-of-toast-how-to-optimize-large-column-storage-in-postgresql-for-top-performance-and-scalability/ http://peter.eisentraut.org/blog/2023/01/30/postgresql-supply-chain https://www.red-gate.com/simple-talk/databases/postgresql/postgresql-

In this episode of Scaling Postgres, we discuss resolving a performance issue, how PG16 allows you to survive without a superuser, reserving connections and handling partition management. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://medium.com/engineering-at-birdie/puzzling-postgres-a-story-of-solving-an-unreproducible-performance-issue-778075ed7998 http://rhaas.blogspot.com/2023/01/surviving-without-superuser-coming-to.html https://www.red-gate.com/simple-talk/databases/postgresql/postgresql-basics-roles-and-privileges/ https://www.dbi-services.com/blog/postgresql-16-reserved_connections/ https://pganalyze.com/blog/5mins-postgres-16-superuser-reserved-connections https://www.timescale.com/blog/how-timescaledb-solves-common-postgresql-problems-in-database-operations-with-data-retention-management/ https://www.percona.com/blog/wal-compression-in-postgresql-and-recent-improvements-in-version-15/ https://www.cybertec-postgresql.com/en/uniq

In this episode of Scaling Postgres, we discuss issues with locking tables, foreign keys and slow decimals. We also cover when joins use memoize. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links: https://thebuild.com/blog/2023/01/16/ok-sometimes-you-can-lock-tables/ https://thebuild.com/blog/2023/01/18/a-foreign-key-pathology-to-avoid/ https://thebuild.com/blog/2023/01/15/how-slow-is-decimal-anyway/ https://pganalyze.com/blog/5mins-postgres-memoize-speed-up-joins https://www.cybertec-postgresql.com/en/usql-universal-psql/ https://www.crunchydata.com/blog/solving-advent-of-code-2022-using-postgres-day-13 https://gavinray97.github.io/blog/postgres-wire-protocol-jdk-21 https://www.citusdata.com/blog/2023/01/18/debugging-postgres-ci-failures-faster-4-tips/ https://postgres.fm/episodes/auditing https://postgresql.life/post/miroslav_sedivy/ https://www.rubberduckdevshow.com/episodes/70-teaching-young-developers-with-melissa-amaya/  

In this episode of Scaling Postgres, we discuss different pagination solutions, how to return modifications, soft deletion alternatives and how to use JSON with PostgreSQL. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.cybertec-postgresql.com/en/pagination-problem-total-result-count/ https://sqlfordevs.com/returning-modified-rows https://brandur.org/fragments/deleted-record-insert https://dev.to/ftisiot/how-to-json-in-postgresqlr-1lj7 https://momjian.us/main/writings/pgsql/beyond.pdf http://peter.eisentraut.org/blog/2023/01/11/postgresql-largest-commits https://supabase.com/blog/postgres-foreign-data-wrappers-rust https://medium.com/@walttonm/using-the-window-function-row-number-to-remove-duplicates-in-postgresql-5aef1edfb78c https://pgdash.io/blog/horizontally-scaling-postgresql.html https://www.percona.com/blog/upgrading-postgresql-extensions/ https://www.timescale.com/blog/a-postgresql-developers-perspective-six-interesting-patches-fr

In this episode of Scaling Postgres, we discuss how to generate test data, how WAL archiving is faster in PG15, how to set date statistics on a timestamp and how to avoid useless indexes. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://kmoppel.github.io/2022-12-23-generating-lots-of-test-data-with-postgres-fast-and-faster/ https://www.percona.com/blog/speed-up-of-the-wal-archiving-in-postgresql-15/ https://www.cybertec-postgresql.com/en/improving-group-by-with-create-statistics/ https://mydbanotebook.org/post/too-many-indexes/ https://www.cybertec-postgresql.com/en/json-logs-in-postgresql-15/ http://peter.eisentraut.org/blog/2022/12/27/postgresql-hidden-gems https://www.softwareandbooz.com/advent-of-code-2022-days-6-10-with-postgresql/ https://www.crunchydata.com/blog/solving-advent-of-code-2022-using-postgres-day-6 https://www.crunchydata.com/blog/solving-advent-of-code-2022-using-postgres-day-7 https://www.crunchydata.com/blog/solving-a

In this episode of Scaling Postgres, we discuss columnar table storage, solving Advent of Code using Postgres, new features coming in Postgres 16 and when Postgres development happens. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.percona.com/blog/powering-postgresql-15-with-columnar-tables/ https://github.com/citusdata/citus/blob/main/src/backend/columnar/README.md https://www.softwareandbooz.com/advent-of-code-2022-with-postgresql-part1/ https://www.crunchydata.com/blog/solving-advent-of-code-2022-using-postgres-day-1 https://www.crunchydata.com/blog/solving-advent-of-code-2022-using-postgres-day-2 https://www.crunchydata.com/blog/solving-advent-of-code-2022-using-postgres-day-3 https://www.crunchydata.com/blog/solving-advent-of-code-2022-using-postgres-day-4 https://www.crunchydata.com/blog/solving-advent-of-code-2022-using-postgres-day-5 https://www.depesz.com/2022/12/14/waiting-for-postgresql-16-add-grantable-maintain-privilege

In this episode of Scaling Postgres, we discuss ENUMs vs. check constraints, querying table facets with roaring bitmaps, a better way to handle scaling and whether you should store your business logic in Postgres. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.crunchydata.com/blog/enums-vs-check-constraints-in-postgres https://www.cybertec-postgresql.com/en/faceting-large-result-sets/ https://github.com/cybertec-postgresql/pgfaceting https://pganalyze.com/blog/5mins-postgres-roaring-bitmaps-pgfaceting-query-performance https://thenewstack.io/iso-better-scaling-instacart-drops-postgres-for-amazon-dynamodb/ https://www.rubberduckdevshow.com/episodes/68-should-you-store-business-logic-in-your-database/ https://www.ongres.com/blog/debugging-postgres-wal-events-with-pg_walinspect/ https://kmoppel.github.io/2022-12-09-the-bountiful-world-of-postgres-indexing-options/ https://andreas.scherbaum.la/blog/archives/1125-PGSQL-Phriday-003-What-is-

In this episode of Scaling Postgres, we discuss merging indexes vs. a composite index, implementing transparent column encryption, developing trusted language extensions, and reviewing the WAL archive module. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://sirupsen.com/index-merges https://supabase.com/blog/transparent-column-encryption-with-postgres https://aws.amazon.com/blogs/aws/new-trusted-language-extensions-for-postgresql-on-amazon-aurora-and-amazon-rds/ https://pganalyze.com/blog/5mins-postgres-pg-tle-custom-extensions-aws-rds-aurora-trusted-languages-hooks https://github.com/aws/pg_tle https://www.percona.com/blog/new-wal-archive-module-library-in-postgresql-15/ https://sqlasylum.wordpress.com/2022/11/29/pgsql-phriday-003-what-is-the-community-to-you/ https://www.softwareandbooz.com/postgresql-community-passionate-knowledgeable-and-thoughtful/ https://www.scarydba.com/2022/12/02/pgsql-phriday-3-what-is-the-postgresql-community-to-y

In this episode of Scaling Postgres, we discuss how PG15 helps reduce replication lag, how to get the parameters used in prepared statements, PostGIS Day and how to use multiple pgbouncers. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://pganalyze.com/blog/5mins-postgres-performance-in-lists-vs-any-operator-bind-parameters https://sqlfordevs.com/ghost-conditions-for-unindexed-columns https://www.crunchydata.com/blog/percentage-calculations-using-postgres-window-functions https://www.postgresonline.com/journal/index.php?/archives/406-VARIADIC-Unnest.html https://supabase.com/blog/sql-or-nosql-both-with-postgresql https://www.cybertec-postgresql.com/en/error-permission-denied-schema-public/ https://www.highgo.ca/2022/11/25/global-index-benchmark-with-pgbench/ https://fluca1978.github.io/2022/11/24/PostgreSQLDataSyncRetry.html https://postgres.fm/episodes/materialized-views https://postgresql.life/post/tobias_bussmann/  

In this episode of Scaling Postgres, we discuss how PG15 helps reduce replication lag, how to get the parameters used in prepared statements, PostGIS Day and how to use multiple pgbouncers. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://pganalyze.com/blog/5mins-postgres-15-maintenance-io-concurrency-reduce-replication-lag https://www.cybertec-postgresql.com/en/explain-that-parameterized-statement/ https://blog.rustprooflabs.com/2022/11/route-the-interesting-things-postgis-day2022 https://www.crunchydata.com/blog/postgres-at-scale-running-multiple-pgbouncers https://www.crunchydata.com/blog/postgres-query-boost-using-any-instead-of-in https://fluca1978.github.io/2022/11/16/psqlEmacsClient.html https://people.planetpostgresql.org/devrim/index.php?/archives/121-Improved-aarch64-arm64-support-is-available-in-the-PostgreSQL-RPM-repository..html https://postgrespro.com/blog/pgsql/5969859 https://www.crunchydata.com/blog/multi-cloud-strategies-wi

In this episode of Scaling Postgres, we discuss the release of Postgres 15.1, a Postgres 10 retrospective, how to reduce replication lag and using listen & notify. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.postgresql.org/about/news/postgresql-151-146-139-1213-1118-and-1023-released-2543/ https://jkatz05.com/post/postgres/postgres-10-tribute/ https://kmoppel.github.io/2022-11-09-postgres-v15-a-billion-transactions-later/ https://www.citusdata.com/blog/2022/11/10/reducing-replication-lag-with-io-concurrency-in-pg15/ https://www.enterprisedb.com/blog/listening-postgres-how-listen-and-notify-syntax-promote-high-availability-application-layer http://amitkapila16.blogspot.com/2022/11/logical-replication-improvements-in.html https://pganalyze.com/blog/5mins-postgres-15-logical-replication-replica-identity https://www.postgresql.fastware.com/blog/performance-impact-of-row-filters-and-column-lists-in-logical-replication https://www.ti

In this episode of Scaling Postgres, we discuss how sequential scans can be forced, the best way to rename a table without downtime, different Postgres terminology and the PGSQL Phriday blogging event. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://code.jeremyevans.net/2022-11-01-forcing-sequential-scans-on-postgresql.html https://pganalyze.com/blog/5mins-postgres-large-integers-causing-sequential-scan-instead-of-using-index https://brandur.org/fragments/postgres-table-rename https://www.crunchydata.com/blog/challenging-postgres-terminology https://andreas.scherbaum.la/blog/archives/1122-PGSQL-Phriday-002-PostgreSQL-Backup-and-Restore.html https://blog.rustprooflabs.com/2022/11/pgsql-phriday-002 https://mydbanotebook.org/post/how-to-do-proper-backups/ https://www.scarydba.com/2022/11/04/validating-backups-pgsql-phriday-002/ https://www.softwareandbooz.com/adventures-in-postgresql-backups/ http://rhaas.blogspot.com/2022/11/pgbasebackup-coul

In this episode of Scaling Postgres, we discuss how to set shared buffers, the different WAL compression algorithm options, how to use the merge command and how to contribute to Postgres. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.enterprisedb.com/blog/harnessing-shared-buffers-part-2 https://pganalyze.com/blog/5mins-postgres-tuning-shared-buffers-pgbench-TPROC-C-TPROC-H https://www.enterprisedb.com/blog/you-can-now-pick-your-favorite-compression-algorithm-your-wals https://www.enterprisedb.com/blog/merge-command https://postgres.fm/episodes/contributing-to-postgres https://supabase.com/blog/postgresql-commitfest http://peter.eisentraut.org/blog/2022/10/25/postgresql-15-statistics https://www.highgo.ca/2022/10/28/cross-partition-uniqueness-guarantee-with-global-unique-index/ https://www.crunchydata.com/blog/postgres-databases-and-schemas https://www.crunchydata.com/blog/easier-upgrades-and-image-management-for-postgres-in-kubernetes

In this episode of Scaling Postgres, we discuss why and what to do when updates are slow, use cases for lateral joins, moving from serial to identity for auto-incrementing ids and changes to Postgres 15. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.cybertec-postgresql.com/en/why-are-my-postgresql-updates-getting-slower/ https://pganalyze.com/blog/5mins-postgres-debug-UPDATE-bloated-tables-auto-explain-pageinspect https://sqlfordevs.com/for-each-loop-lateral-join https://adamj.eu/tech/2022/10/21/migrate-postgresql-ids-serial-identity-django-4.1/ https://blog.rustprooflabs.com/2022/10/postgres-15-config-changes https://pgconfig.rustprooflabs.com/param/change/14/15 https://www.enterprisedb.com/blog/exclusive-backup-mode-finally-removed-postgres-15 https://fluca1978.github.io/2022/10/21/PostgreSQL15JsonLogs.html https://www.enterprisedb.com/blog/postgres-15-adds-copy-header-and-matching https://www.endpointdev.com/blog/2022/10/upgrading-

In this episode of Scaling Postgres, we discuss the release of Postgres 15, the performance of different file systems with Postgres, options for connection pooling and how to secure your connections. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.postgresql.org/about/news/postgresql-15-released-2526/ https://www.crunchydata.com/blog/be-ready-public-schema-changes-in-postgres-15 https://www.crunchydata.com/blog/a-look-at-postgres-15-merge-command-with-examples https://www.enterprisedb.com/blog/postgres-vs-file-systems-performance-comparison https://www.cybertec-postgresql.com/en/pgbouncer-types-of-postgresql-connection-pooling/ https://tailscale.com/blog/introducing-pgproxy/ https://www.timescale.com/blog/vpc-peering-from-zero-to-hero/ https://www.softwareandbooz.com/pgsql-phriday-001-wrap-up-and-review/ https://knock.app/blog/postgres-index-findings https://pganalyze.com/blog/5mins-postgres-15-managing-indexes https://www.percona.com/b