Scaling Postgres

In this episode of Scaling Postgres, we discuss the process of decomposing your database, how to search text in Postgres quickly, how to understand the output of pg_stat_activity and how to use window functions to solve a use case. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://about.gitlab.com/blog/2022/06/02/splitting-database-into-main-and-ci/ https://gitlab.com/groups/gitlab-org/-/epics/6168 https://docs.gitlab.com/ee/development/database/multiple_databases.html https://www.rubberduckdevshow.com/episodes/49-fast-text-search-in-postgres/ https://www.depesz.com/2022/07/05/understanding-pg_stat_activity/ https://www.cybertec-postgresql.com/en/handling-bonus-programs-in-sql/ https://www.crunchydata.com/blog/choosing-a-postgresql-number-format https://www.crunchydata.com/blog/data-to-go-postgres-logical-replication https://www.percona.com/blog/postgresql-replication-and-conflicts/ https://www.timescale.com/blog/state-of-postgresql-2022-fi

In this episode of Scaling Postgres, we discuss psql shortcuts, how to debug deadlocks, how to find & stop queries and how to understand and use SCRAM authentication. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.postgresql.org/about/news/postgresql-15-beta-2-released-2479/ https://www.crunchydata.com/blog/postgres-locking-when-is-it-concerning https://www.percona.com/blog/importance-of-postgresql-vacuum-tuning-and-custom-scheduled-vacuum-job/ https://www.cybertec-postgresql.com/en/transaction-anomalies-with-select-for-update/ https://andreas.scherbaum.la/blog/archives/1117-ctid-and-other-PostgreSQL-table-internals.html https://www.postgresql.fastware.com/blog/introducing-publication-row-filters https://pganalyze.com/blog/5mins-postgres-tuning-statistics-target-selectivity-spatial-indexes https://andreas.scherbaum.la/blog/archives/1118-2-+-2-4-16.html https://supabase.com/blog/2022/06/28/partial-postgresql-data-dumps-with-rls ht

In this episode of Scaling Postgres, we discuss psql shortcuts, how to debug deadlocks, how to find & stop queries and how to understand and use SCRAM authentication. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://elephas.io/psql-helper-managing-connections-and-simplifying-queries/ https://www.cybertec-postgresql.com/en/debugging-deadlocks-in-postgresql/ https://adamj.eu/tech/2022/06/20/how-to-find-and-stop-running-queries-on-postgresql/ https://www.youtube.com/watch?v=whvXiAa_T6A https://andreas.scherbaum.la/blog/archives/1116-PostgreSQL-Upgrades-are-hard!.html https://fluca1978.github.io/2022/06/23/PostgreSQLOrdinality.html https://www.crunchydata.com/blog/indexes-selectivity-and-statistics https://blog.rustprooflabs.com/2022/06/h3-indexes-on-postgis-data https://pganalyze.com/blog/5mins-postgres-fuzzy-text-search-case-insensitive-ICU-collations https://www.timescale.com/blog/how-we-made-data-aggregation-better-and-faster-on-postgre

In this episode of Scaling Postgres, we discuss a PG 14.4 release, how to skip transactions with logical replication, how to use schemas and Citus going fully open source. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.postgresql.org/about/news/postgresql-144-released-2470/ https://www.postgresql.fastware.com/blog/addressing-replication-conflicts-using-alter-subscription-skip https://aaronoellis.com/articles/using-postgres-schemas https://www.citusdata.com/blog/2022/06/17/citus-11-goes-fully-open-source/ https://www.cybertec-postgresql.com/en/case-insensitive-pattern-matching-in-postgresql/ https://blog.brendanscullion.com/postgresql-text-search https://www.crunchydata.com/blog/quick-and-easy-postgres-data-compare https://sqlfordevs.io/uuid-prevent-enumeration-attack https://andreas.scherbaum.la/blog/archives/1113-My-PostgreSQL-database-is-empty!.html https://pganalyze.com/blog/5mins-postgres-reducing-aws-aurora-io-costs-table-partitio

In this episode of Scaling Postgres, we discuss a Postgres 14 out-of-cycle release next week, whether you should upgrade, the state of Postgres survey and a tool for automatic indexing. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode:

In this episode of Scaling Postgres, we discuss a severe bug with index creation in Postgres 14, view permissions options, dealing with logical replication conflicts and looking under the hood of AlloyDB. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://pganalyze.com/blog/5mins-postgres-14-reindex-create-index-concurrently-bug-amcheck-extension https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=042b584c7f7d6216c54359c0ee0f613ba3b3d9c2 https://www.postgresql.org/docs/14/amcheck.html https://www.cybertec-postgresql.com/en/view-permissions-and-row-level-security-in-postgresql/ https://www.postgresql.fastware.com/blog/how-to-handle-logical-replication-conflicts-in-postgresql https://cloud.google.com/blog/products/databases/alloydb-for-postgresql-intelligent-scalable-storage https://cloud.google.com/blog/products/databases/alloydb-for-postgresql-columnar-engine https://www.enterprisedb.com/blog/using-virtual-environment-plpython3-post

In this episode of Scaling Postgres, we discuss mistakes you can make when doing schema changes, how best to cancel statements, looking into pg_rman, and looking at a pedantry removal. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://postgres.ai/blog/20220525-common-db-schema-change-mistakes https://www.cybertec-postgresql.com/en/cancel-hanging-postgresql-query/ https://www.highgo.ca/2022/05/27/how-to-do-backup-and-restore-for-postgresql-using-pg_rman/ https://mydbanotebook.org/post/pedantry/ https://pganalyze.com/blog/5mins-postgres-15-speeding-up-sort-performance https://www.timescale.com/blog/opentelemetry-where-sql-is-better-than-the-original/ https://www.crunchydata.com/blog/exporter2perfdata-connecting-postgres-monitoring-tools https://postgresql.life/post/masahiko_sawada/ https://www.rubberduckdevshow.com/episodes/43-typed-or-untyped-ruby/  

In this episode of Scaling Postgres, we discuss the Beta 1 release of Postgres 15, optimizing a query by breaking it down, improvements to sort performance and prepared statements for partitioned tables. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.postgresql.org/about/news/postgresql-15-beta-1-released-2453/ https://www.crunchydata.com/blog/postgres-query-optimization-left-join-vs-union-all https://www.citusdata.com/blog/2022/05/19/speeding-up-sort-performance-in-postgres-15/ https://amitlan.com/2022/05/16/param-query-partition-woes.html https://pganalyze.com/blog/5mins-postgres-partition-pruning-prepared-statements-generic-vs-custom-query-plans https://www.ongres.com/blog/solving-postgresql-statistic-corruption/ https://www.craigkerstiens.com/2022/05/18/unfinished-business-with-postgres/ https://ptc-it.de/collate-docker-postgres-bug/ https://geopet85.github.io/2022/05/15/rds-partitioned-tables-export.html https://www.crunchydata.c

In this episode of Scaling Postgres, we discuss parallel server-side backup compression, IO in Postgres, parquet files and the new pg_stat_monitor extension. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.postgresql.org/about/news/postgresql-143-137-1211-1116-and-1021-released-2449/ https://cloud.google.com/blog/products/databases/introducing-alloydb-for-postgresql https://www.youtube.com/watch?v=o5pKfH0Tonw https://www.cybertec-postgresql.com/en/time-zone-management-in-postgresql/ https://www.enterprisedb.com/blog/pg-phriday-what-you-should-know-about-basic-postgres-authentication https://www.crunchydata.com/blog/the-vectors-of-database-encryption https://www.crunchydata.com/blog/demystifying-database-performance-for-developers https://pganalyze.com/blog/5mins-postgres-for-app-developers-tables-indexes https://kiranrao.ca/2022/05/04/zero-downtime-migrations.html https://www.se-radio.net/2022/05/episode-511-ant-wilson-on-supabase-post

In this episode of Scaling Postgres, we discuss parallel server-side backup compression, IO in Postgres, parquet files and the new pg_stat_monitor extension. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: http://rhaas.blogspot.com/2022/05/parallel-server-side-backup-compression.html https://www.youtube.com/watch?v=3Oj7fBAqVTw https://www.crunchydata.com/blog/parquet-and-postgres-in-the-data-lake https://www.percona.com/blog/understand-your-postgresql-workloads-better-with-pg_stat_monitor/ https://www.timescale.com/blog/point-in-time/ https://www.cybertec-postgresql.com/en/postgresql-15-using-merge-in-sql/ https://www.highgo.ca/2022/04/30/upcoming-feature-in-postgresql-15-merge-statement/ https://andreas.scherbaum.la/blog/archives/1110-Its-beer-oclock-for-PostgreSQL.html https://pganalyze.com/blog/5mins-postgres-15-beta1-incremental-materialized-views-pg-ivm https://www.highgo.ca/2022/05/06/atomic-commit-with-2-phase-commit-in-fdw-distributed-setu

In this episode of Scaling Postgres, we discuss ways you can optimize the planner, becoming part of the Postgres Community, considerations for large databases and mapping three different types of spatial data. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.aha.io/engineering/articles/optimizing-with-the-postgresql-deterministic-query-planner https://pganalyze.com/blog/5mins-postgres-tuning-deterministic-query-planner-extended-statistics-join-collapse-limits https://www.timescale.com/blog/the-future-of-community-in-light-of-babelfish/ https://www.percona.com/blog/working-with-large-postgresql-databases/ https://blog.rustprooflabs.com/2022/04/postgis-h3-intro https://selectfrom.dev/mapping-fracking-wells-in-colorado-with-postgresql-db17fc3b8812 https://www.cybertec-postgresql.com/en/analyzing-historical-flight-data-with-mobilitydb/ https://www.highgo.ca/2022/04/29/global-deadlock-in-a-distributed-database-cluster/ https://www.endpointde

In this episode of Scaling Postgres, we discuss dealing with slow queries, upgrading a 4TB database, running Postgres on ZFS and handling a storage stampede. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://pgdash.io/blog/slow-queries-postgres.html https://retool.com/blog/how-we-upgraded-postgresql-database/ https://lackofimagination.org/2022/04/our-experience-with-postgresql-on-zfs/ https://www.enterprisedb.com/blog/steady-storage-stampede https://tech.instacart.com/the-vanishing-thread-and-postgresql-tcp-connection-parameters-93afc0e1208c https://www.cybertec-postgresql.com/en/find-and-fix-a-missing-postgresql-index/ https://www.percona.com/blog/postgresql-14-b-tree-index-reduced-bloat-with-bottom-up-deletion/ https://pganalyze.com/blog/5mins-postgres-hot-updates-vs-bottom-up-index-deletion https://www.youtube.com/watch?v=oJj-pltxBUM https://www.postgresql.fastware.com/blog/two-phase-commits-for-logical-replication-publications-subscription

In this episode of Scaling Postgres, we discuss working with multiranges, missing Postgres metrics, PostGIS for newbies and videos from CitusCon. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.cybertec-postgresql.com/en/multiranges-in-postgresql-14/ https://coroot.com/blog/pg-missing-metrics https://blog.crunchydata.com/blog/postgis-for-newbies https://www.youtube.com/playlist?list=PLlrxD0HtieHjSzUZYCMvqffEU5jykfPTd#cituscon2022 https://pganalyze.com/blog/5mins-postgres-sql-json-table-postgres15 https://yugonagata-pgsql.blogspot.com/2022/04/pgivm-postgresql-extension-providing.html https://www.postgresql.org/about/news/update-on-the-trademark-actions-against-the-postgresql-community-2437/ https://pgstef.github.io/2022/04/15/pgbackrest_multi-repositories_tips_and_tricks.html https://postgresql.life/post/jehan-guillaume_de_rorthais/ https://www.rubberduckdevshow.com/episodes/38-small-projects/  

In this episode of Scaling Postgres, we discuss how to bin date times, search every field of a table, use JSON_TABLE and methods for a client to pick the primary database. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.cybertec-postgresql.com/en/postgresql-mapping-timestamps-date_bin/ https://www.depesz.com/2022/04/05/dba-tips-n-tricks-search-in-every-field-in-a-table/ https://www.depesz.com/2022/04/06/waiting-for-postgresql-15-json_table/ https://www.postgresql.org/docs/14/libpq-connect.html#LIBPQ-CONNECT-TARGET-SESSION-ATTRS https://www.cybertec-postgresql.com/en/new-target_session_attrs-settings-for-high-availability-and-scaling-in-postgresql-v14/ https://www.postgresql.fastware.com/blog/enhancement-to-target-session-attrs https://pganalyze.com/blog/5mins-postgres-tuning-vacuum-autovacuum https://www.endpointdev.com/blog/2022/04/shapefiles-postgis/ https://dev.to/yugabyte/how-to-enable-ssl-for-postgres-connections-5321 https://pos

In this episode of Scaling Postgres, we discuss how to optimize vacuum, how to drop roles, the new merge command in PG15 and a pgbouncer tutorial. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.enterprisedb.com/blog/pg-phriday-tidying-vacuum https://www.cybertec-postgresql.com/en/drop-role-or-drop-user-postgresql/ https://www.depesz.com/2022/03/31/waiting-for-postgresql-15-add-support-for-merge-sql-command/ https://www.postgresql.org/docs/devel/sql-merge.html https://www.youtube.com/watch?v=ddKm7a7xOpk https://www.depesz.com/2022/03/31/waiting-for-postgresql-15-sql-json-constructors/ https://www.depesz.com/2022/04/01/waiting-for-postgresql-15-is-json-predicate/ https://www.depesz.com/2022/04/01/waiting-for-postgresql-15-sql-json-query-functions/ https://www.highgo.ca/2022/03/31/parallel-commit-in-postgres-fdw/ https://pganalyze.com/blog/how-postgres-chooses-index https://postgrespro.com/blog/pgsql/5969403 https://www.citusdata.com/blog

In this episode of Scaling Postgres, we discuss a new version of pgbouncer, query parameter types, a performance improvement from a lateral join and security invoker views. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: http://www.pgbouncer.org/2022/03/pgbouncer-1-17-0 https://www.cybertec-postgresql.com/en/query-parameter-data-types-performance/ https://www.depesz.com/2022/03/22/waiting-for-postgresql-15-add-support-for-security-invoker-views/ https://pganalyze.com/blog/5mins-postgres-max-wal-size-full-page-writes-uuid-bigint https://b-peng.blogspot.com/2022/03/pgpool-debian.html https://postgresql.life/post/michael_christofides/ https://www.rubberduckdevshow.com/episodes/35-organizing-your-tests/  

In this episode of Scaling Postgres, we discuss how best to configure autovacuum, the dangers of duplicate key violations, how to set up pipelining, and how to tune max_wal_size. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.depesz.com/2022/03/13/is-my-autovacuum-configured-properly/ https://aws.amazon.com/blogs/database/hidden-dangers-of-duplicate-key-violations-in-postgresql-and-how-to-avoid-them/ https://www.cybertec-postgresql.com/en/pipeline-mode-better-performance-on-slow-network/ https://www.enterprisedb.com/blog/tuning-maxwalsize-postgresql https://www.enterprisedb.com/blog/pg-phriday-wrapping-transactions-right-round https://www.endpointdev.com/blog/2022/03/using-pgtap-automate-database-testing/ https://ctodive.com/hooks-the-secret-feature-powering-the-postgres-ecosystem-f05b3b82e0ba https://www.citusdata.com/blog/2022/03/12/how-to-benchmark-performance-of-citus-and-postgres-with-hammerdb/ https://pganalyze.com/blog/5mins-po

In this episode of Scaling Postgres, we discuss 100x optimizations, how to audit table activity, designing schemas and how to set up table checksums. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://towardsdatascience.com/how-we-optimized-postgresql-queries-100x-ff52555eabe https://supabase.com/blog/2022/03/08/audit https://pganalyze.com/blog/5mins-postgres-auditing-pgaudit-supabase-supa-audit https://www.endpointdev.com/blog/2022/03/database-design-with-documents/ https://www.cybertec-postgresql.com/en/postgresql-creating-checksums-for-tables/ https://blog.crunchydata.com/blog/performance-and-spatial-joins https://blog.crunchydata.com/blog/cql-filtering-in-pg_featureserv https://postgrespro.com/blog/pgsql/5969296 https://postgresql.life/post/ibrar_ahmed/ https://www.rubberduckdevshow.com/episodes/33-reviewing-owasp-6-10/  

In this episode of Scaling Postgres, we discuss the best ways to index with tsearch, what to check if tuples are not yet removable, wicked problems with Postgres and different pattern matching techniques. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.depesz.com/2022/03/01/how-to-index-data-with-tsearch/ https://pganalyze.com/blog/5mins-postgres-autovacuum-dead-tuples-not-yet-removable-postgres-xmin-horizon https://www.slideshare.net/AlexanderKorotkov/solving-postgresql-wicked-problems https://github.com/orioledb/orioledb https://blog.crunchydata.com/blog/how-to-cheat-at-wordle-with-postgresql https://databaseci.com/docs/migra https://b-peng.blogspot.com/2022/02/auto-failback.html https://blog.crunchydata.com/blog/cloud-data-sources-for-the-postgres-operator-for-kubernetes https://www.highgo.ca/2022/02/28/a-snippet-to-acquire-a-lightweight-lock/ https://postgresql.life/post/gianni_ciolli/ https://www.rubberduckdevshow.com/episodes/32-r

In this episode of Scaling Postgres, we discuss optimizing trigram searches, a review of Postgres replication, improvements to logical replication and a significant Timescale investment. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://alexklibisz.com/2022/02/18/optimizing-postgres-trigram-search.html https://pganalyze.com/blog/5mins-postgres-optimizing-postgres-text-search-trigrams-gist-indexes https://www.enterprisedb.com/blog/replication-revue https://www.percona.com/blog/logical-replication-decoding-improvements-in-postgresql-13-and-14/ https://www.timescale.com/blog/year-of-the-tiger-110-million-to-build-the-future-of-data-for-developers-worldwide/ https://www.shayon.dev/post/2022/47/pg-osc-zero-downtime-schema-changes-in-postgresql/ https://www.cybertec-postgresql.com/en/monitoring-google-cloud-postgresql-with-pgwatch2/ https://postgresql.verite.pro/blog/2022/02/21/psql-hack-select-except.html https://postgresql.life/post/julia_gugel/