Scaling Postgres

In this episode of Scaling Postgres, we review articles covering terabyte scaling, PostgreSQL 11, managing connections and backup of terabyte databases. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.postgresql.org/about/news/1894/ https://momjian.us/main/blogs/pgblog/2018.html#October_15_2018 https://blog.2ndquadrant.com/add-new-table-column-default-value-postgresql-11/ https://www.cybertec-postgresql.com/en/ideas-for-scaling-postgresql-to-multi-terabyte-and-beyond/ https://brandur.org/postgres-connections https://blog.okmeter.io/pgbouncer-monitoring-improvements-in-recent-versions-5905f5d1a1ed https://medium.com/leboncoin-engineering-blog/managing-postgresql-backup-and-replication-for-very-large-databases-61fb36e815a0 https://www.citusdata.com/blog/2018/10/17/commenting-your-postgresql-database/ https://www.youtube.com/user/postgresopen/videos https://www.youtube.com/channel/UCLjZee4-RXhkOxz-p-h5hBg/videos http://pgsqlpgpool.blogspot.

In this episode of Scaling Postgres, we review articles covering all about inserts, how to handle nearest neighbor, freezing confirmation and Comment. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.postgresql.org/about/news/1893/ https://pgdash.io/blog/postgres-insert.html https://info.crunchydata.com/blog/wont-you-be-my-neighbor-quickly-finding-who-is-nearby https://blog.2ndquadrant.com/managing-freezing/ http://www.databasesoup.com/2012/09/freezing-your-tuples-off-part-1.html http://www.databasesoup.com/2012/10/freezing-your-tuples-off-part-2.html https://www.postgresql.org/docs/10/static/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND https://www.cybertec-postgresql.com/en/index-your-foreign-key/ https://www.compose.com/articles/postgresql-tips-documenting-the-database/ https://blog.2ndquadrant.com/pg-phriday-stored-procedures-postgres-11/ https://severalnines.com/blog/dynamic-monitoring-postgresql-instances-using-pgtop https://www.per

In this episode of Scaling Postgres, we review articles covering per-table autovacuum, FDW for data synchronization, distinct on and Postgres Open. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.keithf4.com/per-table-autovacuum-tuning/ https://medium.com/@jarnaldich/postgres-foreign-data-wrappers-for-synchronization-af16174735a3 https://www.geekytidbits.com/postgres-distinct-on/ https://www.youtube.com/user/postgresopen/videos https://pganalyze.com/blog/postgres11-jit-compilation-auto-prewarm-sql-stored-procedures https://blog.2ndquadrant.com/webinar-postgresql-11-new-features/ https://www.percona.com/blog/2018/10/02/scaling-postgresql-using-connection-poolers-and-load-balancers-for-an-enterprise-grade-environment/ https://momjian.us/main/blogs/pgblog/2018.html#October_3_2018 http://blog.cleverelephant.ca/2018/09/postgis-external-storage.html http://blog.cleverelephant.ca/2018/09/postgis-sprint-1.html http://blog.cleverelephant.ca/2018/10/

In this episode of Scaling Postgres, we review articles covering sysadmin concerns, the power of indexing, pgbouncer monitoring and pg_prewarm. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.postgresql.org/about/news/1890/ http://postgis.net/2018/09/23/postgis-2.5.0/ https://heapanalytics.com/blog/engineering/clocksource-aws-ec2-vdso https://news.ycombinator.com/item?id=18043439 https://blog.2ndquadrant.com/video-power-indexing-in-postgresql/ https://blog.okmeter.io/use-red-and-real-world-pgbouncer-monitoring-61b34ebeebb8 https://postgresrocks.enterprisedb.com/t5/Postgres-Gems/Autoprewarm-a-new-functionality-in-pg-prewarm/ba-p/1908 https://blog.2ndquadrant.com/on-rocks-and-sand/ https://www.citusdata.com/blog/2018/09/27/fun-with-sql-unions-in-postgres/ https://severalnines.com/blog/understanding-check-constraints-postgresql  

In this episode of Scaling Postgres, we review articles covering a CTE warning, Postgres 11 new features, death by database and column correlation. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://medium.com/@hakibenita/be-careful-with-cte-in-postgresql-fca5e24d2119 https://lwn.net/Articles/764515/ http://www.craigkerstiens.com/2018/09/20/postgresql-11-a-first-look/ http://momjian.us/main/writings/pgsql/features.pdf https://info.crunchydata.com/blog/why-covering-indexes-are-incredibly-helpful https://blog.2ndquadrant.com/partitioning-improvements-pg11/ https://ovid.github.io/articles/death-by-database.html https://www.cybertec-postgresql.com/en/column-correlation-explained/  

In this episode of Scaling Postgres, we review articles covering parallellism in Postgres 11, JIT compiling, indexes in depth and administration. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://speakerdeck.com/macdice/parallelism-in-postgresql-11 http://blog.cleverelephant.ca/2018/09/parallel-postgis-3.html https://www.citusdata.com/blog/2018/09/11/postgresql-11-just-in-time/ https://blog.2ndquadrant.com/webinar-database-security-postgresql/ https://www.slideshare.net/LouiseGrandjonc/indexes-in-postgres https://www.percona.com/blog/2018/09/07/setting-up-streaming-replication-postgresql/ https://www.scalingpostgres.com/tutorials/postgresql-streaming-replication/ https://momjian.us/main/writings/pgsql/administration.pdf https://blog.timescale.com/announcing-1-0-enterprise-ready-time-series-database-sql-open-source-400b4342dfa0  

In this episode of Scaling Postgres, we review articles covering parallel indexing, SQL vs. ORM, logical replication upgrades and development DBs. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.cybertec-postgresql.com/en/postgresql-parallel-create-index-for-better-performance/ https://erikbern.com/2018/08/30/i-dont-want-to-learn-your-garbage-query-language.html https://blog.2ndquadrant.com/upgrading-to-postgresql-11-with-logical-replication/ https://www.citusdata.com/blog/2018/09/04/12-factor-dev-prod-parity-for-database/ https://pgsnake.blogspot.com/2018/09/why-do-we-install-as-root.html https://thebuild.com/blog/2018/09/04/char-what-is-it-good-for/ https://www.percona.com/blog/2018/08/31/tuning-postgresql-database-parameters-to-optimize-performance/  

In this episode of Scaling Postgres, we review articles covering watching Star Wars, fast text search, column defaults and importing large CSVs. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.youtube.com/watch?v=Lf--bckgEFo https://pgdash.io/blog/postgres-psql-tips-tricks.html?h https://austingwalters.com/fast-full-text-search-in-postgresql/ https://brandur.org/postgres-default https://trineo.com/blog/2018/08/using-copy-in-postgres-for-importing-large-csvs https://www.scalingpostgres.com/tutorials/fast-postgresql-data-loading-using-ruby/ https://www.citusdata.com/blog/2018/08/29/datatypes-you-should-consider-using/ https://thebuild.com/blog/2018/08/28/dont-lock-tables-just-dont/ https://blog.2ndquadrant.com/video-data-integration-postgresql/ https://www.youtube.com/watch?v=clrtT_4WBAw https://www.cybertec-postgresql.com/en/what-hot_standby_feedback-in-postgresql-really-does/ https://www.percona.com/blog/2018/08/29/tune-linux-kernel-parame

In this episode of Scaling Postgres, we review articles covering scaling with multiple databases, detecting slow queries, and sorting performance. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://tech.instacart.com/scaling-at-instacart-distributing-data-across-multiple-postgres-databases-with-rails-13b1e4eba202 https://www.cybertec-postgresql.com/en/3-ways-to-detect-slow-queries-in-postgresql/ https://www.cybertec-postgresql.com/en/postgresql-improving-sort-performance/ https://karolgalanciak.com/blog/2018/08/19/indexes-on-rails-how-to-make-the-most-of-your-postgres-database/ https://rob.conery.io/2018/08/13/transactional-data-operations-in-postgresql-using-common-table-expressions/ https://haughtcodeworks.com/blog/software-development/recursive-sql-queries-using-ctes/ https://www.citusdata.com/blog/2018/05/15/fun-with-sql-recursive-ctes/ https://www.percona.com/blog/2018/08/21/foreign-data-wrappers-postgresql-postgres_fdw/ https://www.perc

In this episode of Scaling Postgres, we review articles covering join costs, zHeap, autovacuum tuning and Postgres configuration for humans. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.brianlikespostgres.com/ https://www.youtube.com/watch?v=ZbdWOuTTWrw http://rhaas.blogspot.com/2018/01/do-or-undo-there-is-no-vacuum.html https://www.percona.com/blog/2018/08/10/tuning-autovacuum-in-postgresql-and-autovacuum-internals/ https://www.youtube.com/watch?v=IFIXpm73qtk https://www.slideshare.net/profyclub_ru/postgresql-configuration-for-humans-alvaro-hernandez-ongres https://pgbouncer.github.io/2018/08/pgbouncer-1-9-0 https://severalnines.com/blog/performance-monitoring-auditing-postgresql-top-resources  

In this episode of Scaling Postgres, we review articles covering graphing with generate_series, partitioning in Postgres 11 and GPUs for queries. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://rob.conery.io/2018/08/01/simple-monthly-reports-in-postgresql-using-generate_series/ https://blog.2ndquadrant.com/slides-partitioning-pg-11/ https://wiki.postgresql.org/images/8/82/Alvherre-partitioning-2018.pdf https://blog.openshift.com/gpu-accelerated-sql-queries-with-postgresql-pg-strom-in-openshift-3-10/ https://www.dataengineeringpodcast.com/postgresql-with-jonathan-katz-episode-42/ https://www.citusdata.com/blog/2018/08/09/fun-with-sql-common-table-expressions/ https://hunleyd.github.io/posts/pgBouncer-and-auth-pass-thru/ https://thebuild.com/blog/2018/08/07/does-anyone-really-know-what-time-it-is/ https://thebuild.com/blog/2018/08/09/three-steps-to-pg_rewind-happiness/ https://severalnines.com/blog/using-kubernetes-deploy-postgresql  

In this episode of Scaling Postgres, we review articles covering full text search, aggregating JSON with jsonb_agg, upgrades and CIS benchmarks. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://rob.conery.io/2018/07/23/setting-up-a-fast-comprehensive-search-routine-with-postgresql/   https://medium.com/@geekuillaume/sql-tip-jsonb-agg-in-postgresql-for-simple-one-to-many-joins-bde8caa30c46   https://www.youtube.com/watch?v=lL6g9PHHin4   https://medium.com/preply-engineering/postgres-multimaster-34f2446d5e14   https://info.crunchydata.com/blog/using-the-cis-postgresql-benchmark-to-enhance-your-security   https://medium.com/@cazzer/designing-the-most-performant-row-level-security-strategy-in-postgres-a06084f31945   https://severalnines.com/blog/new-webinar-introduction-performance-monitoring-postgresql   https://severalnines.com/blog/postgresql-triggers-and-stored-function-basics   https://severalnines.com/blog/understanding-a

In this episode of Scaling Postgres, we review articles covering UUIDs, the source of write spikes, uses of distinct and SQL standards support. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: http://richardstartin.uk/uuids-and-compressibility/   https://medium.com/cermati-tech/troubleshooting-daily-write-spikes-on-cermatis-postgresql-rds-9016d7a0c2e4   https://www.citusdata.com/blog/2018/07/25/high-performance-distributed-dml/   https://medium.com/statuscode/the-many-faces-of-distinct-in-postgresql-c52490de5954   https://modern-sql.com/slides/PG-stays-ahead-20180718.pdf   https://tapoueh.org/blog/2018/07/batch-updates-and-concurrency/   https://info.crunchydata.com/blog/wheres-my-replica-troubleshooting-streaming-replication-synchronization-in-postgresql

In this episode of Scaling Postgres, we review articles covering concurrency, how to use listen & notify, zfs, caution regarding pg_rewind. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://tapoueh.org/blog/2018/07/postgresql-event-based-processing/   https://tapoueh.org/blog/2018/07/postgresql-listen/notify/   https://aws.amazon.com/blogs/aws/amazon-ec2-instance-update-faster-processors-and-more-memory/   https://www.citusdata.com/blog/2018/07/19/ZFS-beta-on-citus-cloud/   http://thebuild.com/blog/2018/07/18/pg_rewind-and-checkpoints-caution/   http://thebuild.com/blog/2018/07/17/that-google-checksum-tool/   https://blog.timescale.com/sql-nosql-data-storage-for-prometheus-devops-monitoring-postgresql-timescaledb-time-series-3cde27fd1e07

In this episode of Scaling Postgres, we review articles covering how to use JSON & JSONB, procedures, deal with concurrency issues and Kubernetes. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://citizen428.net/asynchronous-notifications-in-postgres-d890d932582d   http://www.pateldenish.com/2018/07/audit-logging-using-jsonb-in-postgres.html   https://blog.2ndquadrant.com/video-introduction-json-data-types-postgresql/   http://www.postgresonline.com/journal/index.php?/archives/390-Using-procedures-for-batch-geocoding-and-other-batch-processing.html   https://www.cybertec-postgresql.com/en/a-beginners-guide-to-postgresqls-update-and-autovacuum/   https://tapoueh.org/blog/2018/07/modeling-for-concurrency/   https://tapoueh.org/blog/2018/07/computing-and-caching/   http://www.databasesoup.com/2018/07/should-i-run-postgres-on-kubernetes.html   http://www.databasesoup.com/2018/07/should-i-run-postgres-on-kubernetes_12.html  

In this episode of Scaling Postgres, we review articles covering declarative partitioning, handling permissions to view data, and the speed of triggers. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://docs.google.com/presentation/d/136DZZJeVYklEmjl3QoX_-C6NPyMlqmUDuljpO8G8bys https://newbiedba.wordpress.com/2018/07/02/postgresql-10-how-to-create-a-partitioned-table-based-on-jsonb-column-field-value/ https://blog.2ndquadrant.com/scaling-iot-time-series-data-postgres-bdr/ https://www.citusdata.com/blog/2018/07/03/masking-columns-in-postgresql/ https://www.cybertec-postgresql.com/en/rules-or-triggers-to-log-bulk-updates/ https://blog.jooq.org/2018/07/05/postgresql-11s-support-for-sql-standard-groups-and-exclude-window-function-clauses/ https://rjuju.github.io/postgresql/2018/07/03/diagnostic-of-unexpected-slowdown.html https://tapoueh.org/blog/2018/07/postgresql-concurrency-isolation-and-locking/  

In this episode of Scaling Postgres, we review articles covering 100,000 tenants, Postgres Extensions, query planning and how to create crosstabs. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.citusdata.com/blog/2018/06/28/scaling-from-one-to-one-hundred-thousand-tenants/ https://www.youtube.com/watch?v=Vk2A5SuHG20 http://www.se-radio.net/2018/06/se-radio-episode-328-bruce-momjian-on-the-postgres-query-planner/ http://ashutoshpg.blogspot.com/2018/06/planning-queries-involving-foreign.html https://postgresql.verite.pro/blog/2018/06/19/crosstab-pivot.html https://medium.com/futuretech-industries/postgres-raffle-10k-connections-35-mo-part-two-b4c2e0c86e37 https://severalnines.com/blog/architecture-and-tuning-memory-postgresql-databases http://ashutoshpg.blogspot.com/2018/06/upgrade-your-partitioning-from.html https://fluca1978.github.io/2018/06/28/PostgreSQLExtendedStatistics.html https://tapoueh.org/blog/2018/06/postgresql-concurrency

In this episode of Scaling Postgres, we review articles covering the impact of virtualization on databases, techniques to compress your data and parallelism. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.jetbrains.com/research/devecosystem-2018/databases/ https://www.scylladb.com/2018/06/21/impact-virtualization-database/ https://www.modio.se/scaling-past-the-single-machine.html https://www.citusdata.com/blog/2018/06/21/fun-with-sql-functions/ http://rhaas.blogspot.com/2018/06/using-forceparallelmode-correctly.html https://www.cybertec-postgresql.com/en/postgresql-clusters-monitoring-performance/ https://hackernoon.com/elephant-in-the-room-database-backup-574da50e6d88 https://medium.com/futuretech-industries/ten-thousand-high-availability-postgresql-connections-for-35-mo-part-one-4b7a2d61c51e https://severalnines.com/blog/performance-cheat-sheet-postgresql https://severalnines.com/blog/tuning-io-operations-postgresql

In this episode of Scaling Postgres, we review articles covering using Postgres for real-time analytics, how indexes can decrease performance and how to configure work_mem. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://www.citusdata.com/blog/2018/06/14/scalable-incremental-data-aggregation/ https://www.cybertec-postgresql.com/en/index-decreases-select-performance/ https://medium.com/jobteaser-dev-team/materialized-views-with-postgresql-for-beginners-9809483db35f https://www.citusdata.com/blog/2018/06/12/configuring-work-mem-on-postgres/ https://blog.2ndquadrant.com/window-functions-time-series-iot-analytics-postgres-bdr/ https://www.stitchdata.com/blog/how-redshift-differs-from-postgresql/ https://severalnines.com/blog/deploying-postgresql-docker-container

In this episode of Scaling Postgres, we review articles covering efficiently storing bits, autovacuum, setting up assertions and logical replication recovery. Subscribe at https://www.scalingpostgres.com to get notified of new episodes. Links for this episode: https://blog.opendigerati.com/using-bits-for-efficient-storage-in-postgres-4f670f125269 http://blog.dataegret.com/2018/06/autovacuum-slides-from-pgcon-2018-ottawa.html https://www.youtube.com/watch?v=17XecHy9Pzg http://databasedoings.blogspot.com/2018/06/ive-posted-my-slides-for-my-asssertions.html   https://medium.com/avitotech/recovery-use-cases-for-logical-replication-in-postgresql-10-a1e6bab03072   http://momjian.us/main/writings/pgsql/forever.pdf   https://blog.timescale.com/time-series-data-cassandra-vs-timescaledb-postgresql-7c2cc50a89ce   https://severalnines.com/blog/multi-datacenter-setups-postgresql   https://seiler.us/2018-06-05-alter-default-privs/