OnCallReady

Lesson 25.7 · PostgreSQL Operations · 27 min read

Roles, privileges and pg_hba.conf

In plain words

Getting into the database is like getting into an office building. First the doors must be open on your side of the building (listen_addresses). Then the guard at the desk checks a list from top to bottom and uses the first line that fits you: where you come from, who you are and which floor you want (pg_hba.conf). Then you show your badge (a password). Finally, inside, each room has its own key list (privileges).

When an app "cannot connect", you find out which of those four checks said no.

Why this lesson exists

"The app can't connect to the database" is the most common Postgres page of all, and it is almost never the database being down. It is a new subnet that is not in pg_hba.conf, a rotated password, a role without LOGIN, a missing GRANT after a migration created a new table, or the server listening only on localhost. Every one of those has a precise error message - this lesson teaches you to read it and fix the right file. It also builds the roles a real service should have, because "the app connects as postgres" is how one SQL injection becomes the loss of a whole cluster.

What you need to know already: peer auth and pg_hba.conf's location, reload vs restart (lesson 2), psql connection options (lesson 3), CIDR notation (the networking chapter).

The words you need first

Roles

$ sudo -u postgres psql -c '\du'
                               List of roles
  Role name   |                         Attributes
--------------+------------------------------------------------------------
 orders_app   |
 orders_owner | Cannot login
 orders_ro    |
 postgres     | Superuser, Create role, Create DB, Replication, Bypass RLS

A fresh cluster has one role you see (postgres, superuser) plus built-in pg_* roles that \du hides (\duS shows them). The pattern for an application is three roles:

roleattributesowns / gets
orders_ownerNOLOGINowns the schema and every table; migrations run as it (SET ROLE)
orders_appLOGIN, passwordSELECT, INSERT, UPDATE, DELETE on the tables, nothing else
orders_roLOGIN, passwordSELECT only: reporting, support, read replicas

The app can then never DROP TABLE, TRUNCATE or ALTER anything, and a leaked read-only password cannot change data. Nobody logs in as the owner: it is a role to own things, not a user.

The lab already has them (ordersDb in the setup made them); look:

$ sudo -u postgres psql -c '\du orders*'
        List of roles
  Role name   |  Attributes
--------------+--------------
 orders_app   |
 orders_owner | Cannot login
 orders_ro    |
$ sudo -u postgres psql -d orders -c '\dp customers'
                                       Access privileges
 Schema |   Name    | Type  |         Access privileges          | Column privileges | Policies
--------+-----------+-------+------------------------------------+-------------------+----------
 public | customers | table | orders_owner=arwdDxtm/orders_owner+|                   |
        |           |       | orders_app=arwd/orders_owner      +|                   |
        |           |       | orders_ro=r/orders_owner           |                   |
(1 row)

The access privileges column is an ACL: grantee=privileges/grantor. Letters: a INSERT (append), r SELECT (read), w UPDATE (write), d DELETE, D TRUNCATE, x REFERENCES, t TRIGGER, m MAINTAIN (new in 17: VACUUM, ANALYZE, REINDEX...). arwdDxtm = everything = the owner.

How they were made:

CREATE ROLE orders_owner NOLOGIN;
CREATE ROLE orders_app LOGIN PASSWORD '...';
CREATE ROLE orders_ro  LOGIN PASSWORD '...';
CREATE DATABASE orders OWNER orders_owner;
REVOKE CONNECT ON DATABASE orders FROM PUBLIC;
GRANT CONNECT ON DATABASE orders TO orders_app, orders_ro;
-- in the orders database:
GRANT USAGE ON SCHEMA public TO orders_app, orders_ro;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO orders_app;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO orders_ro;
ALTER DEFAULT PRIVILEGES FOR ROLE orders_owner IN SCHEMA public
  GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO orders_app;
ALTER DEFAULT PRIVILEGES FOR ROLE orders_owner IN SCHEMA public
  GRANT SELECT ON TABLES TO orders_ro;

Three traps in there:

  1. PUBLIC is every role. By default every role may CONNECT to every database (and, before 15, create tables in public). Revoke what you do not want.
  2. GRANT ... ON ALL TABLES is a snapshot: tables created tomorrow get nothing. The classic incident: a migration adds a table, the app gets ERROR: permission denied for table refunds. ALTER DEFAULT PRIVILEGES FOR ROLE <the creator> fixes it for the future - and it only applies to objects that role creates, which is why migrations must run as the owner.
  3. Sequences and functions are separate objects. An identity column or serial uses a sequence; with GENERATED ... AS IDENTITY (as here) privileges on the table are enough, with old serial columns the app also needs USAGE on the sequence.

Logging in over TCP

The app connects over TCP with a password, so that is what to test - not the socket:

$ PGPASSWORD=ro-lab-pw psql -h 127.0.0.1 -U orders_ro -d orders -c 'select count(*) from customers'
 count
-------
  4000
(1 row)
$ PGPASSWORD=ro-lab-pw psql -h 127.0.0.1 -U orders_ro -d orders -c "delete from customers where id = 1"
ERROR:  permission denied for table customers
$ PGPASSWORD=wrong psql -h 127.0.0.1 -U orders_ro -d orders -c 'select 1'
psql: error: connection to server at "127.0.0.1", port 5432 failed: FATAL:  password authentication failed for user "orders_ro"

The client only ever says password authentication failed. The server log says why, and which pg_hba.conf line matched - that is where you look:

$ sudo tail -n 3 /var/log/postgresql/postgresql-18-main.log
2026-09-22 20:01:52.250 UTC [17418] orders_ro@orders STATEMENT:  delete from customers where id = 1
2026-09-22 20:01:52.500 UTC [17419] orders_ro@orders FATAL:  password authentication failed for user "orders_ro"
2026-09-22 20:01:52.500 UTC [17419] orders_ro@orders DETAIL:  Connection matched file "/etc/postgresql/18/main/pg_hba.conf" line 125: "host    all             all             127.0.0.1/32            scram-sha-256"

A plain wrong password over SCRAM logs only the rule that matched (line 125 here). When something else is wrong, the DETAIL: names it first: Role "x" does not exist., User "x" has no password assigned., User "x" has an expired password. (VALID UNTIL passed), User "x" does not have a valid SCRAM secret. (an old md5 password with a scram-sha-256 rule), or Password does not match for user "x". (md5/password rules). The client deliberately gets none of it, so an attacker cannot probe which role names exist.

Change a password the safe way - from psql, so it never appears in a shell history or the server log (\password sends the already-hashed verifier):

orders=# \password orders_ro
Enter new password for user "orders_ro":
Enter it again:

pg_hba.conf

$ sudo grep -v '^\s*#' /etc/postgresql/18/main/pg_hba.conf | grep -v '^$'
local   all             postgres                                peer
local   all             all                                     peer
host    all             all             127.0.0.1/32            scram-sha-256
host    all             all             ::1/128                 scram-sha-256
local   replication     all                                     peer
host    replication     all             127.0.0.1/32            scram-sha-256
host    replication     all             ::1/128                 scram-sha-256

Each line: type, database, user, address (not for local), method, [options]. The server reads them top to bottom and the first match wins - a later, better rule is never reached.

fieldvalues
typelocal (Unix socket), host (TCP, SSL or not), hostssl (TCP with SSL only), hostnossl
databaseall, a name, sameuser, replication (only for replication connections), @file, a comma list, /regex
userall, a name, +group (members of a role), a comma list, /regex
address10.244.0.0/16, 10.64.0.20/32, ::1/128, samenet, a host name
methodscram-sha-256, md5 (deprecated), peer (local only), trust (no check at all - never on TCP), reject, cert, ldap, oauth (new in 18)

Changes to pg_hba.conf need a reload, never a restart, and a broken file is refused as a whole - the server keeps the old rules and logs the error. Check what the server will actually use before you reload:

$ sudo -u postgres psql -c "select line_number, type, database, user_name, address, netmask, auth_method, error from pg_hba_file_rules"
 line_number | type  |   database    | user_name  |  address  |                 netmask                 |  auth_method  | error
-------------+-------+---------------+------------+-----------+-----------------------------------------+---------------+-------
         118 | local | {all}         | {postgres} |           |                                         | peer          |
         123 | local | {all}         | {all}      |           |                                         | peer          |
         125 | host  | {all}         | {all}      | 127.0.0.1 | 255.255.255.255                         | scram-sha-256 |
         127 | host  | {all}         | {all}      | ::1       | ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff | scram-sha-256 |
         130 | local | {replication} | {all}      |           |                                         | peer          |
         131 | host  | {replication} | {all}      | 127.0.0.1 | 255.255.255.255                         | scram-sha-256 |
         132 | host  | {replication} | {all}      | ::1       | ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff | scram-sha-256 |
(7 rows)

pg_hba_file_rules parses the file on disk now, so a typo shows up in the error column before your reload throws it away.

The errors you will see, one per cause:

client errormeansfix
connection refused / Is the server running on that host and accepting TCP/IP connections?nothing listens on that address: down, wrong port, or listen_addresses = 'localhost'listen_addresses (restart), firewall
no pg_hba.conf entry for host "10.244.1.17", user "orders_app", database "orders", no encryptionno rule matched at alladd a rule, reload
pg_hba.conf rejects connection for host ...the first match was rejectreorder
password authentication failed for user "x"a password rule matched, the check failedthe server log's DETAIL
role "x" does not exist / database "x" does not existwhat it saystypo or not created
permission denied for database "orders" + User does not have CONNECT privilege.in through hba, but no CONNECTGRANT CONNECT
role "x" is not permitted to log inthe role is NOLOGINlog in as a real user
sorry, too many clients alreadymax_connections reachedlesson 5

Opening the server to the network

Two separate gates, and both must be open:

  1. listen_addresses (default localhost): which interfaces the server binds. A postmaster setting - restart. '*' = all interfaces; better, the specific address.
  2. pg_hba.conf: which clients may authenticate on them - reload.
$ sudo ss -ltn 'sport = :5432'
State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0      200            [::1]:5432         [::]:*
LISTEN 0      200        127.0.0.1:5432      0.0.0.0:*

Only 127.0.0.1 and ::1: pods on 10.244.x.x get connection refused long before pg_hba.conf is consulted. With listen_addresses = '*' and a restart you would see 0.0.0.0:5432 and [::]:5432.

A good rule for an app on a Kubernetes pod network, in its own place in the file (before any broader rule):

# TYPE  DATABASE  USER        ADDRESS         METHOD
host    orders    orders_app  10.244.0.0/16   scram-sha-256

Narrow on all four: one database, one role, the pod CIDR, a real password. Never host all all 0.0.0.0/0 trust. Use hostssl when the traffic crosses a network you do not control (and sslmode=verify-full on the client side: require only encrypts, it does not check who answered).

In an interview: "A new app on Kubernetes can't connect to Postgres - how do you debug it?" - read the exact error: connection refused means nothing listens there (listen_addresses, needs a restart, or a firewall); no pg_hba.conf entry for host ... means add a narrow host <db> <role> <pod CIDR> scram-sha-256 rule and reload; password authentication failed means read the server log's DETAIL (wrong password, no role, expired). Check pg_hba_file_rules before reloading.

What you can do now

Why it helps

"The app can't connect to the database" is the most common PostgreSQL page, and it is almost never the database being down. Each cause - a closed listener, a missing or misordered pg_hba.conf rule, a wrong or expired password, a missing privilege - has a precise error message, and the server log's DETAIL tells you which line matched.

The same lesson builds the role design every service should have: an owner nobody logs in as, an app role that cannot change the schema, a read-only role, and default privileges so new tables do not break the app.

Commands in this lesson

psql tail grep ss

FAQ

Why does the client only say "password authentication failed"?

On purpose: if the client were told whether the role exists, an attacker could probe for valid user names. The server log says why - Role does not exist, User has an expired password, no valid SCRAM secret, or just the pg_hba.conf line that matched for a plain wrong password. Always read the server side.

Why did my new pg_hba.conf rule not work?

Usually one of three things: the server was not reloaded, an earlier line matched first (the first match wins, including reject lines), or one field does not match - a typo in the database or role name, the wrong CIDR, or "all" in the database column for a replication connection. pg_hba_file_rules shows how the server parses the file on disk.

Why does the app get "permission denied for table" after a migration?

GRANT ... ON ALL TABLES only covers tables that exist at that moment. A new table created by a migration has no grants for the app role. ALTER DEFAULT PRIVILEGES FOR ROLE <owner> IN SCHEMA public GRANT ... ON TABLES TO <app> covers every table that role creates later, which is why migrations should run as the owner role.

Do I need a restart to open the server to the network?

For listen_addresses, yes: it is a postmaster setting, and until it changes remote clients get "connection refused" before pg_hba.conf is even consulted. pg_hba.conf itself only needs a reload. Plan the restart, then the reload, and test from the client's network.

Why SCRAM and not md5?

SCRAM-SHA-256 stores a salted verifier and never sends the password or a replayable hash over the network. md5 is weaker, and PostgreSQL 18 deprecates it. A role with an old md5 password cannot log in through a scram-sha-256 rule ("does not have a valid SCRAM secret"); set the password again to store a SCRAM verifier.

In an interview Junior

A new app on Kubernetes cannot connect to PostgreSQL. How do you debug it?

Read the exact error, then the server log. connection refused means nothing listens on that address: the server is down, the port is wrong, or listen_addresses is still localhost (a restart setting). no pg_hba.conf entry for host ... means no rule matched: add a narrow host <db> <role> <pod CIDR> scram-sha-256 line above any broader reject rule and reload, checking pg_hba_file_rules first. password authentication failed means read the log's DETAIL (wrong password, missing role, expired VALID UNTIL). permission denied for database means no CONNECT privilege.

Also asked: How would you design database roles for an application? · What does "first match wins" mean in pg_hba.conf? · Why should an application never connect as a superuser?

Practise this lesson in the terminal Free, in your browser - a real Ubuntu terminal to try it in, with missions that check your work.