starred/ntfy

Fork 0

mirror of https://github.com/binwiederhier/ntfy.git synced 2026-05-09 08:26:00 +02:00

[GH-ISSUE #1644] Unable to publish poll request - Forbidden #1147

New issue

Open

opened 2026-05-07 00:30:35 +02:00 by BreizhHardware · 7 comments

BreizhHardware commented

2026-05-07 00:30:35 +02:00

Owner

Originally created by @Maathias on GitHub (Mar 8, 2026).
Original GitHub issue: https://github.com/binwiederhier/ntfy/issues/1644

🐞 Describe the bug
I'm hosting ntfy in docker, on a small vps. It's set up to use upstream-url for iOS notifications. Recently they stopped working, I'm constantly getting Unable to publish poll request

I tried adding a token, but that didn't help. I tried replaying the request, captured via mitmproxy, and here is where I'm getting totally lost.
When using the proxy hosted on a different host with different IP, request goes through. My first thought is - my vps is blocked by ip.
BUT, when i replayed the same request via curl on the vps, it also goes through. Both via ipv6 and ipv4. I tried fuzzing the id/topic, still 200 OK. Same when replaying from inside the container. Seems like only ntfy has this issue, and only then calling ntfy.sh directly
I think I've run out of things to debug, help appreciated.

💻 Components impacted
ntfy server, and iOS app

💡 Screenshots and/or logs
Error from docker logs

ntfy-1  | 2026/03/08 00:49:32 WARN Unable to publish poll request (error=Post "https://ntfy.sh/...": Forbidden, message_body_size=6, message_event=message, message_id=..., message_sender=..., message_sequence_id=..., message_time=1772927372, topic=notused, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:..., visitor_ip=..., visitor_messages=1, visitor_messages_limit=17280, visitor_messages_remaining=17279, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=44.694065798, visitor_seen=2026-03-08T00:49:32.481+01:00)

Same request that casues errors, replayed in shell on the same host

> POST /topic-hash... HTTP/2
> Host: ntfy.sh
> Accept: */*
> Accept-Encoding: deflate, gzip, br, zstd
> User-Agent: ntfy/2.18.0
> Authorization: Bearer tk_...
> X-Poll-Id: ...
> content-length: 0
> 

< HTTP/2 200 
< server: nginx
< date: Sun, 08 Mar 2026 00:10:54 GMT
< content-type: application/json
< content-length: 188
< access-control-allow-origin: *
< 
{"id":"...","time":1772928654,"event":"poll_request","topic":"...","message":"New message","poll_id":"..."}

docker-compose.yml

services:
  ntfy:
    image: binwiederhier/ntfy
    command:
      - serve
    environment:
      - TZ=...
    volumes:
      - ./ntfy-cache:/var/cache/ntfy
      - ./ntfy:/etc/ntfy
    restart: always

  caddy:
    image: caddy:2-alpine
    ports:
      - "[::]:443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - fullpath.../certificates:/certs:ro

server.yml

listen-http: ":80"
upstream-base-url: "https://ntfy.sh"
upstream-access-token: "tk_..."
base-url: "https://ntfy.mydomain.com"
behind-proxy: true
log-level: trace

🔮 Additional context

vps has a public ipv6 as well as ipv4 acces behind nat
issue started without a token, possible after a broken service spammed notifications
token was created on a fresh account

Originally created by @Maathias on GitHub (Mar 8, 2026). Original GitHub issue: https://github.com/binwiederhier/ntfy/issues/1644 :lady_beetle: **Describe the bug** I'm hosting ntfy in docker, on a small vps. It's set up to use upstream-url for iOS notifications. Recently they stopped working, I'm constantly getting `Unable to publish poll request` I tried adding a token, but that didn't help. I tried replaying the request, captured via mitmproxy, and here is where I'm getting totally lost. When using the proxy hosted on a different host with different IP, request goes through. My first thought is - my vps is blocked by ip. BUT, when i replayed the same request via curl on the vps, it also goes through. Both via ipv6 and ipv4. I tried fuzzing the id/topic, still 200 OK. Same when replaying from inside the container. Seems like only ntfy has this issue, and only then calling ntfy.sh directly I think I've run out of things to debug, help appreciated. :computer: **Components impacted** ntfy server, and iOS app :bulb: **Screenshots and/or logs** Error from docker logs ``` ntfy-1 | 2026/03/08 00:49:32 WARN Unable to publish poll request (error=Post "https://ntfy.sh/...": Forbidden, message_body_size=6, message_event=message, message_id=..., message_sender=..., message_sequence_id=..., message_time=1772927372, topic=notused, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:..., visitor_ip=..., visitor_messages=1, visitor_messages_limit=17280, visitor_messages_remaining=17279, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=44.694065798, visitor_seen=2026-03-08T00:49:32.481+01:00) ``` Same request that casues errors, replayed in shell on the same host ``` > POST /topic-hash... HTTP/2 > Host: ntfy.sh > Accept: */* > Accept-Encoding: deflate, gzip, br, zstd > User-Agent: ntfy/2.18.0 > Authorization: Bearer tk_... > X-Poll-Id: ... > content-length: 0 > < HTTP/2 200 < server: nginx < date: Sun, 08 Mar 2026 00:10:54 GMT < content-type: application/json < content-length: 188 < access-control-allow-origin: * < {"id":"...","time":1772928654,"event":"poll_request","topic":"...","message":"New message","poll_id":"..."} ``` docker-compose.yml ``` services: ntfy: image: binwiederhier/ntfy command: - serve environment: - TZ=... volumes: - ./ntfy-cache:/var/cache/ntfy - ./ntfy:/etc/ntfy restart: always caddy: image: caddy:2-alpine ports: - "[::]:443:443" volumes: - ./Caddyfile:/etc/caddy/Caddyfile - fullpath.../certificates:/certs:ro ``` server.yml ``` listen-http: ":80" upstream-base-url: "https://ntfy.sh" upstream-access-token: "tk_..." base-url: "https://ntfy.mydomain.com" behind-proxy: true log-level: trace ``` :crystal_ball: **Additional context** - vps has a public ipv6 as well as ipv4 acces behind nat - issue started without a token, possible after a broken service spammed notifications - token was created on a fresh account

BreizhHardware added the

🪲 bug

unified-push

labels

2026-05-07 00:30:35 +02:00

BreizhHardware commented

2026-05-07 00:30:36 +02:00

Author

Owner

@binwiederhier commented on GitHub (Mar 8, 2026):

This is most curious. If you provide the topic hash (or a part of it) to me, I can look on the ntfy.sh logs.

@binwiederhier commented on GitHub (Mar 8, 2026): This is most curious. If you provide the topic hash (or a part of it) to me, I can look on the ntfy.sh logs.

BreizhHardware commented

2026-05-07 00:30:36 +02:00

Author

Owner

@Maathias commented on GitHub (Mar 8, 2026):

Here is the last half of it dcc0a90e0454dea288f7a538edf6cff2

@Maathias commented on GitHub (Mar 8, 2026): Here is the last half of it `dcc0a90e0454dea288f7a538edf6cff2`

BreizhHardware commented

2026-05-07 00:30:36 +02:00

Author

Owner

@binwiederhier commented on GitHub (Mar 8, 2026):

I looked at the logs and I see once recent instance of a 401. How often does it happen? You are certain the token is correct?

It is more curious because the success and 401-failure are only seconds apart.

access.log.1:2a01:4f9:...          - - [07/Mar/2026:23:32:57 +0000] "POST /...88f7a538edf6cff2 HTTP/2.0" 200 184 "-" "ntfy/2.x" http_host=ntfy.sh request_time=0.002 upstream_status=200 upstream_response_time=0.000 upstream_connect_time=0.000 upstream_header_time=0.000
access.log.1:65.108...             - - [07/Mar/2026:23:33:14 +0000] "POST /...88f7a538edf6cff2 HTTP/2.0" 200 184 "-" "ntfy/2.x" http_host=ntfy.sh request_time=0.002 upstream_status=200 upstream_response_time=0.000 upstream_connect_time=0.000 upstream_header_time=0.000
access.log.1:65.108...             - - [07/Mar/2026:23:34:19 +0000] "POST /...88f7a538edf6cff2 HTTP/1.1" 200 184 "-" "Wget" http_host=ntfy.sh request_time=0.107 upstream_status=200 upstream_response_time=0.000 upstream_connect_time=0.000 upstream_header_time=0.000
access.log.1:188.121...            - - [07/Mar/2026:23:51:29 +0000] "POST /...88f7a538edf6cff2 HTTP/1.1" 200 187 "-" "ntfy/2.18.0" http_host=ntfy.sh request_time=0.002 upstream_status=200 upstream_response_time=0.000 upstream_connect_time=0.000 upstream_header_time=0.000
access.log.1:2a01:4f9:...:76       - - [07/Mar/2026:23:51:56 +0000] "POST /...88f7a538edf6cff2 HTTP/2.0" 200 187 "-" "ntfy/2.18.0" http_host=ntfy.sh request_time=0.002 upstream_status=200 upstream_response_time=0.000 upstream_connect_time=0.000 upstream_header_time=0.000
access.log.1:2a01:4f9:...:77       - - [07/Mar/2026:23:52:12 +0000] "POST /...88f7a538edf6cff2 HTTP/2.0" 200 187 "-" "ntfy/2.18.0" http_host=ntfy.sh request_time=0.002 upstream_status=200 upstream_response_time=0.000 upstream_connect_time=0.000 upstream_header_time=0.000
access_error.log.1:2a01:4f9:...:77 - - [07/Mar/2026:23:52:46 +0000] "POST /...88f7a538edf6cff2 HTTP/2.0" 401 103 "-" "ntfy/2.18.0" http_host=ntfy.sh request_time=0.001 upstream_status=401 upstream_response_time=0.000 upstream_connect_time=0.000 upstream_header_time=0.000
access.log.1:2a01:4f9:...:77       - - [07/Mar/2026:23:52:57 +0000] "POST /...88f7a538edf6cff2 HTTP/2.0" 200 188 "-" "ntfy/2.18.0" http_host=ntfy.sh request_time=0.002 upstream_status=200 upstream_response_time=0.000 upstream_connect_time=0.000 upstream_header_time=0.000

I cannot think of a legitimate reason why this would be the case.

Claude suggests a sqlite blip:

● Looking at the timeline, the 401 at 23:52:46 is from the same IP (2a01:4f9:...:77) and same user agent (ntfy/2.18.0) that got 200s at 23:52:12 and 23:52:57 — just seconds    
  apart.                                                                                                                                                                                
                                                                                                                                                                                        
  Since a 401 means authentication failed (not authorization, which would be 403), the request included an auth header (Basic or Bearer) that was rejected. Possible causes:            
                                                                                                                                                                                        
  1. Transient SQLite issue — A brief database lock/timeout during token lookup in userManager.AuthenticateToken() or userManager.Authenticate() would return ErrUnauthenticated → 401. 
  This is the most likely cause given it's a one-off between successful requests.                                                                                                       
  2. Token rotation race — If the client was refreshing/rotating its bearer token, one request could have hit with an expired or not-yet-committed token.                               
  3. Visitor race condition — There's a potential race in maybeAuthenticate() where concurrent requests from the same IP can interfere with each other's visitor state. At 23:52:46, the
   request from :77 overlaps closely with the other IPs hitting the same topic, and visitor objects are shared by IP.                                                                   
  4. Upstream forwarding — If this IP is an upstream ntfy server forwarding messages, the forwarded auth credentials could have been stale for one request.                             
                                                                                                                                                                                        
  The most likely explanation is (1) — a brief SQLite hiccup causing the token/password lookup to fail transiently. The response body size (103 bytes) matches the standard             
  errHTTPUnauthorized JSON response, and the request_time=0.001 is notably faster than the successful requests (0.002), consistent with an early database error bail-out rather than    
  completing the full bcrypt comparison.           ```

How often did you say this happens? Does it still happen?

@binwiederhier commented on GitHub (Mar 8, 2026): I looked at the logs and I see once recent instance of a 401. How often does it happen? You are certain the token is correct? It is more curious because the success and 401-failure are only seconds apart. ``` access.log.1:2a01:4f9:... - - [07/Mar/2026:23:32:57 +0000] "POST /...88f7a538edf6cff2 HTTP/2.0" 200 184 "-" "ntfy/2.x" http_host=ntfy.sh request_time=0.002 upstream_status=200 upstream_response_time=0.000 upstream_connect_time=0.000 upstream_header_time=0.000 access.log.1:65.108... - - [07/Mar/2026:23:33:14 +0000] "POST /...88f7a538edf6cff2 HTTP/2.0" 200 184 "-" "ntfy/2.x" http_host=ntfy.sh request_time=0.002 upstream_status=200 upstream_response_time=0.000 upstream_connect_time=0.000 upstream_header_time=0.000 access.log.1:65.108... - - [07/Mar/2026:23:34:19 +0000] "POST /...88f7a538edf6cff2 HTTP/1.1" 200 184 "-" "Wget" http_host=ntfy.sh request_time=0.107 upstream_status=200 upstream_response_time=0.000 upstream_connect_time=0.000 upstream_header_time=0.000 access.log.1:188.121... - - [07/Mar/2026:23:51:29 +0000] "POST /...88f7a538edf6cff2 HTTP/1.1" 200 187 "-" "ntfy/2.18.0" http_host=ntfy.sh request_time=0.002 upstream_status=200 upstream_response_time=0.000 upstream_connect_time=0.000 upstream_header_time=0.000 access.log.1:2a01:4f9:...:76 - - [07/Mar/2026:23:51:56 +0000] "POST /...88f7a538edf6cff2 HTTP/2.0" 200 187 "-" "ntfy/2.18.0" http_host=ntfy.sh request_time=0.002 upstream_status=200 upstream_response_time=0.000 upstream_connect_time=0.000 upstream_header_time=0.000 access.log.1:2a01:4f9:...:77 - - [07/Mar/2026:23:52:12 +0000] "POST /...88f7a538edf6cff2 HTTP/2.0" 200 187 "-" "ntfy/2.18.0" http_host=ntfy.sh request_time=0.002 upstream_status=200 upstream_response_time=0.000 upstream_connect_time=0.000 upstream_header_time=0.000 access_error.log.1:2a01:4f9:...:77 - - [07/Mar/2026:23:52:46 +0000] "POST /...88f7a538edf6cff2 HTTP/2.0" 401 103 "-" "ntfy/2.18.0" http_host=ntfy.sh request_time=0.001 upstream_status=401 upstream_response_time=0.000 upstream_connect_time=0.000 upstream_header_time=0.000 access.log.1:2a01:4f9:...:77 - - [07/Mar/2026:23:52:57 +0000] "POST /...88f7a538edf6cff2 HTTP/2.0" 200 188 "-" "ntfy/2.18.0" http_host=ntfy.sh request_time=0.002 upstream_status=200 upstream_response_time=0.000 upstream_connect_time=0.000 upstream_header_time=0.000 ``` I cannot think of a legitimate reason why this would be the case. Claude suggests a sqlite blip: ``` ● Looking at the timeline, the 401 at 23:52:46 is from the same IP (2a01:4f9:...:77) and same user agent (ntfy/2.18.0) that got 200s at 23:52:12 and 23:52:57 — just seconds apart. Since a 401 means authentication failed (not authorization, which would be 403), the request included an auth header (Basic or Bearer) that was rejected. Possible causes: 1. Transient SQLite issue — A brief database lock/timeout during token lookup in userManager.AuthenticateToken() or userManager.Authenticate() would return ErrUnauthenticated → 401. This is the most likely cause given it's a one-off between successful requests. 2. Token rotation race — If the client was refreshing/rotating its bearer token, one request could have hit with an expired or not-yet-committed token. 3. Visitor race condition — There's a potential race in maybeAuthenticate() where concurrent requests from the same IP can interfere with each other's visitor state. At 23:52:46, the request from :77 overlaps closely with the other IPs hitting the same topic, and visitor objects are shared by IP. 4. Upstream forwarding — If this IP is an upstream ntfy server forwarding messages, the forwarded auth credentials could have been stale for one request. The most likely explanation is (1) — a brief SQLite hiccup causing the token/password lookup to fail transiently. The response body size (103 bytes) matches the standard errHTTPUnauthorized JSON response, and the request_time=0.001 is notably faster than the successful requests (0.002), consistent with an early database error bail-out rather than completing the full bcrypt comparison. ``` How often did you say this happens? Does it still happen?

BreizhHardware commented

2026-05-07 00:30:36 +02:00

Author

Owner

@Maathias commented on GitHub (Mar 8, 2026):

It still happens now, for every request. The token is correct, but it seems to not be used in the container, Published Messages on the account page is not changing. Config is still the same as above. Here are two tries I did just now

ntfy-1  | 2026/03/08 16:41:41 WARN Unable to publish poll request (error=Post "https://ntfy.sh/...64b901073b780483efffaadf7b6c46a4": Forbidden, message_body_size=15, message_event=message, message_id=AuswgA4BwhGL, message_sender=..., message_sequence_id=AuswgA4BwhGL, message_time=1772984500, topic=hosts, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:..., visitor_ip=..., visitor_messages=2, visitor_messages_limit=17280, visitor_messages_remaining=17278, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.05560676, visitor_seen=2026-03-08T16:41:40.878+01:00)

ntfy-1  | 2026/03/08 16:38:19 WARN Unable to publish poll request (error=Post "https://ntfy.sh/...367be108536b4ceedfe0118e60681c94": Forbidden, message_body_size=18, message_event=message, message_id=UeA7D83JWmOL, message_sender=..., message_sequence_id=UeA7D83JWmOL, message_time=1772984298, topic=arr, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:..., visitor_ip=..., visitor_messages=1, visitor_messages_limit=17280, visitor_messages_remaining=17279, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.0897120872, visitor_seen=2026-03-08T16:38:18.991+01:00)

@Maathias commented on GitHub (Mar 8, 2026): It still happens now, for every request. The token is correct, but it seems to not be used in the container, Published Messages on the account page is not changing. Config is still the same as above. Here are two tries I did just now ``` ntfy-1 | 2026/03/08 16:41:41 WARN Unable to publish poll request (error=Post "https://ntfy.sh/...64b901073b780483efffaadf7b6c46a4": Forbidden, message_body_size=15, message_event=message, message_id=AuswgA4BwhGL, message_sender=..., message_sequence_id=AuswgA4BwhGL, message_time=1772984500, topic=hosts, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:..., visitor_ip=..., visitor_messages=2, visitor_messages_limit=17280, visitor_messages_remaining=17278, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.05560676, visitor_seen=2026-03-08T16:41:40.878+01:00) ``` ``` ntfy-1 | 2026/03/08 16:38:19 WARN Unable to publish poll request (error=Post "https://ntfy.sh/...367be108536b4ceedfe0118e60681c94": Forbidden, message_body_size=18, message_event=message, message_id=UeA7D83JWmOL, message_sender=..., message_sequence_id=UeA7D83JWmOL, message_time=1772984298, topic=arr, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=30, visitor_id=ip:..., visitor_ip=..., visitor_messages=1, visitor_messages_limit=17280, visitor_messages_remaining=17279, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.0897120872, visitor_seen=2026-03-08T16:38:18.991+01:00) ```

BreizhHardware commented

2026-05-07 00:30:37 +02:00

Author

Owner

@binwiederhier commented on GitHub (Mar 8, 2026):

Just to be sure: The upstream-access-token is a token from the ntfy.sh server, right? i.e. you made an account on ntfy.sh with a username and you are using a token that you generated on ntfy.sh/account. Using a token from your selfhosted server would lead to 401s.

The upstream-access-token is meant to allow you to send more than 250 pushes to your phone if you have a paid ntfy.sh account. Otherwise setting it has no impact.

@binwiederhier commented on GitHub (Mar 8, 2026): Just to be sure: The `upstream-access-token` is a token from the `ntfy.sh` server, right? i.e. you made an account on ntfy.sh with a username and you are using a token that you generated on ntfy.sh/account. Using a token from your selfhosted server would lead to 401s. The `upstream-access-token` is meant to allow you to send more than 250 pushes to your phone if you have a paid ntfy.sh account. Otherwise setting it has no impact.

BreizhHardware commented

2026-05-07 00:30:37 +02:00

Author

Owner

@Maathias commented on GitHub (Mar 8, 2026):

Yup, also just tried creating a new one, no luck

@Maathias commented on GitHub (Mar 8, 2026): Yup, also just tried creating a new one, no luck <img width="889" height="427" alt="Image" src="https://github.com/user-attachments/assets/f636f015-709a-421a-b1ab-7eaade00e62c" />

BreizhHardware commented

2026-05-07 00:30:37 +02:00

Author

Owner

@Maathias commented on GitHub (Mar 8, 2026):

This really seams like an issue with the way docker sends the requests, i just did again the manual curl on the same machine, just the topic hash and poll-id, no token. 200 OK, iphone gets a notification

While reading some of the issues here, I saw someone had an issue with the MTU on hetzner hosting (#1003). The vps is the free tier of https://mikr.us/, which I belive is an LXC on mikrus' VM on hetzner. Maybe it's something related to this?

@Maathias commented on GitHub (Mar 8, 2026): This really seams like an issue with the way docker sends the requests, i just did again the manual curl on the same machine, just the topic hash and poll-id, no token. 200 OK, iphone gets a notification While reading some of the issues here, I saw someone had an issue with the MTU on hetzner hosting (#1003). The vps is the free tier of https://mikr.us/, which I belive is an LXC on mikrus' VM on hetzner. Maybe it's something related to this?