[PR #1656] [MERGED] S3 attachment storage #1680

New issue

Closed

opened 2026-05-07 01:03:13 +02:00 by BreizhHardware · 0 comments

BreizhHardware commented

2026-05-07 01:03:13 +02:00

Owner

📋 Pull Request Information

Original PR: https://github.com/binwiederhier/ntfy/pull/1656
Author: @binwiederhier
Created: 3/15/2026
Status: ✅ Merged
Merged: 3/23/2026
Merged by: @binwiederhier

Base: main ← Head: attachment-s3

📝 Commits (10+)

d517ce4 WIP: S3
b4ec6fa AWS deps..
4487299 Merge branch 'main' into attachment-s3
790ba24 S3 WIP
458fbad Merge branch 'main' into attachment-s3
86015e1 Multipart upload
6b11bc7 Merge branch 'main' into attachment-s3
a47d692 Fix bug
cffa579 Logs
ef31496 Refactor

📊 Changes

38 files changed (+3089 additions, -502 deletions)

View changed files

📝 .github/workflows/build.yaml (+4 -1)
📝 .github/workflows/release.yaml (+1 -0)
📝 .github/workflows/test.yaml (+5 -1)
📝 .gitignore (+1 -0)
➕ attachment/backend.go (+23 -0)
➕ attachment/backend_file.go (+94 -0)
➕ attachment/backend_s3.go (+51 -0)
➕ attachment/store.go (+232 -0)
➕ attachment/store_file_test.go (+16 -0)
➕ attachment/store_s3_test.go (+120 -0)
➕ attachment/store_test.go (+352 -0)
📝 cmd/serve.go (+1 -1)
📝 docs/config.md (+159 -86)
📝 docs/releases.md (+4 -0)
📝 go.mod (+1 -1)
📝 go.sum (+2 -2)
📝 message/cache.go (+31 -45)
📝 message/cache_postgres.go (+3 -0)
📝 message/cache_sqlite.go (+3 -0)
➕ s3/client.go (+302 -0)

...and 18 more files

📄 Description

Summary

Adds S3-compatible object storage as a backend for attachment caching, as an alternative to local filesystem storage.

Minimal S3 client (s3/ package): implements PutObject (with automatic multipart upload for large files), GetObject, DeleteObjects, and ListObjectsV2 using AWS Signature V4 signing — no AWS SDK dependency
Attachment store (attachment/ package): unified Store with pluggable backend interface (fileBackend / s3Backend), shared size tracking, rate limiting, and background sync
Background cleanup: every 15 minutes, reconciles the S3 bucket (or prefix) with the message database — deletes orphaned objects whose file IDs are no longer referenced, and aborts incomplete multipart uploads older than 1 hour
Configuration: attachment-cache-dir accepts an S3 URL (s3://ACCESS_KEY:SECRET_KEY@BUCKET[/PREFIX]?region=REGION[&endpoint=ENDPOINT]), auto-detecting path-style addressing when a custom endpoint is set
Documentation: updated docs/config.md with S3 configuration examples and cleanup behavior

S3 client features

AWS Signature V4 request signing (no external dependencies)
Simple PUT for small objects, automatic multipart upload (5 MB parts) for large objects
Batch delete (up to 1000 keys per call)
Paginated ListObjectsV2 with prefix filtering
ListMultipartUploads + AbortMultipartUpload for cleaning up incomplete uploads
Path-style and virtual-hosted-style addressing
Comprehensive test suite with mock S3 server

Test plan

go test ./s3/... — S3 client unit tests (signing, URL construction, prefix handling) and integration tests (mock S3 server)
go test ./attachment/... — attachment store tests for both file and S3 backends
go test ./server/... — server tests pass
go test ./util/... — LimitReader, CountingReader, and LimitWriter tests

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/binwiederhier/ntfy/pull/1656 **Author:** [@binwiederhier](https://github.com/binwiederhier) **Created:** 3/15/2026 **Status:** ✅ Merged **Merged:** 3/23/2026 **Merged by:** [@binwiederhier](https://github.com/binwiederhier) **Base:** `main` ← **Head:** `attachment-s3` --- ### 📝 Commits (10+) - [`d517ce4`](https://github.com/binwiederhier/ntfy/commit/d517ce4a2aa21ef8379a8dc013fc6e58cdebbebe) WIP: S3 - [`b4ec6fa`](https://github.com/binwiederhier/ntfy/commit/b4ec6fa8df41f9ad7e7079227a9a9c9e13b02534) AWS deps.. - [`4487299`](https://github.com/binwiederhier/ntfy/commit/4487299a800d0ed28b09b98614ec3cf0fbf87e7e) Merge branch 'main' into attachment-s3 - [`790ba24`](https://github.com/binwiederhier/ntfy/commit/790ba243c766fed20f42df126dd0a69e9e662bb6) S3 WIP - [`458fbad`](https://github.com/binwiederhier/ntfy/commit/458fbad77085b20f54729f1780ea6dd7ddc3103f) Merge branch 'main' into attachment-s3 - [`86015e1`](https://github.com/binwiederhier/ntfy/commit/86015e100c00b908dbd810f6f93e2a74befbc9ee) Multipart upload - [`6b11bc7`](https://github.com/binwiederhier/ntfy/commit/6b11bc7468761a62c3bfc89bbc8c194bc2beb408) Merge branch 'main' into attachment-s3 - [`a47d692`](https://github.com/binwiederhier/ntfy/commit/a47d692cbf0aa8d85b763f8d2d0edf87e9edc253) Fix bug - [`cffa579`](https://github.com/binwiederhier/ntfy/commit/cffa57950a50af7572e570e1a743582e6850a8b6) Logs - [`ef31496`](https://github.com/binwiederhier/ntfy/commit/ef314960d015d12fcab803848f4f8a6865d0ae50) Refactor ### 📊 Changes **38 files changed** (+3089 additions, -502 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/build.yaml` (+4 -1) 📝 `.github/workflows/release.yaml` (+1 -0) 📝 `.github/workflows/test.yaml` (+5 -1) 📝 `.gitignore` (+1 -0) ➕ `attachment/backend.go` (+23 -0) ➕ `attachment/backend_file.go` (+94 -0) ➕ `attachment/backend_s3.go` (+51 -0) ➕ `attachment/store.go` (+232 -0) ➕ `attachment/store_file_test.go` (+16 -0) ➕ `attachment/store_s3_test.go` (+120 -0) ➕ `attachment/store_test.go` (+352 -0) 📝 `cmd/serve.go` (+1 -1) 📝 `docs/config.md` (+159 -86) 📝 `docs/releases.md` (+4 -0) 📝 `go.mod` (+1 -1) 📝 `go.sum` (+2 -2) 📝 `message/cache.go` (+31 -45) 📝 `message/cache_postgres.go` (+3 -0) 📝 `message/cache_sqlite.go` (+3 -0) ➕ `s3/client.go` (+302 -0) _...and 18 more files_ </details> ### 📄 Description ## Summary Adds S3-compatible object storage as a backend for attachment caching, as an alternative to local filesystem storage. - **Minimal S3 client** (`s3/` package): implements PutObject (with automatic multipart upload for large files), GetObject, DeleteObjects, and ListObjectsV2 using AWS Signature V4 signing — no AWS SDK dependency - **Attachment store** (`attachment/` package): unified `Store` with pluggable backend interface (`fileBackend` / `s3Backend`), shared size tracking, rate limiting, and background sync - **Background cleanup**: every 15 minutes, reconciles the S3 bucket (or prefix) with the message database — deletes orphaned objects whose file IDs are no longer referenced, and aborts incomplete multipart uploads older than 1 hour - **Configuration**: `attachment-cache-dir` accepts an S3 URL (`s3://ACCESS_KEY:SECRET_KEY@BUCKET[/PREFIX]?region=REGION[&endpoint=ENDPOINT]`), auto-detecting path-style addressing when a custom endpoint is set - **Documentation**: updated `docs/config.md` with S3 configuration examples and cleanup behavior ## S3 client features - AWS Signature V4 request signing (no external dependencies) - Simple PUT for small objects, automatic multipart upload (5 MB parts) for large objects - Batch delete (up to 1000 keys per call) - Paginated ListObjectsV2 with prefix filtering - ListMultipartUploads + AbortMultipartUpload for cleaning up incomplete uploads - Path-style and virtual-hosted-style addressing - Comprehensive test suite with mock S3 server ## Test plan - [x] `go test ./s3/...` — S3 client unit tests (signing, URL construction, prefix handling) and integration tests (mock S3 server) - [x] `go test ./attachment/...` — attachment store tests for both file and S3 backends - [x] `go test ./server/...` — server tests pass - [x] `go test ./util/...` — LimitReader, CountingReader, and LimitWriter tests --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>