[GH-ISSUE #638] [Bug] Spurious errors in some tests, possibly due to synchronisation #474

New issue

Closed

opened 2026-05-07 00:24:33 +02:00 by BreizhHardware · 4 comments

BreizhHardware commented

2026-05-07 00:24:33 +02:00

Owner

Originally created by @cyqsimon on GitHub (Feb 26, 2023).
Original GitHub issue: https://github.com/binwiederhier/ntfy/issues/638

I recently re-enabled tests on my COPR repo, and I've noticed some random and spurious test errors.

It seems to happen randomly on all chroots I've selected, so I assume it's an issue with the tests. Also interesting is that there seems to be a higher likelihood of errors when the build service is under heavy load, which leads me to speculate that this is some kind of multithreading synchronisation issue.

The specific tests that are erroring are TestAccount_Reservation_Add_Kills_Other_Subscribers and TestAccount_Persist_UserStats_After_Tier_Change, and the errors in both cases are assertion NEQ errors.

An example:

--- FAIL: TestAccount_Persist_UserStats_After_Tier_Change (0.61s)
    server_account_test.go:824: 
        	Error Trace:	/builddir/build/BUILD/ntfy-2.1.0/server/server_account_test.go:824
        	Error:      	Not equal: 
        	            	expected: 2
        	            	actual  : 1
        	Test:       	TestAccount_Persist_UserStats_After_Tier_Change
FAIL
FAIL	heckel.io/ntfy/server	14.486s

See attached for build logs of all errored builds in the screenshot above.

build-logs.tar.gz

Originally created by @cyqsimon on GitHub (Feb 26, 2023). Original GitHub issue: https://github.com/binwiederhier/ntfy/issues/638 I recently re-enabled tests on [my COPR repo](https://copr.fedorainfracloud.org/coprs/cyqsimon/ntfysh/builds/), and I've noticed some random and spurious test errors. ![Summary](https://user-images.githubusercontent.com/28627918/221420596-230326fc-b59d-4de8-823e-13b93178e2c3.png) It seems to happen randomly on all chroots I've selected, so I assume it's an issue with the tests. Also interesting is that there seems to be a higher likelihood of errors when the build service is under heavy load, which leads me to speculate that this is some kind of multithreading synchronisation issue. --- The specific tests that are erroring are `TestAccount_Reservation_Add_Kills_Other_Subscribers` and `TestAccount_Persist_UserStats_After_Tier_Change`, and the errors in both cases are assertion NEQ errors. An example: ``` --- FAIL: TestAccount_Persist_UserStats_After_Tier_Change (0.61s) server_account_test.go:824: Error Trace: /builddir/build/BUILD/ntfy-2.1.0/server/server_account_test.go:824 Error: Not equal: expected: 2 actual : 1 Test: TestAccount_Persist_UserStats_After_Tier_Change FAIL FAIL heckel.io/ntfy/server 14.486s ``` See attached for build logs of all errored builds in the screenshot above. [build-logs.tar.gz](https://github.com/binwiederhier/ntfy/files/10834059/build-logs.tar.gz)

BreizhHardware closed this issue

2026-05-07 00:24:33 +02:00

BreizhHardware commented

2026-05-07 00:24:34 +02:00

Author

Owner

@binwiederhier commented on GitHub (Feb 27, 2023):

I have always had issues with intermittent test failures. Tests that rely on time are hard to get right without either taking forever or intense changes in the main code.

Especially TestAccount_Reservation_Add_Kills_Other_Subscribers is quite annoying. I thought I had them all solid now though. But apparently not...

@binwiederhier commented on GitHub (Feb 27, 2023): I have always had issues with intermittent test failures. Tests that rely on time are hard to get right without either taking forever or intense changes in the main code. Especially `TestAccount_Reservation_Add_Kills_Other_Subscribers` is quite annoying. I thought I had them all solid now though. But apparently not...

BreizhHardware commented

2026-05-07 00:24:34 +02:00

Author

Owner

@cyqsimon commented on GitHub (Feb 27, 2023):

I am not very familiar with Go so I cannot help fix the tests; sorry about that.

Is it possible to disable individual tests? Maybe I can skip these two trouble-makers for the time being.

@cyqsimon commented on GitHub (Feb 27, 2023): I am not very familiar with Go so I cannot help fix the tests; sorry about that. Is it possible to disable individual tests? Maybe I can skip these two trouble-makers for the time being.

BreizhHardware commented

2026-05-07 00:24:34 +02:00

Author

Owner

@binwiederhier commented on GitHub (Feb 27, 2023):

Well: https://github.com/binwiederhier/ntfy/actions/runs/4284367614/jobs/7461222100

I removed TestAccount_Reservation_Add_Kills_Other_Subscribers and replaced it with something simpler. It is not a full end-to-end test anymore, but I guess that's fine.....

The other one I gave a little more time. Let me know if it doesnt work. I'm gonna close this for now. If it happens again feel free to comment again and I'll reopen.

@binwiederhier commented on GitHub (Feb 27, 2023): Well: https://github.com/binwiederhier/ntfy/actions/runs/4284367614/jobs/7461222100 I removed TestAccount_Reservation_Add_Kills_Other_Subscribers and replaced it with something simpler. It is not a full end-to-end test anymore, but I guess that's fine..... The other one I gave a little more time. Let me know if it doesnt work. I'm gonna close this for now. If it happens again feel free to comment again and I'll reopen.

BreizhHardware commented

2026-05-07 00:24:34 +02:00

Author

Owner

@binwiederhier commented on GitHub (Mar 1, 2023):

I think I fixed the other one too. Let me know if it doesnt work for you.

@binwiederhier commented on GitHub (Mar 1, 2023): I think I fixed the other one too. Let me know if it doesnt work for you.