starred/the-bastion
1
0
Fork 0
mirror of https://github.com/ovh/the-bastion.git synced 2026-05-09 08:25:27 +02:00
Code Issues 46 Projects Packages Wiki Activity

[GH-ISSUE #486] SCP freezes wthen downloading large files: #125

New issue
Closed
opened 2026-05-07 00:19:12 +02:00 by BreizhHardware · 15 comments
BreizhHardware commented 2026-05-07 00:19:12 +02:00
Owner
Copy link

Originally created by @tgeek77 on GitHub (Jul 10, 2024).
Original GitHub issue: https://github.com/ovh/the-bastion/issues/486

This is the command that I am using at the output:

> ~/.ssh/bsshscp -P 222 root@foo:/root/logs.tgz ~/temp
*------------------------------------------------------------------------------*
|THIS IS A PRIVATE COMPUTER SYSTEM, UNAUTHORIZED ACCESS IS STRICTLY PROHIBITED.|
|ALL CONNECTIONS ARE LOGGED. IF YOU ARE NOT AUTHORIZED, DISCONNECT NOW.        |
*------------------------------------------------------------------------------*
 will try the following accesses you have:
  - group-member of admin with ED25519-256 key SHA256:nNbqefYIKSOMTha2XMXpMFu8KDx3Ws0GulB7VIRIBps [2024/06/19] 

MFA_TOKEN=notrequired
Connection to dev-bastion closed.
*------------------------------------------------------------------------------*
|THIS IS A PRIVATE COMPUTER SYSTEM, UNAUTHORIZED ACCESS IS STRICTLY PROHIBITED.|
|ALL CONNECTIONS ARE LOGGED. IF YOU ARE NOT AUTHORIZED, DISCONNECT NOW.        |
*------------------------------------------------------------------------------*
>>> Hello user, transferring your file through the bastion from root@192.168.0.1:222...
logs.tgz                                                                                                                                                  96% 2079MB  97.1MB/s   00:00 ETA

As you can see, the file is over 2GB. If I try this with a small file, it works. If I try it with directly scp (not using the bastion), it also works. This seems to be a hard limit to upload/download sizes that is not documented. I checked here and here.

Originally created by @tgeek77 on GitHub (Jul 10, 2024). Original GitHub issue: https://github.com/ovh/the-bastion/issues/486 This is the command that I am using at the output: ``` > ~/.ssh/bsshscp -P 222 root@foo:/root/logs.tgz ~/temp *------------------------------------------------------------------------------* |THIS IS A PRIVATE COMPUTER SYSTEM, UNAUTHORIZED ACCESS IS STRICTLY PROHIBITED.| |ALL CONNECTIONS ARE LOGGED. IF YOU ARE NOT AUTHORIZED, DISCONNECT NOW. | *------------------------------------------------------------------------------* will try the following accesses you have: - group-member of admin with ED25519-256 key SHA256:nNbqefYIKSOMTha2XMXpMFu8KDx3Ws0GulB7VIRIBps [2024/06/19] MFA_TOKEN=notrequired Connection to dev-bastion closed. *------------------------------------------------------------------------------* |THIS IS A PRIVATE COMPUTER SYSTEM, UNAUTHORIZED ACCESS IS STRICTLY PROHIBITED.| |ALL CONNECTIONS ARE LOGGED. IF YOU ARE NOT AUTHORIZED, DISCONNECT NOW. | *------------------------------------------------------------------------------* >>> Hello user, transferring your file through the bastion from root@192.168.0.1:222... logs.tgz 96% 2079MB 97.1MB/s 00:00 ETA ``` As you can see, the file is over 2GB. If I try this with a small file, it works. If I try it with directly scp (not using the bastion), it also works. This seems to be a hard limit to upload/download sizes that is not documented. I checked [here](https://ovh.github.io/the-bastion/using/sftp_scp.html) and [here](https://ovh.github.io/the-bastion/administration/configuration/bastion_conf.html).
BreizhHardware commented 2026-05-07 00:19:13 +02:00
Author
Owner
Copy link

@Kladislav commented on GitHub (Aug 14, 2024):

same issue

<!-- gh-comment-id:2289708897 --> @Kladislav commented on GitHub (Aug 14, 2024): same issue
BreizhHardware commented 2026-05-07 00:19:13 +02:00
Author
Owner
Copy link

@Kladislav commented on GitHub (Aug 14, 2024):

@tgeek77 mb already solve?

<!-- gh-comment-id:2289710541 --> @Kladislav commented on GitHub (Aug 14, 2024): @tgeek77 mb already solve?
BreizhHardware commented 2026-05-07 00:19:13 +02:00
Author
Owner
Copy link

@tgeek77 commented on GitHub (Aug 15, 2024):

No, it's still happening on my system.

<!-- gh-comment-id:2290970907 --> @tgeek77 commented on GitHub (Aug 15, 2024): No, it's still happening on my system.
BreizhHardware commented 2026-05-07 00:19:14 +02:00
Author
Owner
Copy link

@Kladislav commented on GitHub (Aug 15, 2024):

idk, it looks like huge bug and no activity here, sad...

<!-- gh-comment-id:2292027500 --> @Kladislav commented on GitHub (Aug 15, 2024): idk, it looks like huge bug and no activity here, sad...
BreizhHardware commented 2026-05-07 00:19:14 +02:00
Author
Owner
Copy link

@speed47 commented on GitHub (Dec 9, 2024):

Hello,

I had some trouble reproducing it, but indeed it seems that on large files, it stalls at the end of the file regardless of its size, as long as it's big enough. Seems like some kind of race condition, I'll dig into it.

<!-- gh-comment-id:2527620470 --> @speed47 commented on GitHub (Dec 9, 2024): Hello, I had some trouble reproducing it, but indeed it seems that on large files, it stalls at the end of the file regardless of its size, as long as it's big enough. Seems like some kind of race condition, I'll dig into it.
BreizhHardware commented 2026-05-07 00:19:14 +02:00
Author
Owner
Copy link

@speed47 commented on GitHub (Dec 10, 2024):

OK, it can actually happen with files as small as 50 MiB, but the bigger the file, the most likely it'll happen. This seems to happen mostly on downloads, not so much on uploads.

After some stracing, it appears that once the file has been read on the sending side, the scp server closes its file descriptor and sends an EOF as it should, but then waits for confirmation that the file has indeed been transferred forever: this is where sometimes, the deadlock happens. When it does, you'll be missing 64 KiB (the size of the read buffer) of the file on the receiving side, no matter what.

I've changed the exec mode of the scp plugin to remove some buffering that happens on the bastion itself, as I guessed this would be where the race condition could happen. With this change, I can no longer reproduce the issue. The only noticeable drawback is that the bastion can no longer report how many bytes have flowed through scp. The line:

>>> Done, 4739 bytes uploaded, 294878 bytes downloaded

will be replaced by:

>>> Done, scp exited successfully.

which seems a negligible compromise to make.

<!-- gh-comment-id:2531161412 --> @speed47 commented on GitHub (Dec 10, 2024): OK, it can actually happen with files as small as 50 MiB, but the bigger the file, the most likely it'll happen. This seems to happen mostly on downloads, not so much on uploads. After some stracing, it appears that once the file has been read on the sending side, the scp server closes its file descriptor and sends an EOF as it should, but then waits for confirmation that the file has indeed been transferred forever: this is where sometimes, the deadlock happens. When it does, you'll be missing 64 KiB (the size of the read buffer) of the file on the receiving side, no matter what. I've changed the exec mode of the scp plugin to remove some buffering that happens on the bastion itself, as I guessed this would be where the race condition could happen. With this change, I can no longer reproduce the issue. The only noticeable drawback is that the bastion can no longer report how many bytes have flowed through scp. The line: `>>> Done, 4739 bytes uploaded, 294878 bytes downloaded` will be replaced by: `>>> Done, scp exited successfully.` which seems a negligible compromise to make.
BreizhHardware commented 2026-05-07 00:19:14 +02:00
Author
Owner
Copy link

@tgeek77 commented on GitHub (Dec 10, 2024):

That sounds like an acceptable compromise to me. My team is using The Bastion in a our production environment and we have people asking to add their own SSH keys to the server as a workaround when manually gathering logs. It would be good to see this fix.

<!-- gh-comment-id:2531224837 --> @tgeek77 commented on GitHub (Dec 10, 2024): That sounds like an acceptable compromise to me. My team is using The Bastion in a our production environment and we have people asking to add their own SSH keys to the server as a workaround when manually gathering logs. It would be good to see this fix.
BreizhHardware commented 2026-05-07 00:19:14 +02:00
Author
Owner
Copy link

@speed47 commented on GitHub (Dec 10, 2024):

fixed in v3.18.00, if you could confirm that the issue does disappear on your environment too with this version, that would be great!

<!-- gh-comment-id:2531631362 --> @speed47 commented on GitHub (Dec 10, 2024): fixed in `v3.18.00`, if you could confirm that the issue does disappear on your environment too with this version, that would be great!
BreizhHardware commented 2026-05-07 00:19:14 +02:00
Author
Owner
Copy link

@tgeek77 commented on GitHub (Dec 10, 2024):

I have scheduled a change window for 1hr 15m from now. I will update with my test results.

<!-- gh-comment-id:2531763927 --> @tgeek77 commented on GitHub (Dec 10, 2024): I have scheduled a change window for 1hr 15m from now. I will update with my test results.
BreizhHardware commented 2026-05-07 00:19:15 +02:00
Author
Owner
Copy link

@tgeek77 commented on GitHub (Dec 10, 2024):

Works for me! Test file was > 100MB

$ bscp -i ~/.ssh/key_ed25519 root@foo.example.com:/var/log/messages.1 ./
*------------------------------------------------------------------------------*
|THIS IS A PRIVATE COMPUTER SYSTEM, UNAUTHORIZED ACCESS IS STRICTLY PROHIBITED.|
|ALL CONNECTIONS ARE LOGGED. IF YOU ARE NOT AUTHORIZED, DISCONNECT NOW.        |
*------------------------------------------------------------------------------*
 will try the following accesses you have:
  - group-member of dev with ED25519-256 key SHA256:RdoCbMCJ+ORa23l4dLRfHeqXjgSg64x8AtEWlpyq3XE [2024/06/19] 
  - group-member of admin with ED25519-256 key SHA256:nNbqefYIKSOMTha2XMXpMFu8KDx3Ws0GulB7VIRIBps [2024/06/19] 

MFA_TOKEN=notrequired
Connection to dev-bastion.dev.office.ad closed.
*------------------------------------------------------------------------------*
|THIS IS A PRIVATE COMPUTER SYSTEM, UNAUTHORIZED ACCESS IS STRICTLY PROHIBITED.|
|ALL CONNECTIONS ARE LOGGED. IF YOU ARE NOT AUTHORIZED, DISCONNECT NOW.        |
*------------------------------------------------------------------------------*
>>> Hello tgeek77, transferring your file through the bastion from root@x.x.x.x:22...
messages.1                                                                                                                                                   100%  126MB  19.1MB/s   00:06    
>>> Done, scp exited successfully.
<!-- gh-comment-id:2532111751 --> @tgeek77 commented on GitHub (Dec 10, 2024): Works for me! Test file was > 100MB ``` $ bscp -i ~/.ssh/key_ed25519 root@foo.example.com:/var/log/messages.1 ./ *------------------------------------------------------------------------------* |THIS IS A PRIVATE COMPUTER SYSTEM, UNAUTHORIZED ACCESS IS STRICTLY PROHIBITED.| |ALL CONNECTIONS ARE LOGGED. IF YOU ARE NOT AUTHORIZED, DISCONNECT NOW. | *------------------------------------------------------------------------------* will try the following accesses you have: - group-member of dev with ED25519-256 key SHA256:RdoCbMCJ+ORa23l4dLRfHeqXjgSg64x8AtEWlpyq3XE [2024/06/19] - group-member of admin with ED25519-256 key SHA256:nNbqefYIKSOMTha2XMXpMFu8KDx3Ws0GulB7VIRIBps [2024/06/19] MFA_TOKEN=notrequired Connection to dev-bastion.dev.office.ad closed. *------------------------------------------------------------------------------* |THIS IS A PRIVATE COMPUTER SYSTEM, UNAUTHORIZED ACCESS IS STRICTLY PROHIBITED.| |ALL CONNECTIONS ARE LOGGED. IF YOU ARE NOT AUTHORIZED, DISCONNECT NOW. | *------------------------------------------------------------------------------* >>> Hello tgeek77, transferring your file through the bastion from root@x.x.x.x:22... messages.1 100% 126MB 19.1MB/s 00:06 >>> Done, scp exited successfully. ```
BreizhHardware commented 2026-05-07 00:19:15 +02:00
Author
Owner
Copy link

@speed47 commented on GitHub (Dec 13, 2024):

Good, internally some people had the issue and they confirm that the issue disappeared too. Closing as resolved then!

<!-- gh-comment-id:2541406691 --> @speed47 commented on GitHub (Dec 13, 2024): Good, internally some people had the issue and they confirm that the issue disappeared too. Closing as resolved then!
BreizhHardware commented 2026-05-07 00:19:15 +02:00
Author
Owner
Copy link

@niallguerin commented on GitHub (Jan 22, 2025):

Just confirming this scp fix works for scp download of files up to 5GB so far for us, so thank you for fixing. I notice rsync pull (and only rsync pull) seems to have a similar pattern of behaviour in that I can push large files via rsync fine from system A to system B via bastion, but when I test rsync pull via bastion, it seems to be ok up to about 100mb max and then it won't let me pull down larger files. Even below 100mb, it sometimes gives errors when it is over 45-50mb for the rsync pull and I had similar test case results with scp download until this fix and that issue is resolved. Let me know if you prefer I open a separate issue for tracking for rsync pull test, if you can recreate. Testing (for scp download) and rsync pull done against bastion version v3.18.00.

Error Result

unexpected tag 90 [receiver/inc]
rsync error: error in rsync protocol data stream (code 12) at io.c(1705) [receiver=3.4.1]
rsync: connection unexpectedly closed (54 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(232) [generator=3.4.1]
<!-- gh-comment-id:2607477140 --> @niallguerin commented on GitHub (Jan 22, 2025): Just confirming this scp fix works for scp download of files up to 5GB so far for us, so thank you for fixing. I notice rsync pull (and only rsync pull) seems to have a similar pattern of behaviour in that I can push large files via rsync fine from system A to system B via bastion, but when I test rsync pull via bastion, it seems to be ok up to about 100mb max and then it won't let me pull down larger files. Even below 100mb, it sometimes gives errors when it is over 45-50mb for the rsync pull and I had similar test case results with scp download until this fix and that issue is resolved. Let me know if you prefer I open a separate issue for tracking for rsync pull test, if you can recreate. Testing (for scp download) and rsync pull done against bastion version v3.18.00. Error Result ``` unexpected tag 90 [receiver/inc] rsync error: error in rsync protocol data stream (code 12) at io.c(1705) [receiver=3.4.1] rsync: connection unexpectedly closed (54 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(232) [generator=3.4.1] ```
BreizhHardware commented 2026-05-07 00:19:15 +02:00
Author
Owner
Copy link

@tgeek77 commented on GitHub (Jan 23, 2025):

@niallguerin I think it would be a good idea to also give an example of the command that you are using. You can obfuscate real hostnames with example.com or use x.x.x.x for IP addresses.

<!-- gh-comment-id:2609695360 --> @tgeek77 commented on GitHub (Jan 23, 2025): @niallguerin I think it would be a good idea to also give an example of the command that you are using. You can obfuscate real hostnames with example.com or use x.x.x.x for IP addresses.
BreizhHardware commented 2026-05-07 00:19:16 +02:00
Author
Owner
Copy link

@speed47 commented on GitHub (Feb 6, 2025):

@niallguerin could you open a new issue indeed? I'll see if I can reproduce. When this happens, does rsync appear to just be stuck, so you CTRL+C it and then it shows the error lines you've pasted, or does this give out an error directly, instead of getting stuck?

If it's stuck, it's probably a similar issue and the fix might be the same, otherwise, it can be a different issue. You might want to include the whole command you've typed and the whole output, removing any sensitive information as @tgeek77 suggested. Thanks!

<!-- gh-comment-id:2640623788 --> @speed47 commented on GitHub (Feb 6, 2025): @niallguerin could you open a new issue indeed? I'll see if I can reproduce. When this happens, does rsync appear to just be stuck, so you CTRL+C it and then it shows the error lines you've pasted, or does this give out an error directly, instead of getting stuck? If it's stuck, it's probably a similar issue and the fix might be the same, otherwise, it can be a different issue. You might want to include the whole command you've typed and the whole output, removing any sensitive information as @tgeek77 suggested. Thanks!
BreizhHardware commented 2026-05-07 00:19:16 +02:00
Author
Owner
Copy link

@niallguerin commented on GitHub (Feb 16, 2025):

Sorry for delay in responding. I'll open a separate issue as requested. I just want to set up some stuff to reproduce on another server at home as well as work. To answer your last reply, unlike the old scp behaviour (that was fixed), the rsync fails immediately without input from me (i.e. I do not hit CTRL+C in terminal) and the rsync pull fail case only occurs when the file is large whereas in the old scp large file download issue it did just become "stuck", rsync large file errors out immediately for me. Smaller files are fine though for rsync pull. I'll send full command and log output with hosts obfuscated as requested and I will log it as separate issue as scp issue is fixed for large files. Thank you.

<!-- gh-comment-id:2661429175 --> @niallguerin commented on GitHub (Feb 16, 2025): Sorry for delay in responding. I'll open a separate issue as requested. I just want to set up some stuff to reproduce on another server at home as well as work. To answer your last reply, unlike the old scp behaviour (that was fixed), the rsync fails immediately without input from me (i.e. I do not hit CTRL+C in terminal) and the rsync pull fail case only occurs when the file is large whereas in the old scp large file download issue it did just become "stuck", rsync large file errors out immediately for me. Smaller files are fine though for rsync pull. I'll send full command and log output with hosts obfuscated as requested and I will log it as separate issue as scp issue is fixed for large files. Thank you.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
expifix
groupcreate_fix_uid
fix_group_gid_fast
gh-pages
clush
expiration
jsonvalidator
v3.23.01
v3.23.00
v3.22.00
v3.21.00
v3.20.00
v3.19.01
v3.19.00
v3.18.99-rc1
v3.18.00
v3.17.01
v3.17.00
v3.16.99-rc3
v3.16.99-rc2
v3.16.99-rc1
v3.16.01
v3.16.00
v3.15.00
v3.14.16
v3.14.15
v3.14.00
v3.13.01
v3.13.00
v3.12.00
v3.11.02
v3.11.01
v3.11.00
v3.10.00
v3.09.02
v3.09.00-really
v3.09.00
v3.09.00-rc3
v3.09.00-rc2
v3.09.00-rc1
v3.08.01
v3.08.00
v3.07.00
v3.06.00
v3.05.01
v3.05.00
v3.04.00
v3.03.99-rc2
v3.03.99-rc1
v3.03.01
v3.03.00
v3.02.00
v3.01.99-rc4
v3.01.99-rc3
v3.01.99-rc2
v3.01.99-rc1
v3.01.03
v3.01.02
v3.01.01
v3.01.00
v3.00.02
v3.00.01
v3.00.00
Labels
Clear labels   
answered

   
bug

   
documentation

   
enhancement

   
enhancement

   
feature

   
feature

   
kept-open-for-info

   
pull-request
Mirrored from GitHub Pull Request

  
question
No labels
answered
bug
documentation
enhancement
enhancement
feature
feature
kept-open-for-info
pull-request
question
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/the-bastion#125
No description provided.