Context
-
Version of iperf3: 3.6
-
Hardware: raspberry pi 3 b
-
Operating system (and distribution, if any): Debian Linux (default pi OS)
Bug Report
I’m trying to run iperf3.6 over a 100kbps link to measuer the actual throughput. I’m running it on two pis.
On the server, I run «iperf3 -s —logfile /dev/tty
On the client, I run «iperf3 -c (IPv6 address of peer) -u -b 390k -l 1024 -t 12»
-
Expected Behavior I see 12 reports, one second apart, then the summary
-
Actual Behavior, on the client I see 11 reports, one second apart, then a pause, then iperf3: error — control socket has closed unexpectedly. On the server I see 16 reports one second apart then iperf3: error — select failed: Bad file descriptor
-
Steps to Reproduce: given above
Server side:
Server listening on 5201
Accepted connection from (IP addr), port 56688
[ 6] local (ip addr) port 5201 connected to (ip addr) port 54260
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 6] 0.00-1.00 sec 9.00 KBytes 73.7 Kbits/sec 37.226 ms 0/9 (0%)
[ 6] 1.00-2.00 sec 10.0 KBytes 81.9 Kbits/sec 55.150 ms 0/10 (0%)
[ 6] 2.00-3.00 sec 11.0 KBytes 90.1 Kbits/sec 65.500 ms 0/11 (0%)
[ 6] 3.00-4.00 sec 9.00 KBytes 73.7 Kbits/sec 72.033 ms 0/9 (0%)
[ 6] 4.00-5.00 sec 11.0 KBytes 90.1 Kbits/sec 73.836 ms 0/11 (0%)
[ 6] 5.00-6.00 sec 10.0 KBytes 81.9 Kbits/sec 74.872 ms 0/10 (0%)
[ 6] 6.00-7.00 sec 11.0 KBytes 90.1 Kbits/sec 74.872 ms 0/11 (0%)
[ 6] 7.00-8.00 sec 10.0 KBytes 81.9 Kbits/sec 75.292 ms 0/10 (0%)
[ 6] 8.00-9.00 sec 10.0 KBytes 81.9 Kbits/sec 75.182 ms 0/10 (0%)
[ 6] 9.00-10.00 sec 11.0 KBytes 90.1 Kbits/sec 75.403 ms 0/11 (0%)
[ 6] 10.00-11.00 sec 10.0 KBytes 81.9 Kbits/sec 89.071 ms 26/36 (72%)
[ 6] 11.00-12.00 sec 10.0 KBytes 81.9 Kbits/sec 53.561 ms 38/48 (79%)
[ 6] 12.00-13.00 sec 10.0 KBytes 81.9 Kbits/sec 36.800 ms 36/46 (78%)
[ 6] 13.00-14.00 sec 10.0 KBytes 81.9 Kbits/sec 25.029 ms 35/45 (78%)
[ 6] 14.00-15.00 sec 11.0 KBytes 90.1 Kbits/sec 17.210 ms 40/51 (78%)
[ 6] 15.00-16.00 sec 10.0 KBytes 81.9 Kbits/sec 13.892 ms 36/46 (78%)
iperf3: error — select failed: Bad file descriptor
Server listening on 5201
Client side:
Connecting to host (IP addr) port 5201
[ 5] local (IP addr) port 54260 connected to (ip addr) port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 1.00-2.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 2.00-3.00 sec 47.0 KBytes 385 Kbits/sec 47
[ 5] 3.00-4.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 4.00-5.00 sec 47.0 KBytes 385 Kbits/sec 47
[ 5] 5.00-6.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 6.00-7.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 7.00-8.00 sec 47.0 KBytes 385 Kbits/sec 47
[ 5] 8.00-9.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 9.00-10.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 10.00-11.00 sec 47.0 KBytes 385 Kbits/sec 47
iperf3: error — control socket has closed unexpectedly
This is on a private network that uses valid public network addresses.
Don’t know why part of the output above is in bold.
Problem only occurs with -l 256 and above. Finishes normally with -l 128
What kind of link is it?
I can run just fine using an ODROID-C2. I don’t have an RPI 3B. Amlogic S905 Quad Core Cortex-A53 (ODROID C2) vs Broadcom BCM2837B0 Quad Core Cortex-A53 (RPI). I am running Ubuntu 20.04.1 LTS.
# ./iperf36 --client 192.168.X.X --udp --bandwidth 390k -l 1024
Connecting to host 192.168.X.X, port 5201
[ 5] 0.00-1.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 1.00-2.00 sec 47.0 KBytes 385 Kbits/sec 47
[ 5] 2.00-3.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 3.00-4.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 4.00-5.00 sec 47.0 KBytes 385 Kbits/sec 47
[ 5] 5.00-6.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 6.00-7.00 sec 47.0 KBytes 385 Kbits/sec 47
[ 5] 7.00-8.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 8.00-9.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 9.00-10.00 sec 48.0 KBytes 393 Kbits/sec 48
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 477 KBytes 391 Kbits/sec 0.000 ms 0/477 (0%) sender
[ 5] 0.00-10.00 sec 477 KBytes 391 Kbits/sec 0.018 ms 0/477 (0%) receiver
iperf Done.
Same for IPv6 …
./iperf36 --client SERVER_IPv6_ADDRESS --udp --bandwidth 390k -l 1024
Connecting to host SERVER_IPv6_ADDRESS, port 5201
[ 5] local CLIENT_IPv6_ADDRESS port 52823 connected to SERVER_IPv6_ADDRESS port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 1.00-2.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 2.00-3.00 sec 47.0 KBytes 385 Kbits/sec 47
[ 5] 3.00-4.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 4.00-5.00 sec 47.0 KBytes 385 Kbits/sec 47
[ 5] 5.00-6.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 6.00-7.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 7.00-8.00 sec 47.0 KBytes 385 Kbits/sec 47
[ 5] 8.00-9.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 9.00-10.00 sec 48.0 KBytes 393 Kbits/sec 48
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 477 KBytes 391 Kbits/sec 0.000 ms 0/477 (0%) sender
[ 5] 0.00-10.00 sec 477 KBytes 391 Kbits/sec 0.019 ms 0/477 (0%) receiver
iperf Done.
Don’t know why part of the output above is in bold
A main issue is that while the client sends about 48KB/sec (390Kbits/sec), the server is receiving only 10KB/sec. It seems that the network throughput is only about 100Kbis/sec. The client probably have a send buffer of about 100KB since the first 100KB were properly received by the server.
After the client completed sending, sending of the 100KB buffered packets continue. This is why the server continue to receive data after the client completed the sending. If my calculations are correct it would take 10 seconds (after the first 12 secs) for the server to receive all buffered data.
Can you send the client and server output when -l 128 was used? In principle it should not change much.
Something happened that that caused the error, I am not sure what, but it may help to try a newer version of iperf3. Version 3.6 is quite old and there were some related fixes since.
Can you try with 3.9? I see this in the release notes for 3.7 …
«The delay for tearing down the control connection for the default timed tests has been increased, to more gracefully handle high-delay paths (#751/#859).»
May be related.
Client is an 1GbE ODROID-C2 running Ubuntu 20.04.1 LTS. Server is a 10GbE rack mount server running Ubuntu 20.04.2 LTS.
I think your link could be causing the issue. Can you provide more details on this 100Kbps link?
# ./iperf36 --client SERVER_IPv4_ADDRESS --udp --bandwidth 390k --time 12 --length 128
Connecting to host SERVER_IPv4_ADDRESS, port 5201
[ 5] local CLIENT_IPv4_ADDRESS port 58317 connected to SERVER_IPv4_ADDRESS port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 1.00-2.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 2.00-3.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 3.00-4.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 4.00-5.00 sec 47.5 KBytes 389 Kbits/sec 380
[ 5] 5.00-6.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 6.00-7.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 7.00-8.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 8.00-9.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 9.00-10.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 10.00-11.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 11.00-12.00 sec 47.5 KBytes 389 Kbits/sec 380
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-12.00 sec 571 KBytes 390 Kbits/sec 0.000 ms 0/4570 (0%) sender
[ 5] 0.00-12.03 sec 571 KBytes 389 Kbits/sec 0.042 ms 0/4570 (0%) receiver
# ./iperf36 --server
Server listening on 5201
-----------------------------------------------------------
Accepted connection from CLIENT_IPv4_ADDRESS, port 38364
[ 5] local SERVER_IPv4_ADDRESS port 5201 connected to CLIENT_IPv4_ADDRESS port 58317
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 46.0 KBytes 377 Kbits/sec 0.028 ms 0/368 (0%)
[ 5] 1.00-2.00 sec 47.6 KBytes 390 Kbits/sec 0.038 ms 0/381 (0%)
[ 5] 2.00-3.00 sec 47.6 KBytes 390 Kbits/sec 0.043 ms 0/381 (0%)
[ 5] 3.00-4.00 sec 47.6 KBytes 390 Kbits/sec 0.038 ms 0/381 (0%)
[ 5] 4.00-5.00 sec 47.6 KBytes 390 Kbits/sec 0.043 ms 0/381 (0%)
[ 5] 5.00-6.00 sec 47.6 KBytes 390 Kbits/sec 0.037 ms 0/381 (0%)
[ 5] 6.00-7.00 sec 47.6 KBytes 390 Kbits/sec 0.019 ms 0/381 (0%)
[ 5] 7.00-8.00 sec 47.5 KBytes 389 Kbits/sec 0.035 ms 0/380 (0%)
[ 5] 8.00-9.00 sec 47.6 KBytes 390 Kbits/sec 0.045 ms 0/381 (0%)
[ 5] 9.00-10.00 sec 47.6 KBytes 390 Kbits/sec 0.034 ms 0/381 (0%)
[ 5] 10.00-11.00 sec 47.6 KBytes 390 Kbits/sec 0.036 ms 0/381 (0%)
[ 5] 11.00-12.00 sec 47.6 KBytes 390 Kbits/sec 0.046 ms 0/381 (0%)
[ 5] 12.00-12.03 sec 1.50 KBytes 361 Kbits/sec 0.042 ms 0/12 (0%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-12.03 sec 571 KBytes 389 Kbits/sec 0.042 ms 0/4570 (0%) receiver
# ./iperf36 --client SERVER_IPv6_ADDRESS --udp --bandwidth 390k --time 12 --length 128
Connecting to host SERVER_IPv6_ADDRESS, port 5201
[ 5] local CLIENT_IPv6_ADDRESS port 60491 connected to SERVER_IPv6_ADDRESS port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 1.00-2.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 2.00-3.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 3.00-4.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 4.00-5.00 sec 47.5 KBytes 389 Kbits/sec 380
[ 5] 5.00-6.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 6.00-7.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 7.00-8.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 8.00-9.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 9.00-10.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 10.00-11.00 sec 47.6 KBytes 390 Kbits/sec 381
[ 5] 11.00-12.00 sec 47.5 KBytes 389 Kbits/sec 380
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-12.00 sec 571 KBytes 390 Kbits/sec 0.000 ms 0/4570 (0%) sender
[ 5] 0.00-12.04 sec 571 KBytes 389 Kbits/sec 0.014 ms 0/4570 (0%) receiver
# ./iperf36 --server
Server listening on 5201
-----------------------------------------------------------
Accepted connection from CLIENT_IPv6_ADDRESS, port 55168
[ 5] local SERVER_IPv6_ADDRESS port 5201 connected to CLIENT_IPv6_ADDRESS port 60491
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 45.9 KBytes 376 Kbits/sec 0.030 ms 0/367 (0%)
[ 5] 1.00-2.00 sec 47.6 KBytes 390 Kbits/sec 0.015 ms 0/381 (0%)
[ 5] 2.00-3.00 sec 47.6 KBytes 390 Kbits/sec 0.022 ms 0/381 (0%)
[ 5] 3.00-4.00 sec 47.6 KBytes 390 Kbits/sec 0.031 ms 0/381 (0%)
[ 5] 4.00-5.00 sec 47.6 KBytes 390 Kbits/sec 0.024 ms 0/381 (0%)
[ 5] 5.00-6.00 sec 47.6 KBytes 390 Kbits/sec 0.036 ms 0/381 (0%)
[ 5] 6.00-7.00 sec 47.5 KBytes 389 Kbits/sec 0.030 ms 0/380 (0%)
[ 5] 7.00-8.00 sec 47.6 KBytes 390 Kbits/sec 0.025 ms 0/381 (0%)
[ 5] 8.00-9.00 sec 47.6 KBytes 390 Kbits/sec 0.034 ms 0/381 (0%)
[ 5] 9.00-10.00 sec 47.6 KBytes 390 Kbits/sec 0.014 ms 0/381 (0%)
[ 5] 10.00-11.00 sec 47.6 KBytes 390 Kbits/sec 0.030 ms 0/381 (0%)
[ 5] 11.00-12.00 sec 47.6 KBytes 390 Kbits/sec 0.030 ms 0/381 (0%)
[ 5] 12.00-12.04 sec 1.62 KBytes 366 Kbits/sec 0.014 ms 0/13 (0%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-12.04 sec 571 KBytes 389 Kbits/sec 0.014 ms 0/4570 (0%) receiver
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.
Already on GitHub?
Sign in
to your account
Open
jiachenjun opened this issue
Jul 28, 2017
· 17 comments
Comments
NOTE: The iperf3 issue tracker is for registering bugs, enhancement
requests, or submissions of code. It is not a means for asking
questions about building or using iperf3. Those are best directed
towards the iperf3 mailing list at iperf-dev@google-groups.com or
question sites such as Stack Overflow
(http://www.stackoverflow.com/). A list of frequently-asked questions
regarding iperf3 can be found at http://software.es.net/iperf/faq.html.
Context
-
Version of iperf3:
iperf 3.1.7 -
Hardware:
VM for linux 64bit kernel version 4.4.70 -
Operating system (and distribution, if any):
LFS
Please note: iperf3 is supported on Linux, FreeBSD, and macOS.
Support may be provided on a best-effort basis to other UNIX-like
platforms. We cannot provide support for building and/or running
iperf3 on Windows, iOS, or Android. -
Other relevant information (for example, non-default compilers,
libraries, cross-compiling, etc.):
Please fill out one of the «Bug Report» or «Enhancement Request»
sections, as appropriate.
Bug Report
- Expected Behavior
iperf3 can send sctp traffic successfully. - Actual Behavior
# ./iperf3 -c 192.168.1.7 --sctp -R -b 300M
Connecting to host 192.168.1.7, port 5201
Reverse mode, remote host 192.168.1.7 is sending
[ 4] local 192.168.1.3 port 60663 connected to 192.168.1.7 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 36.4 MBytes 306 Mbits/sec
**iperf3: error - control socket has closed unexpectedly**
- Steps to Reproduce
create sever in one VM with./iperf3 -s 192.168.1.7
start send traffic on another VM with./iperf3 -c 192.168.1.7 --sctp - Possible Solution
N/A
Please submit patches or code changes as a pull request.
Enhancement Request
-
Current behavior
-
Desired behavior
-
Implementation notes
If submitting a proposed implementation of an enhancement request,
please use the pull request mechanism.
EDIT: Do some MarkDown for legibility. —bmah
I download iperf3.2 and try again, same issue still existed.
#./iperf3 -v
iperf 3.2 (cJSON 1.5.2)
Linux 4.4.70-pc64 #1 SMP Mon Jun 26 18:34:34 UTC 2017 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, SCTP, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication
#./iperf3 -c 192.168.1.7 -p 10001 --sctp
Connecting to host 192.168.1.7, port 10001
[ 5] local 192.168.1.3 port 59847 connected to 192.168.1.7 port 10001
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 200 MBytes 1.68 Gbits/sec
iperf3: error - unable to write to stream socket: Connection reset by peer
EDIT: MarkDown for legibility. —bmah
I agree it shouldn’t do that. Can you show server-side output as well?
bmah888
changed the title
iperf3 issue:»iperf3: error — control socket has closed unexpectedly»
SCTP test produces «iperf3: error — control socket has closed unexpectedly»
Aug 1, 2017
No feedback after 2.5 months, and I can’t reproduce this with tip of master branch code running on CentOS 7. Closing for now, I can reopen this issue later if necessary.
hi. I am testing this on 3G with udp — packet length 1448B for a 30 second interval. using iperf 3.5 unable to go beyond 28 seconds.
pushing 3 Mbps upstream towards a server
Server output comes as » iperf3 : error — select failed : Bad file descripter.
Hi All
I have figured out that iperf 3.5 fixes this. I actually had iperf 3.4 there. issue is resolved in iperf 3.5
Hi All,
How to install iperf 3.5 in linux?
you can download source code, and upload to VM where you want to use it.
compile & install in vm directly.
Something like:
mkdir iperf3
cd iperf3
tar -xf ../iperf3.5.tar
cd iperf3
./configure
make && make install
From: Shruti Hegde <notifications@github.com>
Sent: Monday, November 5, 2018 3:25 PM
To: esnet/iperf <iperf@noreply.github.com>
Cc: Jia, Chen Jun (NSB — CN/Hangzhou) <chen_jun.jia@nokia-sbell.com>; Author <author@noreply.github.com>
Subject: Re: [esnet/iperf] SCTP test produces «iperf3: error — control socket has closed unexpectedly» (#620)
Hi All,
How to install iperf 3.5 in linux?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#620 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AUX_NNpEnLLk9EM0uoc4VinD4Jw6OQuuks5ur-fEgaJpZM4OmQM1>.
I get this issue on debian buster x64 with the distribution’s version of iperf3.
Client:
# iperf3 --sctp -b 10m -c 10.99.11.1
Connecting to host 10.99.11.1, port 5201
[ 5] local 10.99.37.240 port 42749 connected to 10.99.11.1 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec
[ 5] 1.00-2.00 sec 1.19 MBytes 9.96 Mbits/sec
iperf3: error - control socket has closed unexpectedly
Server:
# /usr/bin/iperf3 -s -d
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
get_parameters:
{
"sctp": true,
"omit": 0,
"time": 10,
"parallel": 1,
"len": 65536,
"bandwidth": 10000000,
"pacing_timer": 1000,
"client_version": "3.6"
}
Accepted connection from 10.99.37.240, port 33444
[ 5] local 10.99.11.1 port 5201 connected to 10.99.37.240 port 33351
interval_len 1.000110 bytes_transferred 1245184
interval forces keep
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.19 MBytes 9.96 Mbits/sec
iperf3: error - unable to read from stream socket: Bad file descriptor
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Version output:
# /usr/bin/iperf3 --version
iperf 3.6 (cJSON 1.5.2)
Linux desbuild 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5 (2019-06-19) x86_64
Optional features available: CPU affinity setting, IPv6 flow label, SCTP, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication
problem persists with iperf3 3.7 built from source
Seeing this intermittently on:
iperf 3.6 (cJSON 1.5.2)
Linux 3efa7a0b256a 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64
Optional features available: CPU affinity setting, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing
I’m affected by this as well, please reopen this issue ticket if there isn’t a more suitable one. @bmah888
$ iperf3 -c 192.168.2.1 -p 4711
iperf3: error - control socket has closed unexpectedly
The issue happens immediately, it doesn’t seem like there is any data transferred at all.
$ iperf3 -c 192.168.2.1 -p 4711 -n 100M -v
iperf 3.7 (cJSON 1.5.2)
Linux ThinkPad-X1 5.4.0-58-generic #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, SCTP, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication
I’m still not able to reproduce this issue, but I believe y’all. Reopening.
Is it possible that those of you with this problem are having some firewall-related issue? That’s a fairly common issue with iperf3 in general.
Thanks for your reply. This is happening inside my local network at home where I have no firewall restrictions at all in place. Is there any way I could give you more information like by adding a very verbose debugging mode? Also, is it possible to query the version of the distant iperf3 server over the wire? I’m experimenting with two Ubuntu servers running iperf3 and there the issue happens only rarely. However, when using my router as server (which has an iperf3 server builtin as well, according to the docs), I always get this message very early on or even before the transfer starts at all.
Thanks for your reply. This is happening inside my local network at home where I have no firewall restrictions at all in place. Is there any way I could give you more information like by adding a very verbose debugging mode? Also, is it possible to query the version of the distant
iperf3server over the wire? I’m experimenting with two Ubuntu servers runningiperf3and there the issue happens only rarely. However, when using my router as server (which has aniperf3server builtin as well, according to the docs), I always get this message very early on or even before the transfer starts at all.
Well, «firewall» could also mean a host-based firewall like iptables or firewalld. But it looks like if you’re doing this between two Ubuntu servers and you get the issue some of the time but not always, that doesn’t really indicate a firewall/packet filtering issue.
There’s a --debug flag in iperf3. If you can run that on the iperf3 server side that might help us a bit. I wonder if having a packet trace (such as from tcpdump) might be helpful.
The server knows the version of the client but not vice versa.
This is a bit of a mystery to me because I have absolutely no idea what’s going wrong.
Er, wait a second @DL6ER . Are you trying this with SCTP (as per the original bug report) or is this with TCP (which is what your command lines indicate)?
I have an observation. I’m using v3.1.3, but this is also repeatable using 3.2 and 3.7.
Using the -A option is breaking iperf and resulting in «control socket has closed unexpectedly». Removing the -A option fixes it.
#iperf3 -c 192.168.1.19 -p 5010 -A 1,3
Connecting to host 192.168.1.19, port 5010
[ 4] local 192.168.1.9 port 54258 connected to 192.168.1.19 port 5010
iperf3: error — control socket has closed unexpectedly
iperf3 -c 192.168.1.19 -p 5010
Connecting to host 192.168.1.19, port 5010
[ 4] local 192.168.1.9 port 54262 connected to 192.168.1.19 port 5010
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 19.0 MBytes 159 Mbits/sec 0 423 KBytes
[ 4] 1.00-2.00 sec 20.1 MBytes 168 Mbits/sec 0 423 KBytes
[ 4] 2.00-3.00 sec 19.1 MBytes 160 Mbits/sec 0 427 KBytes
[ 4] 3.00-4.00 sec 20.6 MBytes 173 Mbits/sec 0 427 KBytes
[ 4] 4.00-5.00 sec 20.6 MBytes 173 Mbits/sec 0 427 KBytes
[ 4] 5.00-6.00 sec 20.9 MBytes 175 Mbits/sec 0 427 KBytes
[ 4] 6.00-7.00 sec 18.8 MBytes 158 Mbits/sec 0 427 KBytes
[ 4] 7.00-8.00 sec 21.0 MBytes 176 Mbits/sec 0 427 KBytes
[ 4] 8.00-9.00 sec 19.6 MBytes 165 Mbits/sec 0 427 KBytes
[ 4] 9.00-10.00 sec 19.6 MBytes 165 Mbits/sec 0 427 KBytes
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 199 MBytes 167 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 197 MBytes 165 Mbits/sec receiver
iperf Done.
Я заменил свой потребительский беспроводной маршрутизатор на коробку linux с четырех гигабитной сетевой картой PCIe и одной гигабитной сетевой платой на материнской плате (для глобальной сети). После включения переадресации IP, маскировки (через iptables ) и настройки подсетей на каждом из четырех интерфейсов локальной сети я провел несколько тестов скорости.
$ ip route default dev ppp0 scope link 10.0.0.0/16 dev enp3s0f0 proto kernel scope link src 10.0.0.1 10.64.0.0/16 dev enp3s0f1 proto kernel scope link src 10.64.0.1 10.192.0.0/16 dev enp4s0f1 proto kernel scope link src 10.192.0.1 aaa.bbb.ccc.ddd dev ppp0 proto kernel scope link src www.xxx.yyy.zzz
- От беспроводного устройства в одной из подсетей ЛВС до самого скоростного сервера в глобальной сети я получаю полные 40 Мбит / с / 5 Мбит / с, за которые я плачу своему провайдеру.
- От хоста маршрутизатора до хоста проводной локальной сети, использующего iperf3 я могу последовательно поддерживать скорость 930+ Мбит / с в течение нескольких минут.
- От проводного устройства в одной из подсетей ЛВС до проводного устройства в другой подсети ЛВС с использованием iperf3 я вначале получаю 80-95 Мбит / с в течение первых нескольких секунд, но оно быстро падает до нуля.
- От проводного устройства в одной из подсетей ЛВС до проводного устройства в другой подсети ЛВС, использующего iperf3 с целевой скоростью передачи данных 20 Мбит / с, я вижу аналогичные результаты (см. Обновление в конце), но может поддерживать около 10 Мбит / с
Connecting to host 10.0.0.2, port 5201 [ 5] local 10.192.128.3 port 35620 connected to 10.0.0.2 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 10.2 MBytes 85.9 Mbits/sec 0 73.5 KBytes [ 5] 1.00-2.00 sec 9.01 MBytes 75.6 Mbits/sec 0 82.0 KBytes [ 5] 2.00-3.00 sec 8.26 MBytes 69.3 Mbits/sec 0 79.2 KBytes [ 5] 3.00-4.00 sec 9.01 MBytes 75.6 Mbits/sec 0 73.5 KBytes [ 5] 4.00-5.00 sec 5.28 MBytes 44.3 Mbits/sec 1 1.41 KBytes [ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes [ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes [ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes [ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes ^C[ 5] 10.00-13.63 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes — — — — — — — — — — — — — — — — — — — — — — — — — [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-13.63 sec 41.8 MBytes 25.7 Mbits/sec 5 sender [ 5] 0.00-13.63 sec 0.00 Bytes 0.00 bits/sec receiver iperf3: interrupt — the client has terminated
Это наводит меня на мысль, что есть проблемы с пересылкой пакетов между подсетями. Сначала я убедился, что мои правила iptables минимальны:
15 вещей о которых жалеют после ремонта и как исправить ошибки
-t nat -A POSTROUTING -o ppp0 -j MASQUERADE # WAN connection is PPPoE and VLAN tagged -t filter -A FORWARD -o ppp0 -p tcp —tcp-flags SYN,RST SYN -j TCPMSS —clamp-mss-to-pmtu
Сбрасывая состояние iptables я вижу низкие количества пакетов для обоих правил.
Затем я проверил на потерю пакета. Кажется, существует небольшое, но постоянное количество потерь / повторных передач пакетов.
Nintendo Switch HOW TO FIX “A server communication error has occurred”
$ sudo netstat -s | egrep -i ‘retransmit|drop’ 498 outgoing packets dropped 25848 fast retransmits
Затем я подумал, что, возможно, там был буфер или очередь, которая заполнялась, и пакеты отбрасывались. Я рассчитал среднее значение пропускной способности и сравнил его с зарезервированной памятью.
$ sudo ping -f 10.0.0.2 -s $((1500-28)) PING 10.0.0.2 (10.0.0.2) 1472(1500) bytes of data. .^C — 10.0.0.2 ping statistics — 9036 packets transmitted, 9035 received, 0% packet loss, time 26512ms rtt min/avg/max/mdev = 1.742/2.817/12.057/0.758 ms, pipe 2, ipg/ewma 2.934/3.091 ms $ echo «1*(1024^3) * 0.003» | bc 3221225.472 $ cat /proc/sys/net/ipv4/tcp_mem 18396 24529 36792 $ getconf PAGESIZE 4096
Кажется, этого достаточно. Так что теперь я немного застрял. Я запустил tcpdump на клиенте iperf3 и вижу, что дела идут хорошо. Затем я вижу длительный (почти 250 мс) период молчания перед множеством повторных передач и повторных подтверждений.
Так как я могу получить достаточную скорость загрузки из глобальной сети, я не подозреваю, что встроенная сетевая карта виновата. Я ищу помощь в диагностике этого четырехядерного сетевого адаптера (подробности ниже) и, возможно, гигабитного коммутатора 2-го уровня (Netgear GS-108) и любой другой конфигурации ядра, которая может возникнуть. Я сомневаюсь, что это коммутатор, поскольку это никогда не было проблемой раньше, и я могу поддерживать скорость от обратной петли маршрутизатора до этой подсети. Похоже, это влияет только на производительность между подсетями.
$ iperf3 -b 20m -c 10.0.0.2 Connecting to host 10.0.0.2, port 5201 [ 5] local 10.192.128.3 port 36554 connected to 10.0.0.2 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 2.49 MBytes 20.9 Mbits/sec 0 158 KBytes [ 5] 1.00-2.00 sec 2.38 MBytes 19.9 Mbits/sec 0 150 KBytes [ 5] 2.00-3.00 sec 2.38 MBytes 19.9 Mbits/sec 1 133 KBytes [ 5] 3.00-4.00 sec 2.38 MBytes 19.9 Mbits/sec 0 73.5 KBytes [ 5] 4.00-5.00 sec 2.38 MBytes 19.9 Mbits/sec 0 70.7 KBytes [ 5] 5.00-6.00 sec 1.12 MBytes 9.44 Mbits/sec 2 1.41 KBytes [ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 2 1.41 KBytes [ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes [ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes iperf3: error — control socket has closed unexpectedly $ iperf3 -b 10m -c 10.0.0.2 Connecting to host 10.0.0.2, port 5201 [ 5] local 10.192.128.3 port 36564 connected to 10.0.0.2 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.24 MBytes 10.4 Mbits/sec 0 201 KBytes [ 5] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 0 118 KBytes [ 5] 2.00-3.00 sec 1.12 MBytes 9.44 Mbits/sec 0 127 KBytes [ 5] 3.00-4.00 sec 1.25 MBytes 10.5 Mbits/sec 0 107 KBytes [ 5] 4.00-5.00 sec 1.12 MBytes 9.44 Mbits/sec 0 110 KBytes [ 5] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec 0 90.0 KBytes [ 5] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec 0 87.2 KBytes [ 5] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec 0 81.6 KBytes [ 5] 8.00-9.00 sec 1.12 MBytes 9.44 Mbits/sec 0 78.8 KBytes [ 5] 9.00-10.00 sec 1.25 MBytes 10.5 Mbits/sec 0 112 KBytes — — — — — — — — — — — — — — — — — — — — — — — — — [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec 0 sender [ 5] 0.00-10.04 sec 12.0 MBytes 10.0 Mbits/sec receiver iperf Done.
Источник: poweruser.guru
Я новичок в использовании iperf но другие в моей компании используют его, но сегодня он действует странно.
С сервера:
iperf3 -s
что приводит к cmd с указанием:
Server listening on 5201 ,
От клиента:
iperf3 —format k —length 1 -t 10 -b 1K -i 1 -c *myservername*
что приводит к cmd показ на стороне клиента:
Connecting to host *myservername*, port 5201
iperft3: error — unable to create a new stream: Permission denied
А также cmd показывая на стороне сервера:
Accepted connection from *myip*, port 20756
iperft3: error — unable to create a new stream: Permission denied
Я не могу найти никакой документации по этому вопросу, поэтому я даже уверен, с чего начать. Кто-нибудь может указать мне правильное направление или что-нибудь, что я могу проверить?
tmwoods 28 июл ’16 в 18:07 2016-07-28 18:07
2016-07-28 18:07
2 ответа
Я оставлю вопрос здесь на случай, если кто-то столкнется с этой проблемой.
Запуск от имени администратора с обоих концов. Очень просто.
Источник: faultserver.ru
Зависание при нагрузочном тестировании.
Публикую один поток, с другой машины эмулирую 500 подписчиков.
После >400 соединений рост прекращается, с этого момента подключаться, публиковать потоки нельзя до перезапуска.
Исходящая скорость при зависании приметно 400-500 мбит/с
Пробовал типы инстансов AWS c5.2xlarge c5.9xlarge — результат один.
Результат почти всегда одинаков при ice_tcp_transport=true и без неё.
Пробовал различные размеры MTU, txqueuelen, ice_tcp_receive_buffer_size, ice_tcp_send_buffer_size, heap_size
Подключить ZGC пока не получилось-перестаёт работать веб-интерфейс и публикация потоков, детально не изучал.
Динамические порты и рабочие порты Flashphoner не пересекаются.
Разные версии java (jdk1.8.0, jdk-15.0.1)
Без CDN.
Логи во вложении.
Linux 4.14.200-116.320.amzn1.x86_64 x86_64
Flashphoner 5.2.798
Требуется ли еще какая-нибудь информация?
Attachments
171.8 KB Views: 237
Last edited: Oct 29, 2020
Max
Administrator
Staff member
Попробуйте все таки разобраться с запуском ZGC.
Рекомендованные JDK, с которыми протестирована работа сервера, это 12 или 14
(Инструкция по установке)
Для настройки ZGC выполните следующие действия:
1. Нужно закомментировать в файле wcs-core.properties следующие строки:
-XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDateStamps -XX:+PrintGCDetails
2. Поправить настройки для логов
-Xlog:gc*:/usr/local/FlashphonerWebCallServer/logs/gc-core.log -XX:ErrorFile=/usr/local/FlashphonerWebCallServer/logs/error%p.log
3. Выставить размер хипа не менее, чем 1/2 физической памяти сервера
### JVM OPTIONS ### -Xmx16g -Xms16g
4. Включаем ZGC
# ZGC -XX:+UnlockExperimentalVMOptions -XX:+UseZGC -XX:+UseLargePages -XX:ZPath=/hugepages
5. Настраиваем страницы памяти. Обязательно нужно рассчитать число страниц под размер выделенного хипа
(1,125*heap_size*1024)/2. Для -Xmx16g это число (1,125*16*1024)/2=9216
mkdir /hugepages echo «echo 9216 >/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages» >>/etc/rc.local echo «mount -t hugetlbfs -o uid=0 nodev /hugepages» >>/etc/rc.local chmod +x /etc/rc.d/rc.local systemctl enable rc-local.service systemctl restart rc-local.service
Затем перезапустите WCS и попробуйте запустить тестирование.
Если проблема будет сохраняться, по возможности, пожалуйста, предоставьте SSH доступ к вашему серверу с помощью приватной формы.
Ilya K.
Member
ZGC заработал с jdk-14, но проблема не ушла. Отправил данные в приватной форме.
Max
Administrator
Staff member
Подключится не удалось. Нужен приватный ключ вашего сервера.
Прислать ключ можно с помощью той же приватной формы
Ilya K.
Member
Отправил повторно.
Ilya K.
Member
Пока изменил адрес, нужен в работоспособном состоянии.
Ilya K.
Member
Сервер сейчас не занят. Посмотрите, пожалуйста.
Max
Administrator
Staff member
Провели нагрузочное тестирование на вашем сервере.
При более 400 входящих соединений увеличивается нагрузка на сервер и стандартного ice_timeout=15 сек становится недостаточно.
Для решения проблемы укажите в файле flashphoner.properties настройку
ice_timeout=120
Дополнительно, в том же файле укажите настройку, которая увеличивает время на прохождение теста:
wcs_activity_timer_timeout=86400000
Max
Administrator
Staff member
По результатам тестирования видно, что максимальное количество подписчиков в Вашем случае зависит не от производительности сервера – ЦП и оперативной памяти, а упирается в пропускную способность сети.
Заявленные в датацентре 1гбит/с могут действовать только в пределах этого датацентра (для локальной сети), поэтому нужно замерить реальную ширину канала до места размещения подписчиков, и, уже исходя из этой информации, делать расчеты по количеству возможных подключений.
Например, если в результате замеров получился канал 500мбит/с, то максимально можно подключить 500 пользователей.
Из расчета 1стрим – 1мбит/с
Измерить пропускную способность канала можно при помощи утилиты iperf. Эта программа выпущена под все основные операционные системы: Windows, MacOS, Ubuntu/Debian, CentOS. iperf в режиме сервера может быть установлена вместе с WCS, что позволяет тестировать канал целиком, от паблишера до зрителя.
Запуск iperf в режиме сервера:
iperf3 -s -p 5201
здесь 5201 — порт, на который iperf ожидает соединений от тестирующих клиентов
Запуск iperf в режиме клиента для тестирования отправки данных от клиента к серверу по TCP:
iperf3 -c test.flashphoner.com -p 5201
Здесь:
test.flashphoner.com — WCS сервер
5201 — порт iperf в режиме сервера
для тестирования приема данных от сервера по TCP:
iperf3 -c test.flashphoner.com -p 5201 -R
Здесь:
test.flashphoner.com — WCS сервер
5201 — порт iperf в режиме сервера
Ilya K.
Member
Здравствуйте.
Большое спасибо за информацию, точно пригодится в тестировании. В данном случае была ошибка в iptables, не хватало портов. Поправил — зависание ушло. Большие нагрузки буду давать позже.
Ilya K.
Member
Вкратце хотим добиться:
Максимально возможное количество соединений на edge-сервере при раздаче потоков, лимит которых будет упираться в лимит CPU/и или памяти.
Edge имеют одно dns имя. Все edge за classic load balancer+auto scaling group. Соответственно потоки клиенты будут получать по dns имени балансировщика. Тесты проводились с одним edge.
Окружение:
клиент (тестирующий) — c5.4xlarge (CPU-16 RAM-32):
-Xmx16G
-Xms16G
edge — c5.4xlarge (CPU-16 RAM-32):
-Xmx16G
-Xms16G
origin — с5.2xlarge (CPU-8 RAM-16):
-Xmx8G
-Xms8G
Везде:
Версия:
5.2.838 (при обновлениях на более новые появляется проблема подключения по ws. По возможности хотел бы вернуться к этому позже. пока проверяю на той, что работает)
сборщик мусора — ZGC
ice_tcp_transport=true
net.ipv4.ip_local_port_range = 54001 65000
media_port_from=31008
media_port_to=47000
wcs_agent_port_from=49001
wcs_agent_port_to=53999
WCS_FD_LIMIT=100000
webrtc_cc2_twcc=false
Есть другие настройки, скорее всего ненужные.
iptables -S:
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -s 127.0.0.0/8 -d 127.0.0.0/8 -i lo -j ACCEPT
-A INPUT -m state —state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m state —state INVALID -j DROP
-A INPUT -p tcp -m tcp ! —tcp-flags FIN,SYN,RST,ACK SYN -m state —state NEW -j DROP
-A INPUT -p icmp -m icmp —icmp-type 0 -j ACCEPT
-A INPUT -p icmp -m icmp —icmp-type 3 -j ACCEPT
-A INPUT -p icmp -m icmp —icmp-type 11 -j ACCEPT
-A INPUT -p icmp -m icmp —icmp-type 8 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 443 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 4431 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 9100 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 8888 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 8443 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 1935 -j ACCEPT
-A INPUT -p udp -m udp —dport 1935 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 554 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 8080 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 8081 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 8084 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 8082 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 8445 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 8444 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 9091 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 20000:53999 -j ACCEPT
-A INPUT -p udp -m udp —dport 20000:53999 -j ACCEPT
-A INPUT -j DROP
-A FORWARD -m state —state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -m state —state INVALID -j DROP
-A FORWARD -j DROP
-A OUTPUT -s 127.0.0.0/8 -d 127.0.0.0/8 -o lo -j ACCEPT
-A OUTPUT -o eth0 -j ACCEPT
-A OUTPUT -m state —state RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp -m tcp ! —tcp-flags FIN,SYN,RST,ACK SYN -m state —state NEW -j DROP
(некоторые порты от прежних настроек, будем убирать)
/etc/sysctl.conf сожержит:
fs.file-max = 100000
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.ipv4.tcp_max_syn_backlog = 10240
net.ipv4.tcp_congestion_control=htcp
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_fin_timeout = 15
net.core.somaxconn = 1024
net.core.netdev_max_backlog = 65536
net.core.optmem_max = 25165824
net.ipv4.tcp_mem = 65536 131072 262144
net.ipv4.udp_mem = 65536 131072 262144
net.core.rmem_default = 25165824
net.core.rmem_max = 25165824
net.ipv4.tcp_rmem = 20480 12582912 25165824
net.ipv4.udp_rmem_min = 16384
net.core.wmem_default = 25165824
net.core.wmem_max = 25165824
net.ipv4.tcp_wmem = 20480 12582912 25165824
net.ipv4.udp_wmem_min = 16384
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_tw_reuse = 1
(по умолчанию не было, выставляли исходя из рекомендаций по настройке сети., ни на что не повлияло)
Проверка канала напрямую по внешнему ip (от тестирующего к edge):
iperf3 -c xxxxxx -p 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 579 MBytes 4.85 Gbits/sec 48 718 KBytes
[ 4] 1.00-2.00 sec 539 MBytes 4.52 Gbits/sec 103 870 KBytes
[ 4] 2.00-3.00 sec 566 MBytes 4.75 Gbits/sec 116 981 KBytes
[ 4] 3.00-4.00 sec 538 MBytes 4.51 Gbits/sec 153 397 KBytes
^C[ 4] 4.00-4.62 sec 349 MBytes 4.72 Gbits/sec 0 922 KBytes
— — — — — — — — — — — — — — — — — — — — — — — — —
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-4.62 sec 2.51 GBytes 4.67 Gbits/sec 420 sender
[ 4] 0.00-4.62 sec 0.00 Bytes 0.00 bits/sec receiver
iperf3 -c xxxxxx -p 5201 -R
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 572 MBytes 4.79 Gbits/sec
[ 4] 1.00-2.00 sec 571 MBytes 4.79 Gbits/sec
[ 4] 2.00-3.00 sec 571 MBytes 4.79 Gbits/sec
[ 4] 3.00-4.00 sec 571 MBytes 4.79 Gbits/sec
^C[ 4] 4.00-4.61 sec 346 MBytes 4.78 Gbits/sec
— — — — — — — — — — — — — — — — — — — — — — — — —
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-4.61 sec 0.00 Bytes 0.00 bits/sec sender
[ 4] 0.00-4.61 sec 2.57 GBytes 4.79 Gbits/sec receiver
Edge находятся в AWS Classic load balancer+ Autoscaling group (настраивал согласно статье https://flashphoner.com/cdn-s-balan. rovaniem-na-baze-amazon-web-services/?lang=ru)
Тестирующий, origin, edge находятся в одной placement group. тип — cluster (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html)
Проверка канала по dns имени балансировщика edge (от тестирующего к edge):
iperf3 -c xxxxxx -p 5201:
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 371 MBytes 3.11 Gbits/sec 95 235 KBytes
[ 4] 1.00-2.00 sec 471 MBytes 3.95 Gbits/sec 115 505 KBytes
[ 4] 2.00-3.00 sec 516 MBytes 4.33 Gbits/sec 229 472 KBytes
[ 4] 3.00-4.00 sec 365 MBytes 3.06 Gbits/sec 274 404 KBytes
[ 4] 4.00-5.00 sec 455 MBytes 3.82 Gbits/sec 106 359 KBytes
[ 4] 5.00-6.00 sec 514 MBytes 4.31 Gbits/sec 148 626 KBytes
[ 4] 6.00-7.00 sec 469 MBytes 3.93 Gbits/sec 182 293 KBytes
[ 4] 7.00-8.00 sec 436 MBytes 3.66 Gbits/sec 266 221 KBytes
[ 4] 8.00-9.00 sec 402 MBytes 3.38 Gbits/sec 144 444 KBytes
[ 4] 9.00-10.00 sec 440 MBytes 3.69 Gbits/sec 157 614 KBytes
— — — — — — — — — — — — — — — — — — — — — — — — —
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 4.34 GBytes 3.72 Gbits/sec 1716 sender
[ 4] 0.00-10.00 sec 3.98 GBytes 3.42 Gbits/sec receiver
iperf3 -c xxxxxx -p 5201 -R:
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 100 KBytes 822 Kbits/sec
[ 4] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec
iperf3: error — control socket has closed unexpectedly
На тестирующей машине выставляется:
wcs_activity_timer_timeout=86400000
Тестировалось с помощью «pull streams» в количестве 1000 соединений. Указывался внешний IP непосредственно edge.
Исходный поток, опубликованный на origin имеет битрейт примерно от 1.5 до 2 мбит
При тестировании наблюдается следующее:
при достижении примерно 300 соединений (примерно 500 мбит/с) перестаёт расти скорость, так и остаётся — максимум 500 мбит/с. Количество соединений при этом увеличивается до 1000, как и было указано при запуске теста. При этом изображение начинает опаздывать, затем тормозить, если соединений больше, чем примерно 300.
Скорость замерялась утилитой nload.
Значительных нагрузок на CPU, память не наблюдается как на edge, так и на тестирующей машине.
Ставили много экспериментов, всё упирается в скорость раздачи в 500 мбит/с. Просьба помочь с настройками, дать рекомендации по настройке инфраструктуры, если допустили концептуальные ошибки.
Те же результаты наблюдались с типом машин edge, тестирующей- с5.2xlarge (CPU-8 RAM-16 -Xmx8G
-Xms8G).
Заранее спасибо.
Источник: forum.flashphoner.com
A few days ago, I reviewed a USB 3.0 to 2.5 Gbps Ethernet adapter based on Realtek RTL8156B chip in Ubuntu 20.04, and let’s say the reliability and performance were underwhelming. I got some recommendations like changing cables, the MTU size, etc…
Playing around with cables did no help, but one comment mentioned the cdc_ncm driver could be the issue, followed by another saying that updating to Linux kernel 5.14 should install the correct r8152 driver… So I just did that:
|
sudo apt install linux—oem—20.04d |
This upgraded Linux 5.13 (shipped with Ubuntu 20.04 + HWE) to Linux 5.14, but still no luck as the system kept using the cdc_ncm driver with a half-duplex link:
|
jaufranc@cnx—laptop—4:~$ inxi —n Network: Device—1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet driver: r8169 IF: enp2s0f1 state: down mac: 98:28:a6:0f:06:07 Device—2: Qualcomm Atheros QCA9377 802.11ac Wireless Network Adapter driver: ath10k_pci IF: wlp3s0 state: up mac: 70:c9:4e:b7:84:77 Device—3: Realtek USB 10/100/1G/2.5G LAN type: USB driver: cdc_ncm IF: enx1cbfced40321 state: up speed: 2500 Mbps duplex: half mac: 1c:bf:ce:d4:03:21 jaufranc@cnx—laptop—4:~$ uname —a Linux cnx—laptop—4 5.14.0—1022—oem #24-Ubuntu SMP Mon Jan 31 16:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
But then I thought I may have to use udev rules to prevent loading the cdc_ncm driver, and there’s indeed 50-usb-realtek-net.rules in r8152 driver to do just that. So I copied the file in /etc/udev/rules.d/ folder. Since I did not want to reboot, I unloaded the modules I did not need, and restart udev to try it out:
|
sudo rmmod cdc_mbim sudo rmmod cdc_ncm service udev restart |
Let’s see…
|
sudo inxi —n Network: Device—1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet driver: r8169 IF: enp2s0f1 state: down mac: 98:28:a6:0f:06:07 Device—2: Qualcomm Atheros QCA9377 802.11ac Wireless Network Adapter driver: ath10k_pci IF: wlp3s0 state: up mac: 70:c9:4e:b7:84:77 Device—3: Realtek USB 10/100/1G/2.5G LAN type: USB driver: r8152 IF: enx1cbfced40321 state: up speed: 2500 Mbps duplex: full mac: 1c:bf:ce:d4:03:21 |
Great! It’s now using r8152 driver and we’ve got a full-duplex connection.
Let’s go through all our tests again to compare the results.
iperf2
upload
|
iperf —t 60 —c 192.168.31.12 —————————————————————————————— Client connecting to 192.168.31.12, TCP port 5001 TCP window size: 1.40 MByte (default) —————————————————————————————— [ 3] local 192.168.31.166 port 41266 connected with 192.168.31.12 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0—60.0 sec 16.4 GBytes 2.35 Gbits/sec |
download:
|
iperf —t 60 —c 192.168.31.166 —————————————————————————————— Client connecting to 192.168.31.166, TCP port 5001 TCP window size: 901 KByte (default) —————————————————————————————— [ 3] local 192.168.31.12 port 37188 connected with 192.168.31.166 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0—60.0 sec 10.8 GBytes 1.55 Gbits/sec |
There’s an improvement to the download speed (was 600 Mbps with cdc_ncm driver), but still not quite close to 2.3 Gbps.
Let’s try full-duplex for fun:
|
Client connecting to 192.168.31.166, TCP port 5001 TCP window size: 799 KByte (default) —————————————————————————————— [ 4] local 192.168.31.12 port 5001 connected with 192.168.31.166 port 41290 [ 6] local 192.168.31.12 port 37194 connected with 192.168.31.166 port 5001 [ 6] 0.0—60.1 sec 8.06 GBytes 1.15 Gbits/sec [ 4] 0.0—60.4 sec 16.3 GBytes 2.32 Gbits/sec |
That’s actually not too bad.
upload:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
iperf3 —t 60 —c 192.168.31.12 —i 5 Connecting to host 192.168.31.12, port 5201 [ 5] local 192.168.31.166 port 32848 connected to 192.168.31.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00—5.00 sec 1.37 GBytes 2.36 Gbits/sec 0 847 KBytes [ 5] 5.00—10.00 sec 1.37 GBytes 2.35 Gbits/sec 0 889 KBytes [ 5] 10.00—15.00 sec 1.37 GBytes 2.35 Gbits/sec 0 1.14 MBytes [ 5] 15.00—20.00 sec 1.37 GBytes 2.35 Gbits/sec 0 1.14 MBytes [ 5] 20.00—25.00 sec 1.37 GBytes 2.35 Gbits/sec 0 1.14 MBytes [ 5] 25.00—30.00 sec 1.37 GBytes 2.35 Gbits/sec 0 1.14 MBytes [ 5] 30.00—35.00 sec 1.37 GBytes 2.35 Gbits/sec 0 1.73 MBytes [ 5] 35.00—40.00 sec 1.37 GBytes 2.35 Gbits/sec 0 1.73 MBytes [ 5] 40.00—45.00 sec 1.37 GBytes 2.35 Gbits/sec 0 3.92 MBytes [ 5] 45.00—50.00 sec 1.37 GBytes 2.35 Gbits/sec 0 3.92 MBytes [ 5] 50.00—55.00 sec 1.37 GBytes 2.35 Gbits/sec 0 3.92 MBytes [ 5] 55.00—60.00 sec 1.37 GBytes 2.35 Gbits/sec 0 3.92 MBytes — — — — — — — — — — — — — — — — — — — — — — — — — [ ID] Interval Transfer Bitrate Retr [ 5] 0.00—60.00 sec 16.4 GBytes 2.35 Gbits/sec 0 sender [ 5] 0.00—60.05 sec 16.4 GBytes 2.35 Gbits/sec receiver iperf Done. |
download:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
iperf3 —t 60 —c 192.168.31.166 —i 5 Connecting to host 192.168.31.166, port 5201 [ 5] local 192.168.31.12 port 53112 connected to 192.168.31.166 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00—5.00 sec 802 MBytes 1.35 Gbits/sec 641 222 KBytes [ 5] 5.00—10.00 sec 856 MBytes 1.44 Gbits/sec 618 83.4 KBytes [ 5] 10.00—15.00 sec 852 MBytes 1.43 Gbits/sec 583 87.7 KBytes [ 5] 15.00—20.00 sec 843 MBytes 1.41 Gbits/sec 592 168 KBytes [ 5] 20.00—25.00 sec 831 MBytes 1.39 Gbits/sec 642 91.9 KBytes [ 5] 25.00—30.00 sec 810 MBytes 1.36 Gbits/sec 666 97.6 KBytes [ 5] 30.00—35.00 sec 831 MBytes 1.39 Gbits/sec 590 123 KBytes [ 5] 35.00—40.00 sec 827 MBytes 1.39 Gbits/sec 652 298 KBytes [ 5] 40.00—45.00 sec 843 MBytes 1.41 Gbits/sec 605 93.3 KBytes [ 5] 45.00—50.00 sec 844 MBytes 1.42 Gbits/sec 635 96.2 KBytes [ 5] 50.00—55.00 sec 862 MBytes 1.45 Gbits/sec 565 84.8 KBytes [ 5] 55.00—60.00 sec 858 MBytes 1.44 Gbits/sec 583 82.0 KBytes — — — — — — — — — — — — — — — — — — — — — — — — — [ ID] Interval Transfer Bitrate Retr [ 5] 0.00—60.00 sec 9.82 GBytes 1.41 Gbits/sec 7372 sender [ 5] 0.00—60.00 sec 9.82 GBytes 1.41 Gbits/sec receiver iperf Done. |
About the same as iperf2.
iperf3 did not have support for full-duplex for many years, but version 3.7 reintroduced the feature. So let’s give it a go:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
iperf3 —t 60 —c 192.168.31.12 —bidir —i 5 Connecting to host 192.168.31.12, port 5201 [ 5] local 192.168.31.166 port 32858 connected to 192.168.31.12 port 5201 [ 7] local 192.168.31.166 port 32860 connected to 192.168.31.12 port 5201 [ ID][Role] Interval Transfer Bitrate Retr Cwnd [ 5][TX—C] 0.00—5.00 sec 1.36 GBytes 2.34 Gbits/sec 0 1.41 MBytes [ 7][RX—C] 0.00—5.00 sec 643 MBytes 1.08 Gbits/sec [ 5][TX—C] 5.00—10.00 sec 1.36 GBytes 2.34 Gbits/sec 0 1.48 MBytes [ 7][RX—C] 5.00—10.00 sec 673 MBytes 1.13 Gbits/sec [ 5][TX—C] 10.00—15.00 sec 1.36 GBytes 2.34 Gbits/sec 0 1.78 MBytes [ 7][RX—C] 10.00—15.00 sec 690 MBytes 1.16 Gbits/sec [ 5][TX—C] 15.00—20.00 sec 1.36 GBytes 2.34 Gbits/sec 0 1.78 MBytes [ 7][RX—C] 15.00—20.00 sec 695 MBytes 1.17 Gbits/sec [ 5][TX—C] 20.00—25.00 sec 1.36 GBytes 2.34 Gbits/sec 0 1.78 MBytes [ 7][RX—C] 20.00—25.00 sec 703 MBytes 1.18 Gbits/sec [ 5][TX—C] 25.00—30.00 sec 1.36 GBytes 2.34 Gbits/sec 0 1.78 MBytes [ 7][RX—C] 25.00—30.00 sec 704 MBytes 1.18 Gbits/sec [ 5][TX—C] 30.00—35.00 sec 1.36 GBytes 2.34 Gbits/sec 0 1.78 MBytes [ 7][RX—C] 30.00—35.00 sec 711 MBytes 1.19 Gbits/sec [ 5][TX—C] 35.00—40.00 sec 1.36 GBytes 2.34 Gbits/sec 0 1.78 MBytes [ 7][RX—C] 35.00—40.00 sec 697 MBytes 1.17 Gbits/sec [ 5][TX—C] 40.00—45.00 sec 28.8 MBytes 48.2 Mbits/sec 4 1.41 KBytes [ 7][RX—C] 40.00—45.00 sec 15.0 MBytes 25.2 Mbits/sec [ 5][TX—C] 45.00—50.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes [ 7][RX—C] 45.00—50.00 sec 0.00 Bytes 0.00 bits/sec [ 5][TX—C] 50.00—55.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes [ 7][RX—C] 50.00—55.00 sec 0.00 Bytes 0.00 bits/sec iperf3: error — control socket has closed unexpectedly |
Oops! What happened? There are some error messages in the kernel log as well.
|
[18424.279351] r8152 2—1:1.0 enx1cbfced40321: Tx status —71 [18424.287497] r8152 2—1:1.0 enx1cbfced40321: Tx status —71 [18424.295735] r8152 2—1:1.0 enx1cbfced40321: Tx status —71 [18424.303858] r8152 2—1:1.0 enx1cbfced40321: Tx status —71 [18430.885965] net_ratelimit: 107 callbacks suppressed [18430.885975] r8152 2—1:1.0 enx1cbfced40321: Tx status —71 [18431.251643] r8152 2—1:1.0 enx1cbfced40321: Tx status —71 [18431.909792] r8152 2—1:1.0 enx1cbfced40321: Tx status —71 [18437.797786] r8152 2—1:1.0 enx1cbfced40321: Tx status —71 |
I’m not the only one to have this problem, and that’s an open issue in r8156 driver’s Github repo. Here’s the answer from the developer for reference:
I suspect a problem on the ethernet adapter side, as there are many reports of it working with the DS918+. (Eg. design of power line, overheat, etc.)
https://github.com/bb-qq/r8152/wiki/CompatibilityCould you try another vendor? Also using a USB Hub with an external power supply might improve the situation.
At this point, the Ethernet did not work at all, and I had to unplug and reinsert the USB dongle. The second time “worked”.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
iperf3 —t 60 —c 192.168.31.12 —bidir —i 5 Connecting to host 192.168.31.12, port 5201 [ 5] local 192.168.31.166 port 32872 connected to 192.168.31.12 port 5201 [ 7] local 192.168.31.166 port 32874 connected to 192.168.31.12 port 5201 [ ID][Role] Interval Transfer Bitrate Retr Cwnd [ 5][TX—C] 0.00—5.00 sec 1.36 GBytes 2.34 Gbits/sec 0 1.59 MBytes [ 7][RX—C] 0.00—5.00 sec 515 MBytes 864 Mbits/sec [ 5][TX—C] 5.00—10.00 sec 1.36 GBytes 2.34 Gbits/sec 0 1.75 MBytes [ 7][RX—C] 5.00—10.00 sec 489 MBytes 820 Mbits/sec [ 5][TX—C] 10.00—15.00 sec 1.36 GBytes 2.34 Gbits/sec 0 1.75 MBytes [ 7][RX—C] 10.00—15.00 sec 530 MBytes 889 Mbits/sec [ 5][TX—C] 15.00—20.00 sec 1.36 GBytes 2.34 Gbits/sec 0 1.75 MBytes [ 7][RX—C] 15.00—20.00 sec 564 MBytes 947 Mbits/sec [ 5][TX—C] 20.00—25.00 sec 1.36 GBytes 2.34 Gbits/sec 0 1.75 MBytes [ 7][RX—C] 20.00—25.00 sec 560 MBytes 940 Mbits/sec [ 5][TX—C] 25.00—30.00 sec 1.36 GBytes 2.34 Gbits/sec 0 2.63 MBytes [ 7][RX—C] 25.00—30.00 sec 578 MBytes 970 Mbits/sec [ 5][TX—C] 30.00—35.00 sec 1.36 GBytes 2.34 Gbits/sec 0 2.63 MBytes [ 7][RX—C] 30.00—35.00 sec 561 MBytes 942 Mbits/sec [ 5][TX—C] 35.00—40.00 sec 1.36 GBytes 2.34 Gbits/sec 0 2.63 MBytes [ 7][RX—C] 35.00—40.00 sec 572 MBytes 960 Mbits/sec [ 5][TX—C] 40.00—45.00 sec 1.36 GBytes 2.34 Gbits/sec 0 2.63 MBytes [ 7][RX—C] 40.00—45.00 sec 570 MBytes 956 Mbits/sec [ 5][TX—C] 45.00—50.00 sec 1.36 GBytes 2.34 Gbits/sec 0 2.63 MBytes [ 7][RX—C] 45.00—50.00 sec 572 MBytes 960 Mbits/sec [ 5][TX—C] 50.00—55.00 sec 1.36 GBytes 2.34 Gbits/sec 0 2.63 MBytes [ 7][RX—C] 50.00—55.00 sec 570 MBytes 957 Mbits/sec [ 5][TX—C] 55.00—60.00 sec 1.36 GBytes 2.34 Gbits/sec 0 2.63 MBytes [ 7][RX—C] 55.00—60.00 sec 571 MBytes 958 Mbits/sec — — — — — — — — — — — — — — — — — — — — — — — — — [ ID][Role] Interval Transfer Bitrate Retr [ 5][TX—C] 0.00—60.00 sec 16.3 GBytes 2.34 Gbits/sec 0 sender [ 5][TX—C] 0.00—60.05 sec 16.3 GBytes 2.34 Gbits/sec receiver [ 7][RX—C] 0.00—60.00 sec 6.50 GBytes 931 Mbits/sec 58 sender [ 7][RX—C] 0.00—60.05 sec 6.50 GBytes 929 Mbits/sec receiver iperf Done. |
There are still retransmissions on the Rx side which could help explain the lower speed.
SAMBA
From my laptop with the RTL8156B dongle, and SATA SSD to UP Xtreme i11 mini PC with a 2.5GbE port, and a MINIX USB-C dock with a 480GB SSD.
Around 930 Mbps with r8152 driver against 750 Mbps with cdc_ncm driver.
Now from the mini PC to the laptop (aka download)…
It’s slower as expected at 837 Mbps, but better than the under 500 Mbps I got with the cdc_ncm driver.
scp
laptop to mini PC:
|
time scp Libero_SoC_v2021.2_lin.bin devkit@192.168.31.12:/home/devkit/NEO_Storage devkit@192.168.31.12‘s password: Libero_SoC_v2021.2_lin.bin 100% 10GB 115.0MB/s 01:32 real 1m34.981s user 0m55.750s sys 0m42.668s |
mini PC to laptop:
|
time scp devkit@192.168.31.12:/home/devkit/NEO_Storage/Libero_SoC_v2021.2_lin.bin . devkit@192.168.31.12‘s password: Libero_SoC_v2021.2_lin.bin 100% 10GB 111.7MB/s 01:35 real 1m36.896s user 0m56.926s sys 0m55.330s |
It’s almost the same speed between download and upload with scp which is odd. The bottleneck here looks to be my SATA SSD:
|
iozone —e —I —a —s 1000M —r 16384k —i 0 —i 1 Iozone: Performance Test of File I/O Version $Revision: 3.489 $ Compiled for 64 bit mode. Build: linux—AMD64 random random bkwd record stride kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 1024000 16384 141794 139550 149591 145020 |
Read is limited by around 145 MB/s, while write is around 140 MB/s.
The SSD used in MINIX NEO Storage Plus USB-C dock is quite faster with, last time I tested, 379MB/s for sequential reads, and 240+MB/s for sequential writes.
So let’s see the speed we can get throwing away the data to /dev/null
download to laptop:
|
time scp —c aes128—ctr devkit@192.168.31.12:/home/devkit/NEO_Storage/Libero_SoC_v2021.2_lin.bin /dev/null devkit@192.168.31.12‘s password: Libero_SoC_v2021.2_lin.bin 100% 10GB 133.0MB/s 01:20 real 1m21.880s user 0m18.931s sys 0m33.838s |
upload from laptop:
|
time scp —c aes128—ctr Libero_SoC_v2021.2_lin.bin devkit@192.168.31.12:/dev/null devkit@192.168.31.12‘s password: Libero_SoC_v2021.2_lin.bin 100% 10GB 239.9MB/s 00:44 real 0m46.094s user 0m17.174s sys 0m36.793s |
So that’s more like it for the upload at least. I still have a problem with the download speed, but the performance was still greatly improved with the r8152 driver.
Testing with NanoPi R4S
While it’s much better, it’s not optimal. I have a NanoPi R4S router with two USB 3.0 ports, so I first tried it using the latest OpenWrt (FriendlyWrt) 21.02 image with Linux 5.15:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
root@FriendlyWrt:~# iperf3 -t 60 -c 192.168.2.207 —bidir -i 5 Connecting to host 192.168.2.207, port 5201 [ 5] local 192.168.2.161 port 21836 connected to 192.168.2.207 port 5201 [ 7] local 192.168.2.161 port 21838 connected to 192.168.2.207 port 5201 [ ID][Role] Interval Transfer Bitrate Retr Cwnd [ 5][TX—C] 0.00—5.00 sec 348 MBytes 583 Mbits/sec 4 413 KBytes [ 7][RX—C] 0.00—5.00 sec 220 MBytes 369 Mbits/sec [ 5][TX—C] 5.00—10.00 sec 480 MBytes 805 Mbits/sec 5 684 KBytes [ 7][RX—C] 5.00—10.00 sec 218 MBytes 366 Mbits/sec [ 5][TX—C] 10.00—15.00 sec 574 MBytes 963 Mbits/sec 47 557 KBytes [ 7][RX—C] 10.00—15.00 sec 183 MBytes 307 Mbits/sec [ 5][TX—C] 15.00—20.00 sec 481 MBytes 807 Mbits/sec 3 699 KBytes [ 7][RX—C] 15.00—20.00 sec 179 MBytes 301 Mbits/sec [ 5][TX—C] 20.00—25.00 sec 464 MBytes 779 Mbits/sec 18 701 KBytes [ 7][RX—C] 20.00—25.00 sec 214 MBytes 359 Mbits/sec [ 5][TX—C] 25.00—30.00 sec 549 MBytes 920 Mbits/sec 26 580 KBytes [ 7][RX—C] 25.00—30.00 sec 178 MBytes 298 Mbits/sec [ 5][TX—C] 30.00—35.00 sec 472 MBytes 792 Mbits/sec 3 526 KBytes [ 7][RX—C] 30.00—35.00 sec 207 MBytes 347 Mbits/sec [ 5][TX—C] 35.00—40.00 sec 465 MBytes 781 Mbits/sec 15 410 KBytes [ 7][RX—C] 35.00—40.00 sec 195 MBytes 326 Mbits/sec [ 5][TX—C] 40.00—45.00 sec 385 MBytes 645 Mbits/sec 0 376 KBytes [ 7][RX—C] 40.00—45.00 sec 217 MBytes 364 Mbits/sec [ 5][TX—C] 45.00—50.00 sec 497 MBytes 833 Mbits/sec 9 478 KBytes [ 7][RX—C] 45.00—50.00 sec 201 MBytes 338 Mbits/sec [ 5][TX—C] 50.00—55.00 sec 434 MBytes 728 Mbits/sec 0 543 KBytes [ 7][RX—C] 50.00—55.00 sec 208 MBytes 349 Mbits/sec [ 5][TX—C] 55.00—60.00 sec 451 MBytes 756 Mbits/sec 6 823 KBytes [ 7][RX—C] 55.00—60.00 sec 220 MBytes 370 Mbits/sec — — — — — — — — — — — — — — — — — — — — — — — — — [ ID][Role] Interval Transfer Bitrate Retr [ 5][TX—C] 0.00—60.00 sec 5.47 GBytes 783 Mbits/sec 136 sender [ 5][TX—C] 0.00—60.01 sec 5.47 GBytes 783 Mbits/sec receiver [ 7][RX—C] 0.00—60.00 sec 2.39 GBytes 342 Mbits/sec 94 sender [ 7][RX—C] 0.00—60.01 sec 2.38 GBytes 341 Mbits/sec receiver iperf Done. |
It’s really ugly with lots of retransmissions on both sides.
|
[ 276.268986] usb 8—1: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=31.00 [ 276.269798] usb 8—1: New USB device strings: Mfr=1, Product=2, SerialNumber=6 [ 276.270471] usb 8—1: Product: USB 10/100/1G/2.5G LAN [ 276.272112] usb 8—1: Manufacturer: Realtek [ 276.272519] usb 8—1: SerialNumber: 0013000000 [ 276.359625] cdc_ncm 8—1:2.0: MAC—Address: 1c:bf:ce:d4:03:21 [ 276.360178] cdc_ncm 8—1:2.0: setting rx_max = 16384 [ 276.360758] cdc_ncm 8—1:2.0: setting tx_max = 16384 [ 276.362948] cdc_ncm 8—1:2.0 eth2: register ‘cdc_ncm’ at usb—xhci—hcd.1.auto—1, CDC NCM, 1c:bf:ce:d4:03:21 [ 459.919597] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready |
Looking at the kernel log, our RTL8156B USB dongle is using that CDC NCM driver again, just like in Ubuntu… There’s an r8152 driver too, but whatever module I remove in /etc/modules.d either the CDC NCM is loaded or the eth2 interface does not show up at all. So I’ve switched to the Ubuntu 20.04-based FriendlyCore OS, also featuring Linux 5.15, that will be closer to the setup on my laptop.
As one might have expected, the RTL8156B adapter uses the CDC NCM driver by default in Ubuntu:
|
[ 682.701529] usb 8—1: new SuperSpeed USB device number 3 using xhci—hcd [ 682.727125] usb 8—1: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=31.00 [ 682.727163] usb 8—1: New USB device strings: Mfr=1, Product=2, SerialNumber=6 [ 682.727179] usb 8—1: Product: USB 10/100/1G/2.5G LAN [ 682.727191] usb 8—1: Manufacturer: Realtek [ 682.727203] usb 8—1: SerialNumber: 0013000000 [ 682.806350] cdc_ncm 8—1:2.0: MAC—Address: 1c:bf:ce:d4:03:21 [ 682.806387] cdc_ncm 8—1:2.0: setting rx_max = 16384 [ 682.806561] cdc_ncm 8—1:2.0: setting tx_max = 16384 [ 682.807963] cdc_ncm 8—1:2.0 eth1: register ‘cdc_ncm’ at usb—xhci—hcd.1.auto—1, CDC NCM, 1c:bf:ce:d4:03:21 [ 682.834550] cdc_ncm 8—1:2.0 enx1cbfced40321: renamed from eth1 |
eth1 is not showing up with ifconfig, so I just went ahead and change udev rules to use the r8152 driver…
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
[ 882.081445] usb 7—1: new high—speed USB device number 3 using xhci—hcd [ 882.209358] usb 7—1: Device not responding to setup address. [ 882.417542] usb 7—1: Device not responding to setup address. [ 882.625432] usb 7—1: device not accepting address 3, error —71 [ 886.597570] usb 8—1: new SuperSpeed USB device number 4 using xhci—hcd [ 886.619484] usb 8—1: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=31.00 [ 886.619534] usb 8—1: New USB device strings: Mfr=1, Product=2, SerialNumber=6 [ 886.619556] usb 8—1: Product: USB 10/100/1G/2.5G LAN [ 886.619574] usb 8—1: Manufacturer: Realtek [ 886.619591] usb 8—1: SerialNumber: 0013000000 [ 886.911033] usb 8—1: reset SuperSpeed USB device number 4 using xhci—hcd [ 886.960598] r8152 8—1:1.0: Direct firmware load for rtl_nic/rtl8156b—2.fw failed with error —2 [ 886.960631] r8152 8—1:1.0: Falling back to sysfs fallback for: rtl_nic/rtl8156b—2.fw [ 947.165730] r8152 8—1:1.0: unable to load firmware patch rtl_nic/rtl8156b—2.fw (—110) [ 947.210940] r8152 8—1:1.0 (unnamed net_device) (uninitialized): netif_napi_add() called with weight 256 [ 947.237480] r8152 8—1:1.0 eth1: v1.12.11 [ 947.254755] usbcore: registered new interface driver cdc_ncm [ 947.257112] usbcore: registered new interface driver cdc_mbim [ 947.290865] r8152 8—1:1.0 enx1cbfced40321: renamed from eth1 |
It’s getting depressing. Let’s update the system first.
|
sudo apt update sudo apt install python3—pip sudo pip3 install apt—mirror—updater apt—mirror—updater —c http://ftp.tu-chemnitz.de/pub/linux/ubuntu-ports sudo apt dist—upgrade |
I followed all those steps since the FriendlyCore image relies on servers from China, which are very slow from where I have (just running apt update can take 15 minutes), so the update was possibly faster that way, although it still took a couple of hours! (See Changing Ubuntu Apt Mirror from the Command Line for details).
It did not help with the firmware issue. So instead I search for that rtl_nic/rtl8156b-2.fw file. It is in the firmware-realtek Debian package, or linux-firmware Ubuntu Impish package.
I downloaded the latter, extracted rtl8156b-2.fw and copied it to /etc/firmware/rtl_nic. It now works:
|
[ 2172.098871] usb 8—1: new SuperSpeed USB device number 4 using xhci—hcd [ 2172.120681] usb 8—1: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=31.00 [ 2172.120731] usb 8—1: New USB device strings: Mfr=1, Product=2, SerialNumber=6 [ 2172.120754] usb 8—1: Product: USB 10/100/1G/2.5G LAN [ 2172.120771] usb 8—1: Manufacturer: Realtek [ 2172.120788] usb 8—1: SerialNumber: 0013000000 [ 2172.183460] cdc_ncm 8—1:2.0: MAC—Address: 1c:bf:ce:d4:03:21 [ 2172.183494] cdc_ncm 8—1:2.0: setting rx_max = 16384 [ 2172.183620] cdc_ncm 8—1:2.0: setting tx_max = 16384 [ 2172.184904] cdc_ncm 8—1:2.0 eth1: register ‘cdc_ncm’ at usb—xhci—hcd.1.auto—1, CDC NCM, 1c:bf:ce:d4:03:21 [ 2172.189439] cdc_ncm 8—1:2.0 eth1: unregister ‘cdc_ncm’ usb—xhci—hcd.1.auto—1, CDC NCM [ 2172.451015] usb 8—1: reset SuperSpeed USB device number 4 using xhci—hcd [ 2172.535166] r8152 8—1:1.0: load rtl8156b—2 v1 04/15/21 successfully [ 2172.598459] r8152 8—1:1.0 eth1: v1.12.11 |
Somehow the interface is still not up, and adding it manually to /etc/network/interfaces.d, does not work.
Back to the Ubuntu laptop with rtl8156b-2.fw firmware
So instead, I went back to my laptop, and copied the firmware file to /lib/firmware/rtl_nic directory. The result was the same as in NanoPi R4S:
|
[23050.245495] usb 2—1: new SuperSpeed USB device number 3 using xhci_hcd [23050.266025] usb 2—1: New USB device found, idVendor=0bda, idProduct=8156, bcdDevice=31.00 [23050.266038] usb 2—1: New USB device strings: Mfr=1, Product=2, SerialNumber=6 [23050.266043] usb 2—1: Product: USB 10/100/1G/2.5G LAN [23050.266047] usb 2—1: Manufacturer: Realtek [23050.266050] usb 2—1: SerialNumber: 0013000000 [23050.325714] cdc_ncm 2—1:2.0: MAC—Address: 1c:bf:ce:d4:03:21 [23050.325720] cdc_ncm 2—1:2.0: setting rx_max = 16384 [23050.325762] cdc_ncm 2—1:2.0: setting tx_max = 16384 [23050.326199] cdc_ncm 2—1:2.0 eth0: register ‘cdc_ncm’ at usb—0000:04:00.3—1, CDC NCM, 1c:bf:ce:d4:03:21 [23050.334723] cdc_ncm 2—1:2.0 eth0: unregister ‘cdc_ncm’ usb—0000:04:00.3—1, CDC NCM [23050.559779] usb 2—1: reset SuperSpeed USB device number 3 using xhci_hcd [23050.617459] r8152 2—1:1.0: load rtl8156b—2 v1 04/15/21 successfully [23050.654344] r8152 2—1:1.0 eth0: v1.12.11 |
Let’s try iperf3 upload:
|
iperf3 —t 30 —c 192.168.31.12 —i 5 Connecting to host 192.168.31.12, port 5201 [ 5] local 192.168.31.166 port 50222 connected to 192.168.31.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00—5.00 sec 1.37 GBytes 2.36 Gbits/sec 0 609 KBytes [ 5] 5.00—10.00 sec 1.37 GBytes 2.35 Gbits/sec 0 691 KBytes [ 5] 10.00—15.00 sec 1.37 GBytes 2.35 Gbits/sec 0 1.35 MBytes [ 5] 15.00—20.00 sec 1.37 GBytes 2.35 Gbits/sec 0 1.35 MBytes [ 5] 20.00—25.00 sec 1.37 GBytes 2.35 Gbits/sec 0 1.35 MBytes [ 5] 25.00—30.00 sec 1.37 GBytes 2.35 Gbits/sec 0 1.35 MBytes — — — — — — — — — — — — — — — — — — — — — — — — — [ ID] Interval Transfer Bitrate Retr [ 5] 0.00—30.00 sec 8.22 GBytes 2.35 Gbits/sec 0 sender [ 5] 0.00—30.04 sec 8.22 GBytes 2.35 Gbits/sec receiver iperf Done. |
Same as before, so no regression. Now for the iperf3 download:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
iperf3 —t 30 —c 192.168.31.12 —i 5 —R Connecting to host 192.168.31.12, port 5201 Reverse mode, remote host 192.168.31.12 is sending [ 5] local 192.168.31.166 port 50226 connected to 192.168.31.12 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00—5.00 sec 969 MBytes 1.63 Gbits/sec [ 5] 5.00—10.00 sec 990 MBytes 1.66 Gbits/sec [ 5] 10.00—15.00 sec 982 MBytes 1.65 Gbits/sec [ 5] 15.00—20.00 sec 984 MBytes 1.65 Gbits/sec [ 5] 20.00—25.00 sec 1012 MBytes 1.70 Gbits/sec [ 5] 25.00—30.00 sec 1007 MBytes 1.69 Gbits/sec — — — — — — — — — — — — — — — — — — — — — — — — — [ ID] Interval Transfer Bitrate Retr [ 5] 0.00—30.02 sec 5.81 GBytes 1.66 Gbits/sec 3181 sender [ 5] 0.00—30.00 sec 5.80 GBytes 1.66 Gbits/sec receiver iperf Done. |
It’s getting better, although we are not there yet, and the retransmissions are through the roof. I’m really tired now… Let me know if you have other ideas.
Realtek RTL8156BG USB 3.0 to 2.5GbE dongle to the rescue.
[Update: Realtek sent me one of their USB 3.0 to 2.5GbE adapters and it does not have the same performance issue
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
$ iperf3 —t 60 —c 192.168.31.12 —bidir —i 5 Connecting to host 192.168.31.12, port 5201 [ 5] local 192.168.31.85 port 47164 connected to 192.168.31.12 port 5201 [ 7] local 192.168.31.85 port 47168 connected to 192.168.31.12 port 5201 [ ID][Role] Interval Transfer Bitrate Retr Cwnd [ 5][TX—C] 0.00—5.00 sec 1.35 GBytes 2.31 Gbits/sec 0 1.54 MBytes [ 7][RX—C] 0.00—5.00 sec 1.25 GBytes 2.14 Gbits/sec [ 5][TX—C] 5.00—10.00 sec 1.37 GBytes 2.35 Gbits/sec 0 1.54 MBytes [ 7][RX—C] 5.00—10.00 sec 1.36 GBytes 2.34 Gbits/sec [ 5][TX—C] 10.00—15.00 sec 1.36 GBytes 2.34 Gbits/sec 0 1.54 MBytes [ 7][RX—C] 10.00—15.00 sec 1.34 GBytes 2.31 Gbits/sec [ 5][TX—C] 15.00—20.00 sec 1.36 GBytes 2.34 Gbits/sec 0 2.34 MBytes [ 7][RX—C] 15.00—20.00 sec 1.32 GBytes 2.28 Gbits/sec [ 5][TX—C] 20.00—25.00 sec 1.36 GBytes 2.34 Gbits/sec 0 2.34 MBytes [ 7][RX—C] 20.00—25.00 sec 1.36 GBytes 2.33 Gbits/sec [ 5][TX—C] 25.00—30.00 sec 1.36 GBytes 2.34 Gbits/sec 0 2.34 MBytes [ 7][RX—C] 25.00—30.00 sec 1.34 GBytes 2.30 Gbits/sec [ 5][TX—C] 30.00—35.00 sec 1.36 GBytes 2.34 Gbits/sec 0 2.34 MBytes [ 7][RX—C] 30.00—35.00 sec 1.36 GBytes 2.33 Gbits/sec [ 5][TX—C] 35.00—40.00 sec 1.36 GBytes 2.34 Gbits/sec 0 2.34 MBytes [ 7][RX—C] 35.00—40.00 sec 1.33 GBytes 2.28 Gbits/sec [ 5][TX—C] 40.00—45.00 sec 1.36 GBytes 2.34 Gbits/sec 0 2.34 MBytes [ 7][RX—C] 40.00—45.00 sec 1.35 GBytes 2.32 Gbits/sec [ 5][TX—C] 45.00—50.00 sec 1.36 GBytes 2.34 Gbits/sec 0 2.34 MBytes [ 7][RX—C] 45.00—50.00 sec 1.35 GBytes 2.32 Gbits/sec [ 5][TX—C] 50.00—55.00 sec 1.36 GBytes 2.34 Gbits/sec 0 2.34 MBytes [ 7][RX—C] 50.00—55.00 sec 1.35 GBytes 2.31 Gbits/sec [ 5][TX—C] 55.00—60.00 sec 1.36 GBytes 2.34 Gbits/sec 0 2.34 MBytes [ 7][RX—C] 55.00—60.00 sec 1.32 GBytes 2.26 Gbits/sec — — — — — — — — — — — — — — — — — — — — — — — — — [ ID][Role] Interval Transfer Bitrate Retr [ 5][TX—C] 0.00—60.00 sec 16.3 GBytes 2.34 Gbits/sec 0 sender [ 5][TX—C] 0.00—60.05 sec 16.3 GBytes 2.34 Gbits/sec receiver [ 7][RX—C] 0.00—60.00 sec 16.0 GBytes 2.29 Gbits/sec 25 sender [ 7][RX—C] 0.00—60.05 sec 16.0 GBytes 2.29 Gbits/sec receiver iperf Done. |
]
Jean-Luc started CNX Software in 2010 as a part-time endeavor, before quitting his job as a software engineering manager, and starting to write daily news, and reviews full time later in 2011.
trying a quick experiment with ACI to see if I can use it to host iperf3 for some on demand network performance testing. The docker image is https://hub.docker.com/r/netgio/iperf3server/
I can get the container up and running but when I try to connect to it the iperf client gives me the error «iperf3: error — control socket has closed unexpectedly»
I was hoping this would be a simple, single port, single instance use case. are there any specifics related to socket connectivity into ACI instances that would affect this?
EDIT:
iPerf3 defaults to TCP port 5201 so to ensure ACI port mappings are working as expected I created an nginx docker image configured to listen on port 5201 and deployed to ACI. This worked as expected (I could access the default home page at http://:5201). So the issue seems to be specific to whatever iPerf3 is doing with its sockets.
Note the error happens almost immediately at launch of the client iperf3 process.
Здравствуйте.
Вкратце хотим добиться:
Максимально возможное количество соединений на edge-сервере при раздаче потоков, лимит которых будет упираться в лимит CPU/и или памяти.
Edge имеют одно dns имя. Все edge за classic load balancer+auto scaling group. Соответственно потоки клиенты будут получать по dns имени балансировщика. Тесты проводились с одним edge.
Окружение:
клиент (тестирующий) — c5.4xlarge (CPU-16 RAM-32):
-Xmx16G
-Xms16G
edge — c5.4xlarge (CPU-16 RAM-32):
-Xmx16G
-Xms16G
origin — с5.2xlarge (CPU-8 RAM-16):
-Xmx8G
-Xms8G
Везде:
Версия:
5.2.838 (при обновлениях на более новые появляется проблема подключения по ws. По возможности хотел бы вернуться к этому позже. пока проверяю на той, что работает)
сборщик мусора — ZGC
ice_tcp_transport=true
net.ipv4.ip_local_port_range = 54001 65000
media_port_from=31008
media_port_to=47000
wcs_agent_port_from=49001
wcs_agent_port_to=53999
WCS_FD_LIMIT=100000
webrtc_cc2_twcc=false
Есть другие настройки, скорее всего ненужные.
iptables -S:
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -s 127.0.0.0/8 -d 127.0.0.0/8 -i lo -j ACCEPT
-A INPUT -m state —state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m state —state INVALID -j DROP
-A INPUT -p tcp -m tcp ! —tcp-flags FIN,SYN,RST,ACK SYN -m state —state NEW -j DROP
-A INPUT -p icmp -m icmp —icmp-type 0 -j ACCEPT
-A INPUT -p icmp -m icmp —icmp-type 3 -j ACCEPT
-A INPUT -p icmp -m icmp —icmp-type 11 -j ACCEPT
-A INPUT -p icmp -m icmp —icmp-type 8 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 443 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 4431 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 9100 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 8888 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 8443 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 1935 -j ACCEPT
-A INPUT -p udp -m udp —dport 1935 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 554 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 8080 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 8081 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 8084 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 8082 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 8445 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 8444 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 9091 -j ACCEPT
-A INPUT -p tcp -m tcp —dport 20000:53999 -j ACCEPT
-A INPUT -p udp -m udp —dport 20000:53999 -j ACCEPT
-A INPUT -j DROP
-A FORWARD -m state —state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -m state —state INVALID -j DROP
-A FORWARD -j DROP
-A OUTPUT -s 127.0.0.0/8 -d 127.0.0.0/8 -o lo -j ACCEPT
-A OUTPUT -o eth0 -j ACCEPT
-A OUTPUT -m state —state RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp -m tcp ! —tcp-flags FIN,SYN,RST,ACK SYN -m state —state NEW -j DROP
(некоторые порты от прежних настроек, будем убирать)
/etc/sysctl.conf сожержит:
fs.file-max = 100000
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.ipv4.tcp_max_syn_backlog = 10240
net.ipv4.tcp_congestion_control=htcp
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_fin_timeout = 15
net.core.somaxconn = 1024
net.core.netdev_max_backlog = 65536
net.core.optmem_max = 25165824
net.ipv4.tcp_mem = 65536 131072 262144
net.ipv4.udp_mem = 65536 131072 262144
net.core.rmem_default = 25165824
net.core.rmem_max = 25165824
net.ipv4.tcp_rmem = 20480 12582912 25165824
net.ipv4.udp_rmem_min = 16384
net.core.wmem_default = 25165824
net.core.wmem_max = 25165824
net.ipv4.tcp_wmem = 20480 12582912 25165824
net.ipv4.udp_wmem_min = 16384
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_tw_reuse = 1
(по умолчанию не было, выставляли исходя из рекомендаций по настройке сети., ни на что не повлияло)
Проверка канала напрямую по внешнему ip (от тестирующего к edge):
iperf3 -c xxxxxx -p 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 579 MBytes 4.85 Gbits/sec 48 718 KBytes
[ 4] 1.00-2.00 sec 539 MBytes 4.52 Gbits/sec 103 870 KBytes
[ 4] 2.00-3.00 sec 566 MBytes 4.75 Gbits/sec 116 981 KBytes
[ 4] 3.00-4.00 sec 538 MBytes 4.51 Gbits/sec 153 397 KBytes
^C[ 4] 4.00-4.62 sec 349 MBytes 4.72 Gbits/sec 0 922 KBytes
— — — — — — — — — — — — — — — — — — — — — — — — —
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-4.62 sec 2.51 GBytes 4.67 Gbits/sec 420 sender
[ 4] 0.00-4.62 sec 0.00 Bytes 0.00 bits/sec receiver
iperf3 -c xxxxxx -p 5201 -R
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 572 MBytes 4.79 Gbits/sec
[ 4] 1.00-2.00 sec 571 MBytes 4.79 Gbits/sec
[ 4] 2.00-3.00 sec 571 MBytes 4.79 Gbits/sec
[ 4] 3.00-4.00 sec 571 MBytes 4.79 Gbits/sec
^C[ 4] 4.00-4.61 sec 346 MBytes 4.78 Gbits/sec
— — — — — — — — — — — — — — — — — — — — — — — — —
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-4.61 sec 0.00 Bytes 0.00 bits/sec sender
[ 4] 0.00-4.61 sec 2.57 GBytes 4.79 Gbits/sec receiver
Edge находятся в AWS Classic load balancer+ Autoscaling group (настраивал согласно статье https://flashphoner.com/cdn-s-balan…rovaniem-na-baze-amazon-web-services/?lang=ru)
Тестирующий, origin, edge находятся в одной placement group. тип — cluster (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html)
Проверка канала по dns имени балансировщика edge (от тестирующего к edge):
iperf3 -c xxxxxx -p 5201:
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 371 MBytes 3.11 Gbits/sec 95 235 KBytes
[ 4] 1.00-2.00 sec 471 MBytes 3.95 Gbits/sec 115 505 KBytes
[ 4] 2.00-3.00 sec 516 MBytes 4.33 Gbits/sec 229 472 KBytes
[ 4] 3.00-4.00 sec 365 MBytes 3.06 Gbits/sec 274 404 KBytes
[ 4] 4.00-5.00 sec 455 MBytes 3.82 Gbits/sec 106 359 KBytes
[ 4] 5.00-6.00 sec 514 MBytes 4.31 Gbits/sec 148 626 KBytes
[ 4] 6.00-7.00 sec 469 MBytes 3.93 Gbits/sec 182 293 KBytes
[ 4] 7.00-8.00 sec 436 MBytes 3.66 Gbits/sec 266 221 KBytes
[ 4] 8.00-9.00 sec 402 MBytes 3.38 Gbits/sec 144 444 KBytes
[ 4] 9.00-10.00 sec 440 MBytes 3.69 Gbits/sec 157 614 KBytes
— — — — — — — — — — — — — — — — — — — — — — — — —
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 4.34 GBytes 3.72 Gbits/sec 1716 sender
[ 4] 0.00-10.00 sec 3.98 GBytes 3.42 Gbits/sec receiver
iperf3 -c xxxxxx -p 5201 -R:
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 100 KBytes 822 Kbits/sec
[ 4] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec
iperf3: error — control socket has closed unexpectedly
На тестирующей машине выставляется:
wcs_activity_timer_timeout=86400000
Тестировалось с помощью «pull streams» в количестве 1000 соединений. Указывался внешний IP непосредственно edge.
Исходный поток, опубликованный на origin имеет битрейт примерно от 1.5 до 2 мбит
При тестировании наблюдается следующее:
при достижении примерно 300 соединений (примерно 500 мбит/с) перестаёт расти скорость, так и остаётся — максимум 500 мбит/с. Количество соединений при этом увеличивается до 1000, как и было указано при запуске теста. При этом изображение начинает опаздывать, затем тормозить, если соединений больше, чем примерно 300.
Скорость замерялась утилитой nload.
Значительных нагрузок на CPU, память не наблюдается как на edge, так и на тестирующей машине.
Ставили много экспериментов, всё упирается в скорость раздачи в 500 мбит/с. Просьба помочь с настройками, дать рекомендации по настройке инфраструктуры, если допустили концептуальные ошибки.
Те же результаты наблюдались с типом машин edge, тестирующей- с5.2xlarge (CPU-8 RAM-16 -Xmx8G
-Xms8G).
Заранее спасибо.