Ruby SSL TCP服务器在超过250个并发连接时卡住

Question

我正在ruby中开发一个SSL TCP服务器，并在多线程客户端上进行测试。当客户端上的线程数小于190时，服务器上没有问题，所有消息都被正确接收。但是，一旦我将客户端的线程数量增加到195个以上，就会出现两个问题：

问题1：服务器端出现ECONNABORTED异常

/usr/local/rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/openssl/ssl.rb:232:in `accept': Software caused connection abort - accept(2) (Errno::ECONNABORTED)
        from /usr/local/rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/openssl/ssl.rb:232:in `accept'
        from server.rb:30:in `block (2 levels) in start_server'

我可以通过重新启动异常处理程序中的accept循环来解决此问题。

问题2：服务器卡住当我增加客户端的线程数量（例如250）时，几秒钟后，服务器被冻结，即不允许出现异常和新连接。这个真的很烦人，因为服务器端没有办法知道它被冻结了。

操作系统：FreeBSD 10.1Ruby版本：2.2.1（也尝试2.1.5）

---

服务器代码：

loop do
    server = TCPServer.new(ip_address, port)
    sslContext = OpenSSL::SSL::SSLContext.new
    sslContext.cert = OpenSSL::X509::Certificate.new(File.open("cert/cert.pem"))
    sslContext.key = OpenSSL::PKey::RSA.new(File.open("cert/key.pem"), SSL_PASSWORD)
    sslServer = OpenSSL::SSL::SSLServer.new(server, sslContext)
    loop do
        Thread.new(sslServer.accept) do |connection|
          begin
            messageIn = connection.gets
            connection.close
          rescue Exception => ex
            puts "Exception in main loop : " + ex.message
            puts "Backtrace : " + ex.backtrace.join("n")
          end
        end
    end
  end
end

客户代码：

def create_client(host, port)
  begin
    socket = TCPSocket.open(host,port)
    ssl_context = OpenSSL::SSL::SSLContext.new()
    ssl_context.cert = OpenSSL::X509::Certificate.new(File.open("lib/cert/cert.pem"))
    ssl_context.key = OpenSSL::PKey::RSA.new(File.open("lib/cert/key.pem"), SSL_PASSWORD)
    ssl_context.ssl_version = :SSLv3
    ssl_context.ssl_timeout = 10
    ssl_socket = OpenSSL::SSL::SSLSocket.new(socket, ssl_context)
    ssl_socket.sync_close = true
    ssl_socket.connect
  rescue Exception => ex
    puts "Exception in create_client"
    sleep 1
    return create_client(host, port )
  end
  return ssl_socket
end

for j in 1..10 do
      threads = []
      for i in 1..n.to_i do
          threads << Thread.new do
            begin
              socket = create_client(ip, port)
              socket.puts("hello")
              socket.flush
              socket.close
            rescue Exception => ex
              puts "Exception"
            end
          end
      end
      threads.each(&:join)
    end

Answer 1

听起来你在服务器端遇到了网络限制。您可能需要使用sysctl kern.ipc.soacceptqueue=250增加队列长度（或在/etc/sysctl.conf中永久设置）。您可以使用netstat -Lan检查应用程序连接队列。

Answer 2

我终于找到了一个解决方案（可能不是唯一一个），它包含/etc/sysctl.conf的以下内容https://calomel.org/freebsd_network_tuning.html

kern.ipc.maxsockbuf=4194304  # (default 2097152)
net.inet.tcp.sendbuf_max=4194304  # (default 2097152)
net.inet.tcp.recvbuf_max=4194304  # (default 2097152)
net.inet.tcp.cc.algorithm=htcp  # (default newreno)
net.inet.tcp.cc.htcp.adaptive_backoff=1 # (default 0 ; disabled)
net.inet.tcp.cc.htcp.rtt_scaling=1 # (default 0 ; disabled)
net.inet.ip.forwarding=1      # (default 0)
net.inet.ip.fastforwarding=1  # (default 0)
kern.ipc.soacceptqueue=1024  # (default 128 ; same as kern.ipc.somaxconn)
net.inet.tcp.mssdflt=1460  # (default 536)
net.inet.tcp.minmss=1300   # (default 216)
net.inet.tcp.rfc1323=1  # (default 1)
net.inet.tcp.rfc3390=1  # (default 1)
net.inet.tcp.sack.enable=1  # (default 1)
net.inet.tcp.tso=0   # (default 1)
net.inet.tcp.nolocaltimewait=1  # (default 0)
net.inet.tcp.experimental.initcwnd10=1        # (default 1 for FreeBSD 10.1)
net.inet.tcp.syncache.rexmtlimit=0  # (default 3)
net.inet.ip.rtexpire=2       # (default 3600)
net.inet.ip.rtminexpire=2    # (default 10  )
net.inet.tcp.syncookies=0  # (default 1)
net.inet.ip.check_interface=1         # verify packet arrives on correct interface (default 0)
net.inet.ip.process_options=0         # ignore IP options in the incoming packets (default 1)
net.inet.ip.redirect=0                # do not send IP redirects (default 1)
net.inet.ip.stealth=1                 # do not reduce the TTL by one(1) when a packets goes through the firewall (default 0)
net.inet.icmp.drop_redirect=1         # no redirected ICMP packets (default 0)
net.inet.tcp.drop_synfin=1            # SYN/FIN packets get dropped on initial connection (default 0)
net.inet.tcp.fast_finwait2_recycle=1  # recycle FIN/WAIT states quickly (helps against DoS, but may cause false RST) (default 0)
net.inet.tcp.icmp_may_rst=0           # icmp may not send RST to avoid spoofed icmp/udp floods (default 1)
net.inet.tcp.msl=5000                 # 5s maximum segment life waiting for an ACK in reply to a SYN-ACK or FIN-ACK (default 30000)
net.inet.tcp.path_mtu_discovery=0     # disable MTU discovery since most ICMP type 3 packets are dropped by others (default 1)
net.inet.udp.blackhole=1              # drop udp packets destined for closed sockets (default 0)
net.inet.tcp.blackhole=2              # drop tcp packets destined for closed ports (default 0)
security.bsd.see_other_uids=0         # users only see their own processes. root can see all (default 1)

我现在可以在客户端运行1000个线程，在服务器端没有任何异常或冻结。