Commit graph

963 commits

Author SHA1 Message Date
Burdette Lamar
a48e01ccba [ruby/net-http] Update lib/net/http.rb
16d042fad6

Co-authored-by: Peter Zhu <peter@peterzhu.ca>
2022-11-11 17:05:29 +00:00
BurdetteLamar
4d9ada223b [ruby/net-http] Prettify class hierarchies in Net::HTTP
4a5732e210
2022-11-11 17:05:28 +00:00
Hiroshi SHIBATA
c5f5403f6e [ruby/net-http] Bump version to 0.3.0 2022-10-24 05:45:44 +00:00
Hiroshi SHIBATA
f88bff7705 [ruby/net-http] Revert "Replace Timeout.timeout in Net:HTTP#connect"
This reverts commit 753cae3bbc.

98caa38204
2022-10-21 12:39:52 +00:00
Hiroshi SHIBATA
1cda414622 Raise ArgumentError with empty host url again.
Fixup dd5118f852

Co-authored-by: Koichi Sasada <ko1@atdot.net>
2022-10-13 12:32:02 +09:00
Jeremy Evans
cd77e71bba [ruby/net-http] Remove ENVIRONMENT_VARIABLE_IS_MULTIUSER_SAFE
This list is out of date.  At least OpenBSD since 2013 does not
allow one user to read the environment variables of a process
run by another user.

While we could try to keep the list updated, I think it's a bad
idea to not use the user/password from the environment, even if
another user on the system could read it.  If http_proxy exists
in the environment, and other users can read it, it doesn't
make it more secure for Ruby to ignore it.  You could argue that
it encourages poor security practices, but net/http should provide
mechanism, not policy.

Fixes [Bug #18908]

1e4585153d
2022-09-28 17:26:03 +09:00
Jean Boussier
739380c97d [ruby/net-protocol] Improve BufferedIO performance
`BufferedIO` is a bit inefficient for reading large responses because
it use the classic `buffer.slice!` technique which cause a lot of
unnecessary string copying.

This is particularly visible on line based protocol when reading
line by line.

Instead of repeatedly shifting the string, we can keep track of
which offset we're at, to know how many bytes are left in the buffer.

This change also open the door to further optimization by increasing
the buffer size, as previously `slice!` would get slower the larger
the buffer is.

Benchmark results:

```
=== 1k ===
Warming up --------------------------------------
                  1k     1.234k i/100ms
              1k opt     1.283k i/100ms
Calculating -------------------------------------
                  1k     12.615k (± 0.9%) i/s -     64.168k in   5.086995s
              1k opt     12.856k (± 0.9%) i/s -     65.433k in   5.090051s

Comparison:
                  1k:    12615.2 i/s
              1k opt:    12856.0 i/s - 1.02x  (± 0.00) faster

=== 10k ===
Warming up --------------------------------------
                 10k     1.165k i/100ms
             10k opt     1.269k i/100ms
Calculating -------------------------------------
                 10k     11.550k (± 2.4%) i/s -     58.250k in   5.046378s
             10k opt     12.736k (± 1.0%) i/s -     64.719k in   5.081969s

Comparison:
                 10k:    11550.3 i/s
             10k opt:    12736.3 i/s - 1.10x  (± 0.00) faster

=== 100k ===
Warming up --------------------------------------
                100k   809.000  i/100ms
            100k opt   926.000  i/100ms
Calculating -------------------------------------
                100k      8.054k (± 3.0%) i/s -     40.450k in   5.028299s
            100k opt      9.286k (± 2.2%) i/s -     47.226k in   5.088841s

Comparison:
                100k:     8053.6 i/s
            100k opt:     9285.5 i/s - 1.15x  (± 0.00) faster

=== 1M ===
Warming up --------------------------------------
                  1M   249.000  i/100ms
              1M opt   315.000  i/100ms
Calculating -------------------------------------
                  1M      2.448k (± 2.5%) i/s -     12.450k in   5.089744s
              1M opt      3.119k (± 2.6%) i/s -     15.750k in   5.053772s

Comparison:
                  1M:     2447.8 i/s
              1M opt:     3118.8 i/s - 1.27x  (± 0.00) faster
```

Profiling before (1MB responses):

```
==================================
  Mode: wall(1000)
  Samples: 5276 (0.00% miss rate)
  GC: 394 (7.47%)
==================================
     TOTAL    (pct)     SAMPLES    (pct)     FRAME
      1622  (30.7%)        1622  (30.7%)     IO#wait_readable
       777  (14.7%)         777  (14.7%)     IO#read_nonblock
       365   (6.9%)         365   (6.9%)     (sweeping)
      2705  (51.3%)         364   (6.9%)     Net::BufferedIO#rbuf_fill
       264   (5.0%)         264   (5.0%)     String#index
       223   (4.2%)         223   (4.2%)     String#sub
       221   (4.2%)         221   (4.2%)     String#slice!
       185   (3.5%)         185   (3.5%)     String#split
       108   (2.0%)         108   (2.0%)     IO#write_nonblock
       101   (1.9%)         101   (1.9%)     String#downcase
        66   (1.3%)          66   (1.3%)     Net::BufferedIO#LOG
        57   (1.1%)          57   (1.1%)     String#count
        51   (1.0%)          51   (1.0%)     String#to_s
       391   (7.4%)          50   (0.9%)     Net::HTTPGenericRequest#write_header
        50   (0.9%)          50   (0.9%)     String#capitalize
        49   (0.9%)          49   (0.9%)     Array#join
        47   (0.9%)          47   (0.9%)     String#b
       106   (2.0%)          36   (0.7%)     Net::HTTPHeader#set_field
        34   (0.6%)          34   (0.6%)     Module#===
        33   (0.6%)          33   (0.6%)     String#[]
       140   (2.7%)          29   (0.5%)     Net::BufferedIO#write0
        29   (0.5%)          29   (0.5%)     (marking)
       281   (5.3%)          27   (0.5%)     Net::BufferedIO#rbuf_consume
      1195  (22.6%)          25   (0.5%)     Net::HTTPResponse#read_body
      1024  (19.4%)          25   (0.5%)     Net::HTTPResponse.each_response_header
        86   (1.6%)          24   (0.5%)     Net::HTTPHeader#set_field
        23   (0.4%)          23   (0.4%)     Net::HTTP#proxy_uri
        51   (1.0%)          23   (0.4%)     Net::HTTPHeader#initialize_http_header
      2225  (42.2%)          22   (0.4%)     Net::BufferedIO#readuntil
        20   (0.4%)          20   (0.4%)     Regexp#===
```

Profiling after (1MB responses):

```
==================================
  Mode: wall(1000)
  Samples: 15180 (0.00% miss rate)
  GC: 1688 (11.12%)
==================================
     TOTAL    (pct)     SAMPLES    (pct)     FRAME
      4534  (29.9%)        4534  (29.9%)     IO#read_nonblock
     10650  (70.2%)        3944  (26.0%)     Net::HTTPOpt::BufferedIOOpt#rbuf_fill
      2101  (13.8%)        2101  (13.8%)     IO#wait_readable
      1442   (9.5%)        1442   (9.5%)     (sweeping)
       360   (2.4%)         360   (2.4%)     String#sub
       312   (2.1%)         312   (2.1%)     String#split
       265   (1.7%)         265   (1.7%)     String#bytesize
       246   (1.6%)         246   (1.6%)     (marking)
       151   (1.0%)         151   (1.0%)     IO#write_nonblock
       125   (0.8%)         125   (0.8%)     String#downcase
       116   (0.8%)         116   (0.8%)     String#index
       113   (0.7%)         113   (0.7%)     Module#===
       162   (1.1%)          89   (0.6%)     Net::HTTPOpt::BufferedIOOpt#rbuf_consume_all_shareable!
       158   (1.0%)          65   (0.4%)     Net::HTTPHeader#set_field
        63   (0.4%)          63   (0.4%)     String#capitalize
        63   (0.4%)          63   (0.4%)     BasicObject#equal?
        58   (0.4%)          58   (0.4%)     Regexp#match
        58   (0.4%)          58   (0.4%)     String#[]
       449   (3.0%)          56   (0.4%)     Net::HTTPGenericRequest#write_header
        53   (0.3%)          53   (0.3%)     String#to_s
        52   (0.3%)          52   (0.3%)     Net::HTTPOpt::BufferedIOOpt#LOG
        52   (0.3%)          52   (0.3%)     String#count
        44   (0.3%)          44   (0.3%)     String#byteslice
        44   (0.3%)          44   (0.3%)     Array#join
      1096   (7.2%)          42   (0.3%)     Net::HTTPResponse.each_response_header
      2617  (17.2%)          40   (0.3%)     Net::HTTPOpt::BufferedIOOpt#readuntil
       132   (0.9%)          30   (0.2%)     Net::HTTPOpt::BufferedIOOpt#rbuf_consume
        28   (0.2%)          28   (0.2%)     Regexp#===
        27   (0.2%)          27   (0.2%)     Net::HTTP#proxy_uri
      8862  (58.4%)          27   (0.2%)     Net::HTTPResponse#read_body
````

Benchmark code:

```ruby

require "fileutils"
DIR = "/tmp/www"
FileUtils.mkdir_p(DIR)
HOST = "127.0.0.1"
PORT = 8080
CONF = <<~EOS
daemon            off;
worker_processes  2;

events {
    worker_connections  128;
}

http {
    server_tokens off;
    charset       utf-8;

    server {
        server_name   localhost;
        listen        #{HOST}:#{PORT};

        keepalive_requests 10000000;
        keepalive_timeout 3600s;

        error_page    500 502 503 504  /50x.html;

        location      / {
            root      #{DIR};
        }

    }

}
EOS

File.write(File.join(DIR, "1k.txt"), 'a' * 1024)
File.write(File.join(DIR, "10k.txt"), 'a' * 1024 * 10)
File.write(File.join(DIR, "100k.txt"), 'a' * 1024 * 100)
File.write(File.join(DIR, "1M.txt"), 'a' * 1024 * 1024)

File.write(File.join(DIR, "nginx.conf"), CONF)

require "benchmark/ips"
require "net/http"

nginx_pid = Process.spawn('nginx', '-c', File.join(DIR, "nginx.conf"))

module Net
  class HTTPOpt < HTTP

    class BufferedIOOpt < ::Net::BufferedIO  #:nodoc: internal use only
      def initialize(io, read_timeout: 60, write_timeout: 60, continue_timeout: nil, debug_output: nil)
        @io = io
        @read_timeout = read_timeout
        @write_timeout = write_timeout
        @continue_timeout = continue_timeout
        @debug_output = debug_output
        @rbuf = ''.b
        @rbuf_offset = 0
      end

      attr_reader :io
      attr_accessor :read_timeout
      attr_accessor :write_timeout
      attr_accessor :continue_timeout
      attr_accessor :debug_output

      def inspect
        "#<#{self.class} io=#{@io}>"
      end

      def eof?
        @io.eof?
      end

      def closed?
        @io.closed?
      end

      def close
        @io.close
      end

      #
      # Read
      #

      public

      def read(len, dest = ''.b, ignore_eof = false)
        LOG "reading #{len} bytes..."
        read_bytes = 0
        begin
          while read_bytes + rbuf_size < len
            if s = rbuf_consume_all_shareable!
              read_bytes += s.bytesize
              dest << s
            end
            rbuf_fill
          end
          s = rbuf_consume(len - read_bytes)
          read_bytes += s.bytesize
          dest << s
        rescue EOFError
          raise unless ignore_eof
        end
        LOG "read #{read_bytes} bytes"
        dest
      end

      def read_all(dest = ''.b)
        LOG 'reading all...'
        read_bytes = 0
        begin
          while true
            if s = rbuf_consume_all_shareable!
              read_bytes += s.bytesize
              dest << s
            end
            rbuf_fill
          end
        rescue EOFError
          ;
        end
        LOG "read #{read_bytes} bytes"
        dest
      end

      def readuntil(terminator, ignore_eof = false)
        offset = @rbuf_offset
        begin
          until idx = @rbuf.index(terminator, offset)
            offset = @rbuf.bytesize
            rbuf_fill
          end
          return rbuf_consume(idx + terminator.bytesize - @rbuf_offset)
        rescue EOFError
          raise unless ignore_eof
          return rbuf_consume
        end
      end

      def readline
        readuntil("\n").chop
      end

      private

      BUFSIZE = 1024 * 16

      def rbuf_fill
        tmp = @rbuf_empty ? @rbuf : nil
        case rv = @io.read_nonblock(BUFSIZE, tmp, exception: false)
        when String
          @rbuf_empty = false
          if rv.equal?(tmp)
            @rbuf_offset = 0
          else
            @rbuf << rv
            rv.clear
          end
          return
        when :wait_readable
          (io = @io.to_io).wait_readable(@read_timeout) or raise Net::ReadTimeout.new(io)
          # continue looping
        when :wait_writable
          # OpenSSL::Buffering#read_nonblock may fail with IO::WaitWritable.
          # http://www.openssl.org/support/faq.html#PROG10
          (io = @io.to_io).wait_writable(@read_timeout) or raise Net::ReadTimeout.new(io)
          # continue looping
        when nil
          raise EOFError, 'end of file reached'
        end while true
      end

      def rbuf_flush
        if @rbuf_empty
          @rbuf.clear
          @rbuf_offset = 0
        end
        nil
      end

      def rbuf_size
        @rbuf.bytesize - @rbuf_offset
      end

      # Warning: this method may share the buffer to avoid
      # copying. The caller must no longer use the returned
      # string once rbuf_fill has been called again
      def rbuf_consume_all_shareable!
        @rbuf_empty = true
        buf = if @rbuf_offset == 0
          @rbuf
        else
          @rbuf.byteslice(@rbuf_offset..-1)
        end
        @rbuf_offset = @rbuf.bytesize
        buf
      end

      def rbuf_consume(len = nil)
        if @rbuf_offset == 0 && (len.nil? || len == @rbuf.bytesize)
          s = @rbuf
          @rbuf = ''.b
          @rbuf_offset = 0
          @rbuf_empty = true
        elsif len.nil?
          s = @rbuf.byteslice(@rbuf_offset..-1)
          @rbuf = ''.b
          @rbuf_offset = 0
          @rbuf_empty = true
        else
          s = @rbuf.byteslice(@rbuf_offset, len)
          @rbuf_offset += len
          @rbuf_empty = @rbuf_offset == @rbuf.bytesize
          rbuf_flush
        end

        @debug_output << %Q[-> #{s.dump}\n] if @debug_output
        s
      end

      #
      # Write
      #

      public

      def write(*strs)
        writing {
          write0(*strs)
        }
      end

      alias << write

      def writeline(str)
        writing {
          write0 str + "\r\n"
        }
      end

      private

      def writing
        @written_bytes = 0
        @debug_output << '<- ' if @debug_output
        yield
        @debug_output << "\n" if @debug_output
        bytes = @written_bytes
        @written_bytes = nil
        bytes
      end

      def write0(*strs)
        @debug_output << strs.map(&:dump).join if @debug_output
        orig_written_bytes = @written_bytes
        strs.each_with_index do |str, i|
          need_retry = true
          case len = @io.write_nonblock(str, exception: false)
          when Integer
            @written_bytes += len
            len -= str.bytesize
            if len == 0
              if strs.size == i+1
                return @written_bytes - orig_written_bytes
              else
                need_retry = false
                # next string
              end
            elsif len < 0
              str = str.byteslice(len, -len)
            else # len > 0
              need_retry = false
              # next string
            end
            # continue looping
          when :wait_writable
            (io = @io.to_io).wait_writable(@write_timeout) or raise Net::WriteTimeout.new(io)
            # continue looping
          end while need_retry
        end
      end

      #
      # Logging
      #

      private

      def LOG_off
        @save_debug_out = @debug_output
        @debug_output = nil
      end

      def LOG_on
        @debug_output = @save_debug_out
      end

      def LOG(msg)
        return unless @debug_output
        @debug_output << msg + "\n"
      end
    end
    BufferedIO = BufferedIOOpt

    # Unchanged from ruby 3.1.1, only allow to lookup the mofidied BufferedIO
    def connect
      if use_ssl?
        # reference early to load OpenSSL before connecting,
        # as OpenSSL may take time to load.
        @ssl_context = OpenSSL::SSL::SSLContext.new
      end

      if proxy? then
        conn_addr = proxy_address
        conn_port = proxy_port
      else
        conn_addr = conn_address
        conn_port = port
      end

      D "opening connection to #{conn_addr}:#{conn_port}..."
      begin
        s = Socket.tcp conn_addr, conn_port, @local_host, @local_port, connect_timeout: @open_timeout
      rescue => e
        e = Net::OpenTimeout.new(e) if e.is_a?(Errno::ETIMEDOUT) #for compatibility with previous versions
        raise e, "Failed to open TCP connection to " +
          "#{conn_addr}:#{conn_port} (#{e.message})"
      end
      s.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_NODELAY, 1)
      D "opened"
      if use_ssl?
        if proxy?
          plain_sock = BufferedIO.new(s, read_timeout: @read_timeout,
                                      write_timeout: @write_timeout,
                                      continue_timeout: @continue_timeout,
                                      debug_output: @debug_output)
          buf = "CONNECT #{conn_address}:#{@port} HTTP/#{HTTPVersion}\r\n"
          buf << "Host: #{@address}:#{@port}\r\n"
          if proxy_user
            credential = ["#{proxy_user}:#{proxy_pass}"].pack('m0')
            buf << "Proxy-Authorization: Basic #{credential}\r\n"
          end
          buf << "\r\n"
          plain_sock.write(buf)
          HTTPResponse.read_new(plain_sock).value
          # assuming nothing left in buffers after successful CONNECT response
        end

        ssl_parameters = Hash.new
        iv_list = instance_variables
        SSL_IVNAMES.each_with_index do |ivname, i|
          if iv_list.include?(ivname)
            value = instance_variable_get(ivname)
            unless value.nil?
              ssl_parameters[SSL_ATTRIBUTES[i]] = value
            end
          end
        end
        @ssl_context.set_params(ssl_parameters)
        @ssl_context.session_cache_mode =
          OpenSSL::SSL::SSLContext::SESSION_CACHE_CLIENT |
          OpenSSL::SSL::SSLContext::SESSION_CACHE_NO_INTERNAL_STORE
        @ssl_context.session_new_cb = proc {|sock, sess| @ssl_session = sess }
        D "starting SSL for #{conn_addr}:#{conn_port}..."
        s = OpenSSL::SSL::SSLSocket.new(s, @ssl_context)
        s.sync_close = true
        # Server Name Indication (SNI) RFC 3546
        s.hostname = @address if s.respond_to? :hostname=
        if @ssl_session and
           Process.clock_gettime(Process::CLOCK_REALTIME) < @ssl_session.time.to_f + @ssl_session.timeout
          s.session = @ssl_session
        end
        ssl_socket_connect(s, @open_timeout)
        if (@ssl_context.verify_mode != OpenSSL::SSL::VERIFY_NONE) && @ssl_context.verify_hostname
          s.post_connection_check(@address)
        end
        D "SSL established, protocol: #{s.ssl_version}, cipher: #{s.cipher[0]}"
      end
      @socket = BufferedIO.new(s, read_timeout: @read_timeout,
                               write_timeout: @write_timeout,
                               continue_timeout: @continue_timeout,
                               debug_output: @debug_output)
      @last_communicated = nil
      on_connect
    rescue => exception
      if s
        D "Conn close because of connect error #{exception}"
        s.close
      end
      raise
    end
    private :connect
  end
end

begin
  sleep 0.2

  connection = Net::HTTP.start(HOST, PORT)
  connection.keep_alive_timeout = 3600
  connection_opt = Net::HTTPOpt.start(HOST, PORT)
  connection_opt.keep_alive_timeout = 3600

  unless connection.request_get("/100k.txt").body == connection_opt.request_get("/100k.txt").body
    abort("bug?")
  end

  if ARGV.first == "profile"
    require 'stackprof'
    require 'json'

    StackProf.run(mode: :wall, out: "/tmp/stackprof-net-http.dump", raw: true) do
      40_000.times do
        connection.request_get("/1M.txt").body
      end
    end
    File.write("/tmp/stackprof-net-http.json", JSON.dump(Marshal.load(File.binread("/tmp/stackprof-net-http.dump"))))
    system("stackprof", "/tmp/stackprof-net-http.rb")

    StackProf.run(mode: :wall, out: "/tmp/stackprof-net-http-opt.dump", raw: true) do
      40_000.times do
        connection_opt.request_get("/1M.txt").body
      end
    end
    File.write("/tmp/stackprof-net-http-opt.json", JSON.dump(Marshal.load(File.binread("/tmp/stackprof-net-http-opt.dump"))))
    system("stackprof", "/tmp/stackprof-net-http-opt.dump")

  else
    %w(1k 10k 100k 1M).each do |size|
      puts "=== #{size} ==="
      Benchmark.ips do |x|
        path = "/#{size}.txt"
        x.report("#{size}") { connection.request_get(path).body }
        x.report("#{size} opt") { connection_opt.request_get(path).body }
        x.compare!(order: :baseline)
      end
      puts
    end
  end
ensure
  Process.kill('TERM', nginx_pid)
  Process.wait(nginx_pid)
end

```

781e400389
2022-08-31 12:37:49 +09:00
Shishir Joshi
c310691dd8 [ruby/net-http] Make Net::HTTPHeader#content_range return nil on non-byte units
* Returning nil from the `content_range` method instead of raising an
  error when the unit in the content-range header is not "bytes".

Fix https://bugs.ruby-lang.org/issues/11450

0b5030dd86

Co-Authored-By: Nobuyoshi Nakada <nobu@ruby-lang.org>
2022-06-16 23:35:27 +09:00
Nobuyoshi Nakada
2223eb082a
Revert "HTTPHeader.content_range throws error on non-byte units"
This reverts commit 63546bfc15.
2022-06-16 22:10:59 +09:00
Shishir Joshi
63546bfc15
HTTPHeader.content_range throws error on non-byte units
* Added a nil check in Net::HTTPHeader#initialize_http_header for keys in the header that do not have any value
* Returning nil from the content_range method instead of raising an error when the unit in the content-range header is not bytes
* Modified initialize_http_header to match trunk

fix [Bug #11450]
fix https://github.com/ruby/ruby/pull/1018
2022-06-16 20:16:47 +09:00
Nobuyoshi Nakada
4cf155e007 [ruby/net-http] [DOC] Get rid of a RDoc bug
RDoc overrides class name by the assigned name unexpectedly when
assigned using a qualified class path.

a7bded0407
2022-05-23 18:23:22 +09:00
Nobuyoshi Nakada
8fa9e168be [ruby/net-http] Make the recommended name formal
`HTTPServerException` is the name deprecated since years ago.

b3028fef5a
2022-05-21 00:41:54 +09:00
Hiroshi SHIBATA
ecf32dbfc0 [ruby/net-http] Bump version to 0.2.2
992d07cb41
2022-05-09 18:20:31 +09:00
Charles Oliver Nutter
cd2613b6a4 [ruby/net-http] Bump version to 0.2.1.pre1
0017cc64c0
2022-04-28 05:56:26 +09:00
Karol Bucek
cf73cf5981 [ruby/net-http] Feature detect to make net/http usable with JRuby
Handle missing session_new_cb= and do not call
session_cache_mode=, as JRuby SSL does not support
these methods.

3237ef4d8c
2022-04-20 13:01:08 +09:00
Jeremy Evans
90ccc5674a [ruby/net-http] Add ignore_eof access to HTTP and HTTPResponse
The ignore_eof setting on HTTPResponse makes it so an EOFError is
raised when reading bodies with a defined Content-Length, if the
body read was truncated due to the socket be closed.

The ignore_eof setting on HTTP sets the values used in responses
that are created by the object.

For backwards compatibility, the default is for both settings is
true.  However, unless you are specifically tested for and handling
truncated responses, it's a good idea to set ignore_eof to false so
that errors are raised for truncated responses, instead of those
errors silently being ignored.

Fixes [Bug #14972]

4d47e34995
2022-04-20 10:22:06 +09:00
Jeremy Evans
0579486f15 [ruby/net-http] Update the content-length heading when decoding bodies
Previously, the content-encoding header was removed and the body
was modified, but the content-length header was not modified,
resulting in the content-length header not matching the body
length.

Don't delete content-length before yielding inflate body, as that
causes a switch to read the entire body instead of reading in
chunks.

Fixes [Bug #16672]

58284e9710

Co-authored-by: st0012 <stan001212@gmail.com>
2022-04-14 00:11:26 +09:00
Jeremy Evans
ebb4378237 [ruby/net-http] Add HTTP#response_body_encoding for setting response body encoding
This allows for the ability to opt-in to a method to set the
encoding of response bodies.  By setting the accessor to a String
or Encoding instance, it will use the specified encoding.
Setting the value of true will try to detect the encoding of the
response body, either using the Content-Type header (assuming it
specifies charset) or by scanning for a <meta> tag in the document
that specifies the encoding.  The default is false in which case
no forcing of encoding will be done (same as before the patch).

Implements [Feature #2567]
Implements [Feature #15517]

6233e6b7c1

Co-authored-by: Yui Naruse <naruse@ruby-lang.org>
2022-04-12 00:17:34 +09:00
Jeremy Evans
7648bae4c8 [ruby/net-http] Revert "Update the content-length heading when decoding bodies"
This reverts commit a7cb30124c.

This is causing errors in Ruby's CI, will revert for now and
try again after testing a fix with Ruby's CI.

7b852b1feb
2022-04-03 06:51:07 +09:00
Jeremy Evans
58adb1636b [ruby/net-http] Update the content-length heading when decoding bodies
Previously, the content-encoding header was removed and the body
was modified, but the content-length header was not modified,
resulting in the content-length header not matching the body
length.

Fixes [Bug #16672]

a7cb30124c
2022-04-02 02:49:21 +09:00
Hiroshi SHIBATA
06f94d1784 [ruby/net-protocol] Bump version to 0.1.3
9cf40af499
2022-04-01 13:56:47 +09:00
Hiroshi SHIBATA
c53bdb8ff6
Removed dependency of net-protocol. There is no plan to remove from stdlib 2022-02-21 17:10:56 +09:00
Hiroshi SHIBATA
79b04790e8
Removed dependency of io-wait. There is no plan to remove from stdlib 2022-02-21 17:10:22 +09:00
Jeremy Evans
7529c53891 [ruby/net-http] Do not set SNI hostname if connecting to IP address
RFC 6066, section 3, explicitly disallows the use of an IP address
as an SNI server name.  So check if the connection is being made
to an IP address using the resolv regexps, and do not set an SNI
hostname in that case.

Recent changes to LibreSSL make it more strictly follow RFC 6066,
resulting an s.hostname= raising an error if passed an IP address.
When such verions of LibreSSL are used, this change not only fixes
the net/http tests, it also fixes tests for webrick and open-uri,
which both make SSL connections to 127.0.0.1 using net/http in
their tests.

Avoid warning in the openssl extension by unsetting
@ssl_context.verify_hostname if connecting to an IP address.
Make changes so that the post_connection_check still happens
when connecting to an IP address, which is necessary to keep
checking that the certificate returned includes the IP address,
which one of the tests depends on.

Revert the previous change that modified the regexp used for
checking the error message.

fa68e64bee
2022-02-03 05:10:21 +09:00
Olle Jonsson
d8c54bac4a [ruby/net-protocol] Fix typo in gem description [ci skip]
35d7b08a54
2022-02-02 21:21:47 +09:00
Olle Jonsson
94687a6826 [ruby/net-protocol] Drop unused gemspec directives
This gem exposes no executables.

3c4def2a64
2022-02-02 21:17:47 +09:00
Brandon Weaver
ac1bb6b510 [ruby/net-http] Rename D to debug in Net::HTTP
Renames `D` to `debug` in `Net::HTTP` and introduces an alias for
backwards compatibility. This was done for readability reasons, in that
`D` did not clearly reflect what the method was doing and can cause some
confusion.

582d6e87d6
2022-01-17 10:17:23 +09:00
Kazuhiro NISHIYAMA
5757696e07
res.response_body_permitted? is not defined
`response_body_permitted?` is a method of request.
2022-01-05 16:20:11 +09:00
Olle Jonsson
c240ccd643 [ruby/net-http] gemspec: Drop unused directive "executables"
This gem exposes no executable files.

3b3743f6ce
2021-11-17 19:09:14 +09:00
Nobuyoshi Nakada
5bff4cd56c [ruby/net-protocol] Update the required ruby version
d4982420e6
2021-11-16 20:56:04 +09:00
Shohei Maeda
52ab9bbee9 [ruby/net-http] Send Accept-Encoding header on HEAD method
9d95c5e3e6
2021-11-11 17:34:00 +09:00
Nobuyoshi Nakada
3d8e1ee40f [ruby/net-http] Warn deprecated old constants
2a97b4729b
2021-11-04 20:51:51 +09:00
Nobuyoshi Nakada
b49dbe025f [ruby/net-http] Fix the typo in a constant name
dada6007bf
2021-11-04 20:51:42 +09:00
Kazuki Yamaguchi
5f2c4e344d [ruby/net-http] Reset keep_alive timer on new connection
The last_communicated timestamp is for HTTP persistent connection, to
decide whether the current TCP connection may be reused for the
subsequent requests or not. Naturally, the timer must be reset if the
connection is recreated since it is no longer relevant.

0a013de42d
2021-11-04 20:44:04 +09:00
180909
6c812c6f4e add missing http response code in doc 2021-10-30 10:16:17 +09:00
Hiroshi SHIBATA
cb4c98f949 [ruby/net-http] Bump up net-http version to 0.2.0
f3e65e2a31
2021-10-21 21:01:06 +09:00
Hiroshi SHIBATA
cc2f76041d [ruby/net-protocol] Bump up net-protocol version to 0.1.2
088e52609a
2021-10-21 21:00:53 +09:00
Miguel Teixeira
60b02db516 [ruby/net-http] Enforce write timeout when body_stream is used
The existing implementation of `Net::HTTP#write_timeout` relies on
`Net::BefferedIO` to trigger the `Net::WriteTimeout` error. This commit
changes `send_request_with_body_stream` to remove the optimization that
was making `Net::HTTP#write_timeout` not work when `body_stream` is
used.

Open issue:
https://bugs.ruby-lang.org/issues/17933

a0fab1ab52
2021-07-29 15:53:54 +09:00
Nobuyoshi Nakada
2b17d2f297 [ruby/net-protocol] Get rid of __send__
Mitigate the security risk:
https://devcraft.io/2021/01/07/universal-deserialisation-gadget-for-ruby-2-x-3-x.html

a9970437e8
2021-06-16 22:23:46 +09:00
Hiroshi SHIBATA
cd2190448d
[ruby/net-protocol] Bump version to 0.1.1
97c4b68528
2021-05-31 17:40:07 +09:00
Hiroshi SHIBATA
be2e2b4805 Promote net-smtp to the bundled gems 2021-05-27 14:42:11 +09:00
Hiroshi SHIBATA
aa9726f7b9 Promote net-pop to the bundled gems 2021-05-27 14:42:11 +09:00
Hiroshi SHIBATA
d5bc6b2337 Promote net-imap to the bundled gems 2021-05-27 14:42:11 +09:00
Hiroshi SHIBATA
e49c998d1e Promote net-ftp to the bundled gems 2021-05-27 14:42:11 +09:00
Shugo Maeda
773c690f25 [ruby/net-ftp] Bump version to 0.1.2
895ba44b3c
2021-05-12 09:20:41 +09:00
Kazuki Yamaguchi
364044e090 [ruby/net-http] Do not require stringio
It is not used in net/http library code since commit 15ccd0118c13
(r36473 in ruby svn trunk, 2012).

require's in test suite are also cleaned up.

996d18a43f
2021-05-06 15:58:41 +09:00
Hiroshi SHIBATA
965719f5eb
Fixed the file path for net-imap.gemspec 2021-05-06 15:24:49 +09:00
Hiroshi SHIBATA
5de6f1ab47
Move net-imap.gemspec to under the lib/net/imap directory. 2021-05-06 15:20:35 +09:00
nicholas a. evans
5a02281fab
[ruby/net-imap] Many documentation improvements
* updated obsoleted RFCs to current versions
* linked most references to their RFCs
* linked extension commands to their RFCs
* removed unidiomatic `()` from instance method links
* escaped `IMAP` in a few places
* converted all response structs to explicit classes: this makes much
  nicer rdoc output than listing them all under "constants"
* grouped flags constants into their own sections

9cd562ac84
2021-05-06 15:20:35 +09:00
nicholas a. evans
b0de2e7fe9
[ruby/net-imap] Move send_*_data into net/imap/command_data
Partially implements #10.

64d1080d63
2021-05-06 15:20:35 +09:00