rewrite regression test for proxy to use a local proxy using webrick

bugfix: fix wrong super call for unexisting super method, when using the datadog plugin
updated cheatsheet examples
2025-08-15 00:03:10 -04:00 · 2023-06-26 20:19:39 +01:00 · 2023-06-26 19:31:51 +01:00 · 2023-06-26 19:27:48 +01:00 · 2023-06-26 18:24:08 +00:00 · 2023-06-26 16:45:27 +01:00
12 changed files with 130 additions and 28 deletions
--- a/.rubocop_todo.yml
+++ b/.rubocop_todo.yml
@ -38,4 +38,8 @@ Naming/AccessorMethodName:
  Enabled: false

 Performance/MethodObjectAsBlock:
+  Enabled: false
+
+# TODO: remove after https://github.com/rubocop/rubocop/issues/11980 is fixed
+Style/RedundantCurrentDirectoryInPath:
  Enabled: false
--- a/CHEATSHEET.md
+++ b/CHEATSHEET.md
@ -24,7 +24,7 @@ require "httpx"

 uri = "https://google.com"

-responses = HTTPX.new(uri, uri)
+responses = HTTPX.get(uri, uri)

 # OR
 HTTPX.wrap do |client|
@ -37,7 +37,7 @@ end
 ## Headers

 ```ruby
-HTTPX.headers("user-agent" => "My Ruby Script").get("https://google.com")
+HTTPX.with(headers: {"user-agent" => "My Ruby Script"}).get("https://google.com")
 ```

 ## HTTP Methods
@ -61,7 +61,7 @@ response = HTTPX.plugin(:basic_authentication).basic_authentication("username",
 # Digest Auth
 response = HTTPX.plugin(:digest_authentication).digest_authentication("username", "password").get("https://google.com")

-# Token Auth
+# Bearer Token Auth
 response = HTTPX.plugin(:authentication).authentication("eyrandomtoken").get("https://google.com")
 ```

@ -74,11 +74,11 @@ require "httpx"
 response = HTTPX.get("https://google.com/")
 response.status # => 301
 response.headers["location"] #=> "https://www.google.com/"
-response.body             # =>  "<HTML><HEAD><meta http-equiv=\"content-type\" ....
-response["cache-control"] # => public, max-age=2592000
+response.headers["cache-control"] #=> public, max-age=2592000
+response.body.to_s           #=>  "<HTML><HEAD><meta http-equiv=\"content-type\" ....
 ```

-## POST form request
+## POST `application/x-www-form-urlencoded` request

 ```ruby
 require "httpx"
@ -88,16 +88,13 @@ uri = URI.parse("http://example.com/search")
 response = HTTPX.post(uri, form: {"q" => "My query", "per_page" => "50"})
 ```

-## File upload - input type="file" style
+## File `multipart/form-data` upload - input type="file" style

 ```ruby
 require "httpx"

-# uses http_form_data API: https://github.com/httprb/form_data
-
-path = "/path/to/your/testfile.txt"
 HTTPX.plugin(:multipart).post("http://something.com/uploads", form: {
-  name: HTTP::FormData::File.new(path)
+  name: Pathname.new("/path/to/your/testfile.txt")
 })
 ```

@ -132,8 +129,7 @@ require "httpx"

 HTTPX.plugin(:cookies).wrap do |client|
  session_response = client.get("https://translate.google.com/")
-  response_cookies = session_response.cookie_jar
-  response = client.cookies(response_cookies).get("https://translate.google.com/#auto|en|Pardon")
+  response = client.get("https://translate.google.com/#auto|en|Pardon")
  puts response
 end
 ```
@ -144,7 +140,7 @@ end
 require "httpx"

 response = HTTPX.plugin(:compression).get("https://www.google.com")
-puts response.headers["content-encoding"] #=> "gzip" 
+puts response.headers["content-encoding"] #=> "gzip"

 ```

@ -183,7 +179,9 @@ HTTPX.with(resolver_class: :https, resolver_options: {uri: "https://9.9.9.9/dns-
 ```ruby
 require "httpx"

-HTTPX.plugin(:follow_redirects).with(follow_insecure_redirects: false, max_redirects: 4).get("https://www.google.com")
+HTTPX.plugin(:follow_redirects)
+     .with(follow_insecure_redirects: false, max_redirects: 4)
+     .get("https://www.google.com")
 ```

 ## Timeouts
@ -191,12 +189,12 @@ HTTPX.plugin(:follow_redirects).with(follow_insecure_redirects: false, max_redir
 ```ruby
 require "httpx"

-HTTPX.with(timeout: {connect_timeout: 10, operation_timeout: 3}).get("https://google.com")
+# full E2E request/response timeout, 10 sec to connect to peer
+HTTPX.with(timeout: {connect_timeout: 10, request_timeout: 3}).get("https://google.com")
 ```

 ## Retries

-
 ```ruby
 require "httpx"
 HTTPX.plugin(:retries).max_retries(5).get("https://www.google.com")
@ -214,4 +212,3 @@ HTTPX.get("https://google.com") #=>  udp://10.0.1.2:53...

 HTTPX.with(debug_level: 1, debug: $stderr).get("https://google.com")
 ```
-
--- a/httpx.gemspec
+++ b/httpx.gemspec
@ -23,7 +23,7 @@ Gem::Specification.new do |gem|
    "changelog_uri" => "https://os85.gitlab.io/httpx/#release-notes",
    "documentation_uri" => "https://os85.gitlab.io/httpx/rdoc/",
    "source_code_uri" => "https://gitlab.com/os85/httpx",
-    "homepage_uri" => "https://os85.gitlab.io/httpx/",
+    "homepage_uri" => "https://honeyryderchuck.gitlab.io/httpx/",
    "rubygems_mfa_required" => "true",
  }

--- a/lib/httpx/adapters/datadog.rb
+++ b/lib/httpx/adapters/datadog.rb
@ -160,7 +160,7 @@ module TRACING_MODULE # rubocop:disable Naming/ClassAndModuleCamelCase

        module RequestMethods
          def __datadog_enable_trace!
-            return super if @__datadog_enable_trace
+            return if @__datadog_enable_trace

            RequestTracer.new(self).call
            @__datadog_enable_trace = true
--- a/lib/httpx/domain_name.rb
+++ b/lib/httpx/domain_name.rb
@ -63,7 +63,7 @@ module HTTPX
      # Normalizes a _domain_ using the Punycode algorithm as necessary.
      # The result will be a downcased, ASCII-only string.
      def normalize(domain)
-        domain = domain.ascii_only? ? domain : domain.chomp(DOT).unicode_normalize(:nfc)
+        domain = domain.chomp(DOT).unicode_normalize(:nfc) unless domain.ascii_only?
        Punycode.encode_hostname(domain).downcase
      end
    end
--- a/lib/httpx/plugins/proxy.rb
+++ b/lib/httpx/plugins/proxy.rb
@ -25,6 +25,10 @@ module HTTPX
          klass.plugin(:"proxy/socks5")
        end

+        def extra_options(options)
+          options.merge(supported_proxy_protocols: [])
+        end
+
        if URI::Generic.methods.include?(:use_proxy?)
          def use_proxy?(*args)
            URI::Generic.use_proxy?(*args)
@ -118,6 +122,12 @@ module HTTPX
        def option_proxy(value)
          value.is_a?(Parameters) ? value : Hash[value]
        end
+
+        def option_supported_proxy_protocols(value)
+          raise TypeError, ":supported_proxy_protocols must be an Array" unless value.is_a?(Array)
+
+          value.map(&:to_s)
+        end
      end

      module InstanceMethods
@ -142,8 +152,12 @@ module HTTPX
            next_proxy = @_proxy_uris.first
            raise Error, "Failed to connect to proxy" unless next_proxy

+            next_proxy = URI(next_proxy)
+
+            raise Error,
+                  "#{next_proxy.scheme}: unsupported proxy protocol" unless options.supported_proxy_protocols.include?(next_proxy.scheme)
+
            if proxy.key?(:no_proxy)
-              next_proxy = URI(next_proxy)

              no_proxy = proxy[:no_proxy]
              no_proxy = no_proxy.join(",") if no_proxy.is_a?(Array)
@ -253,10 +267,11 @@ module HTTPX
        end

        def send(request)
-          return super unless @options.proxy
-          return super unless connecting?
+          return super unless (
+            @options.proxy && @state != :idle && connecting?
+          )

-          @pending << request
+          (@proxy_pending ||= []) << request
        end

        def connecting?
@ -294,6 +309,12 @@ module HTTPX
          when :idle
            transition(:connecting)
          when :connected
+            if @proxy_pending
+              while (req = @proxy_pendind.shift)
+                send(req)
+              end
+            end
+
            transition(:open)
          end
        end
--- a/lib/httpx/plugins/proxy/http.rb
+++ b/lib/httpx/plugins/proxy/http.rb
@ -6,6 +6,12 @@ module HTTPX
  module Plugins
    module Proxy
      module HTTP
+        class << self
+          def extra_options(options)
+            options.merge(supported_proxy_protocols: options.supported_proxy_protocols + %w[http])
+          end
+        end
+
        module InstanceMethods
          def with_proxy_basic_auth(opts)
            with(proxy: opts.merge(scheme: "basic"))
--- a/lib/httpx/plugins/proxy/socks4.rb
+++ b/lib/httpx/plugins/proxy/socks4.rb
@ -16,6 +16,12 @@ module HTTPX

        Error = Socks4Error

+        class << self
+          def extra_options(options)
+            options.merge(supported_proxy_protocols: options.supported_proxy_protocols + PROTOCOLS)
+          end
+        end
+
        module ConnectionMethods
          def interests
            if @state == :connecting
--- a/lib/httpx/plugins/proxy/socks5.rb
+++ b/lib/httpx/plugins/proxy/socks5.rb
@ -18,8 +18,14 @@ module HTTPX

        Error = Socks5Error

-        def self.load_dependencies(*)
-          require_relative "../authentication/socks5"
+        class << self
+          def load_dependencies(*)
+            require_relative "../authentication/socks5"
+          end
+
+          def extra_options(options)
+            options.merge(supported_proxy_protocols: options.supported_proxy_protocols + %w[socks5])
+          end
        end

        module ConnectionMethods
--- a/lib/httpx/session_extensions.rb
+++ b/lib/httpx/session_extensions.rb
@ -9,10 +9,13 @@ module HTTPX
    # redefine the default options static var, which needs to
    # refresh options_class
    options = proxy_session.class.default_options.to_hash
-    options.freeze
    original_verbosity = $VERBOSE
    $VERBOSE = nil
+    const_set(:Options, proxy_session.class.default_options.options_class)
+    options[:options_class] = Class.new(options[:options_class])
+    options.freeze
    Options.send(:const_set, :DEFAULT_OPTIONS, options)
+    Session.instance_variable_set(:@default_options, Options.new(options))
    $VERBOSE = original_verbosity
  end

--- a/regression_tests/bug_0_24_1_test.rb
+++ b/regression_tests/bug_0_24_1_test.rb
@ -0,0 +1,52 @@
+# frozen_string_literal: true
+
+require "webrick"
+require "webrick/httpproxy"
+require "test_helper"
+require "support/http_helpers"
+require "support/proxy_helper"
+require "support/minitest_extensions"
+
+class Bug_0_24_1_Test < Minitest::Test
+  include HTTPHelpers
+  include ProxyHelper
+
+  Plugin = Module.new do
+    @requests = []
+
+    class << self
+      attr_accessor :requests
+    end
+
+    self::ConnectionMethods = Module.new do
+      def send(req)
+        Plugin.requests << req
+        super
+      end
+    end
+  end
+
+  def test_proxy_plugin_silencing_conn_send_based_plugin
+    start_test_servlet(WEBrick::HTTPProxyServer) do |server|
+      def server.origin
+        sock = listeners.first
+        _, sock, ip, _ = sock.addr
+        "http://#{ip}:#{sock}"
+      end
+      proxy_uri = server.origin
+      http = HTTPX.plugin(Plugin).plugin(:proxy).plugin(ProxyResponseDetector).with_proxy(uri: proxy_uri)
+      uri = build_uri("/get")
+      response = http.get(uri)
+      verify_status(response, 200)
+      assert response.proxied?
+
+      assert Plugin.requests.size == 1
+    end
+  end
+
+  private
+
+  def scheme
+    "http://"
+  end
+end
--- a/test/proxy_test.rb
+++ b/test/proxy_test.rb
@ -28,6 +28,13 @@ class ProxyTest < Minitest::Test
    end
  end

+  def test_proxy_unsupported_scheme
+    ex = assert_raises(HTTPX::HTTPProxyError) do
+      HTTPX.plugin(:proxy).with_proxy(uri: "https://proxy:123").get("http://smth.com")
+    end
+    assert ex.message == "https: unsupported proxy protocol"
+  end
+
  private

  def parameters(uri: "http://proxy", **args)
Author	SHA1	Message	Date
HoneyryderChuck	e4338979a6	rewrite regression test for proxy to use a local proxy using webrick	2023-06-26 20:19:39 +01:00
HoneyryderChuck	85f0ac8ed3	bugfix: fix wrong super call for unexisting super method, when using the datadog plugin	2023-06-26 19:31:51 +01:00
HoneyryderChuck	e25ac201d2	updated cheatsheet examples	2023-06-26 19:27:48 +01:00
HoneyryderChuck	38b871aa8e	Merge branch 'issue-238' into 'master' proxy: fix incorrect connection #send definition never calling super Closes #238 See merge request os85/httpx!262	2023-06-26 18:24:08 +00:00
HoneyryderChuck	0b18bb63e8	lint issue	2023-06-26 16:45:27 +01:00
HoneyryderChuck	afbde420a7	proxy: fix incorrect connection #send definition never calling super this made several plugins unusable with the proxy plugin, because a lot of them are dependent on Connection#send being called and overwritten. This was done so to avoid piping requests when intermediate connect-level parsers are in place. So in the way, when the conn is initial, original send is closed; when not, which should almost never happen, a second list is created, which is then piped when the connection is established, back to original send.	2023-06-26 16:45:27 +01:00
HoneyryderChuck	244563720a	proxy plugin: fail early if proxy scheme is not supported The error which it was appearing when erroneously using "https" in a proxy url was too cryptic.	2023-06-26 15:54:16 +01:00
HoneyryderChuck	886c091901	Merge branch 'fix-rubygems-website' into 'master' Fix wrong homepage URL in rubygems See merge request os85/httpx!263	2023-06-26 11:58:13 +00:00
Arash Mousavi	11942b2c74	Update file httpx.gemspec	2023-06-26 10:59:57 +00:00
HoneyryderChuck	b2848ea718	remove Style/RedundantCurrentDirectoryInPath cop support due to upstream bug	2023-06-25 01:22:50 +01:00