The read action immediately returned with an error “ i/o deadline reached”. With this, we were able to pinpoint the error that, after the reverse proxy finishes the upgrade of the connection to SPDY when it tries to set up the pipe between the FRP server and the apiserver, it can not read from the FRP server in spc.copyToBackend(). So we make a customized version of the ReverseProxy with more debugging logs. The plugin uses the default httputil.ReverseProxy does not have the capability to add connection-level debug logging. It is used to start an HTTPS connection with valid certificates matching the Kubernetes apiserver endpoint domain name. The plugin is nothing but yet another reverse proxy. With more debugging on both FRP server and client, we put our focus on the https2https plugin within the FRP client. So the next question is, is it that the FRP server does not receive the request, or that the FRP server can not get a response from the backend so can’t respond to kubectl? Now we know that kubectl sends out the stream creation request, but it doesn't get a response. Port-forwarding is a component of most SSH client and server programs. This method is regularly used to circumvent standard firewall security protocols. However, after receiving the upgrade response, there is no more progress, no more verbose logging, then it either stuck, then timed out or immediately errored out.Įxample verbose logging with port-forwarding like below: The encrypted SSH ‘tunnel’ serves as a vessel to transfer assorted data and deliver it safely to the remote system. More verbose logging shows that, after making an HTTP POST /exec call to the apiserver, kubectl receives the HTTP response with the correct status 101 switching protocol to SPDY, together with correct response headers. Kubectl port-forward and attach have similar symptoms. What we saw was, kubectl exec either stuck without printing anything, then timed out, or errored out immediately with some io error. Issue: kubectl exec/attach/port-forward doesn’t work The upgrade will be done across all the HTTP/1.x connections. When kubectl starts exec, apiserver will try to upgrade the HTTP/1.x connection to SPDY, which is a deprecated binary streaming protocol. The FRP client uses an https2https plugin to call apiserver to make sure the apiserver certificates match the domain name. The SSH tunnel is setup in a PowerShell job called Kubectl-Tunnel and can be found by running Get-Job. The client will initiate and set up a TLS tunnel between the server. Create a Kubectl SSH tunnel to the managed clusters dashboard. FRP server has a publicly accessible endpoint, and the FRP client runs within the private network. Defaults to the operating system username.The following diagram shows the overall architecture. Useful for auditing operations executed by 3rd party tools. user string Specifies the user executing the operation. stderrthreshold severity logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=false) (default 2) skip_log_headers If true, avoid headers when opening log files (no effect when -logtostderr=true) skip_headers If true, avoid header prefixes in the log messages skip-audit Skip recording the current command in the audit logs. rootless Force to use rootless driver (docker and podman driver only) This can be set to allow having multiple instances of minikube independently. The user can create a websocket backed network tunnel to a port inside the. p, -profile string The name of the minikube VM being used. Place SSH public key into a Secret kubectl create secret generic my-pub-key. one_output If true, only write logs to their native severity level (vs also writing to each lower severity level no effect when -logtostderr=true) logtostderr log to standard error instead of files If the value is 0, the maximum file size is unlimited. log_file_max_size uint Defines the maximum size a log file can grow to (no effect when -logtostderr=true). log_file string If non-empty, use this log file (no effect when -logtostderr=true) log_dir string If non-empty, write log files in this directory (no effect when -logtostderr=true) log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0) b, -bootstrapper string The name of the cluster bootstrapper that will set up the Kubernetes cluster. alsologtostderr log to standard error as well as files (no effect when -logtostderr=true) Options inherited from parent commands -add_dir_header If true, adds the file directory to the header of the log messages Useful for the machine drivers when they will not start with 'Waiting for SSH'. Set to 'false' to use the command line 'ssh' command when accessing the docker machine. Options -native-ssh Use native Golang SSH client (default true).
0 Comments
Leave a Reply. |