在Mesos上运行k8s时，如何通过响应错误找到匹配的http请求

Question

我指的是在Mesos上开始使用Kubernetes，并尝试在同一本地Ubuntu上部署k8s和Mesos。执行以下命令：

$ km controller-manager 
  --master=${KUBERNETES_MASTER_IP}:8888 
  --cloud-provider=mesos 
  --cloud-config=./mesos-cloud.conf  
  --v=8 >controller.log 2>&1 &

来自controller.log:

$ more controller.log
I1210 05:33:13.570789   12082 controllermanager.go:217] Creating hostIP:hostPort endpoint controller
I1210 05:33:13.571253   12082 mesos.go:75] new mesos cloud, master='16.187.250.141:5050'
I1210 05:33:13.571294   12082 standalone.go:46] creating new standalone detector for &MasterInfo{Id:*master,Ip:*0,Port:*5050,Pid:*master@16
.187.250.141:5050,Hostname:*16.187.250.141,Version:nil,Address:nil,XXX_unrecognized:[],}
I1210 05:33:13.571469   12082 standalone.go:67] Detect()
I1210 05:33:13.571485   12082 standalone.go:69] spinning up asyc master detector poller
I1210 05:33:13.571501   12082 standalone.go:78] spawning asyc master detector listener
I1210 05:33:13.571608   12082 nodecontroller.go:132] Sending events to api server.
I1210 05:33:13.571706   12082 standalone.go:140] polling for master leadership at '16.187.250.141:5050'
I1210 05:33:13.571729   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/services
I1210 05:33:13.571757   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.571772   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.571831   12082 standalone.go:80] waiting for polled to send updates
I1210 05:33:13.571986   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/replicationcontrollers
I1210 05:33:13.572010   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572026   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
E1210 05:33:13.571994   12082 controllermanager.go:152] Failed to start service controller: the cloud provider does not support TCP load ba
lancers.
I1210 05:33:13.572124   12082 round_trippers.go:222] GET http://16.187.250.141:8888/apis/extensions/v1beta1/daemonsets
I1210 05:33:13.572151   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572167   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572208   12082 persistentvolume_claim_binder_controller.go:346] Starting PersistentVolumeClaimBinder
I1210 05:33:13.572262   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572268   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572300   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572314   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572305   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572331   12082 plugins.go:265] Loaded volume plugin "kubernetes.io/host-path"
I1210 05:33:13.572371   12082 plugins.go:265] Loaded volume plugin "kubernetes.io/nfs"
I1210 05:33:13.572393   12082 persistentvolume_recycler_controller.go:211] Starting PersistentVolumeRecycler
I1210 05:33:13.572452   12082 resource_quota_controller.go:133] Resource quota controller queued all resource quota for full calculation of
 usage
I1210 05:33:13.572479   12082 client.go:77] Reloading cached Mesos state
I1210 05:33:13.572283   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572499   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572346   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572508   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572531   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572535   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572547   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572571   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572584   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572521   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572612   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572632   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/namespaces
I1210 05:33:13.572662   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572676   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572684   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/resourcequotas
I1210 05:33:13.572702   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572714   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572756   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/persistentvolumeclaims
I1210 05:33:13.572551   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572791   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572815   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/persistentvolumes
I1210 05:33:13.572329   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572846   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572858   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572864   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/serviceaccounts?fieldSelector=metadata.name%3Dde
fault
I1210 05:33:13.572869   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/persistentvolumes
I1210 05:33:13.572885   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572892   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572777   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572948   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572167   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.573010   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573019   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572834   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573072   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572876   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573120   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572641   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/namespaces
I1210 05:33:13.573193   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573202   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.573286   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.573304   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573313   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.576806   12082 standalone.go:214] Got mesos state, content length 1795
I1210 05:33:13.577013   12082 standalone.go:147] detected leadership change from '<nil>' to 'master@16.187.250.141:5050'
I1210 05:33:13.577098   12082 standalone.go:185] master leader poller sleeping for 29.994633799s
I1210 05:33:13.577118   12082 standalone.go:88] detected master change: &MasterInfo{Id:*master,Ip:*0,Port:*5050,Pid:*master@16.187.250.141:
5050,Hostname:*16.187.250.141,Version:nil,Address:nil,XXX_unrecognized:[],}
I1210 05:33:13.577218   12082 client.go:155] cloud master changed to '16.187.250.141:5050'
I1210 05:33:14.192579   12082 round_trippers.go:247] Response Status: 403 Forbidden in 619 milliseconds
I1210 05:33:14.192610   12082 round_trippers.go:250] Response Headers:
I1210 05:33:14.192621   12082 round_trippers.go:253]     Proxy-Connection: Keep-Alive
E1210 05:33:14.192639   12082 statusupdater.go:68] Error listing slaves without kubelet: HTTP request failed with code 403: 403 Forbidden
I1210 05:33:14.192667   12082 round_trippers.go:253]     Connection: Keep-Alive
I1210 05:33:14.192692   12082 round_trippers.go:253]     Content-Length: 606
I1210 05:33:14.192703   12082 round_trippers.go:253]     Cache-Control: no-cache
I1210 05:33:14.192712   12082 round_trippers.go:253]     Pragma: no-cache
I1210 05:33:14.192722   12082 round_trippers.go:253]     Content-Type: text/html; charset=utf-8
I1210 05:33:14.192765   12082 round_trippers.go:247] Response Status: 403 Forbidden in 620 milliseconds
I1210 05:33:14.192800   12082 round_trippers.go:250] Response Headers:
I1210 05:33:14.192812   12082 round_trippers.go:253]     Pragma: no-cache
I1210 05:33:14.192823   12082 round_trippers.go:253]     Content-Type: text/html; charset=utf-8
I1210 05:33:14.192833   12082 round_trippers.go:253]     Proxy-Connection: Keep-Alive
I1210 05:33:14.192842   12082 round_trippers.go:253]     Connection: Keep-Alive
I1210 05:33:14.192851   12082 round_trippers.go:253]     Content-Length: 606
I1210 05:33:14.192861   12082 round_trippers.go:253]     Cache-Control: no-cache
I1210 05:33:14.192905   12082 request.go:804] Response Body: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
I1210 05:33:14.192935   12082 request.go:861] Response Body: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
I1210 05:33:14.192785   12082 request.go:804] Response Body: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
I1210 05:33:14.193025   12082 round_trippers.go:247] Response Status: 403 Forbidden in 620 milliseconds
I1210 05:33:14.193034   12082 request.go:861] Response Body: <HTML><HEAD>

我可以看到有一些"403"http响应错误。但是，我如何才能找到导致此响应错误的http请求？

Answer 1

经过调查，有两个步骤：

（1） 使用lsof查找http连接：

lsof -p pid -P -n

（2） 使用tcpdump捕获网络包：

sudo tcpdump -vv -A -s 0 'tcp port xxxx' -i em1 -w capture.pcap

然后分析软件包。