问题描述:
php开发新增定时任务每天中午12点都会通过并发curl请求https://**facebook.com/*** ,导致系统cpu负载极高,系统告警了呀
问题排查:
1、跟开发协商将并发的curl数量降低,但是问题依旧。那会是什么问题呢
2、用strace追踪curl执行吧,看看都干了什么
A、先来统计一下系统调用 ,发现一堆access系统调用,而且接近全部都是错误的,尴尬了
1 |
strace -T -s 1024 -c curl "https://graph.facebook.com/v2.6/me/messages?*********" |
B、继续strace追踪看看access什么东西,发现一堆这些 /etc/pki/nssdb/*** , 如下所示
1 |
strace -T -s 1024 -e access curl "https://graph.facebook.com/v2.6/*****" |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
access("/etc/pki/nssdb/.2438912988_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000016> access("/etc/pki/nssdb/.2438912989_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438912990_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438912991_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438912992_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000014> access("/etc/pki/nssdb/.2438912993_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438912994_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000014> access("/etc/pki/nssdb/.2438912995_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438912996_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438912997_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438912998_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438912999_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438913000_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438913001_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438913002_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438913003_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438913004_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438913005_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000014> access("/etc/pki/nssdb/.2438913006_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438913007_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438913008_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438913009_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438913010_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438913011_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/etc/pki/nssdb/.2438913012_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000014> access("/etc/pki/nssdb/cert9.db", F_OK) = 0 <0.000014> access("/etc/pki/nssdb/cert9.db-journal", F_OK) = -1 ENOENT (No such file or directory) <0.000014> access("/etc/pki/nssdb/cert9.db-wal", F_OK) = -1 ENOENT (No such file or directory) <0.000014> access("/var/tmp", R_OK|W_OK|X_OK) = 0 <0.000015> access("/var/tmp/etilqs_F0dXOKTvM4QXZ2y", F_OK) = -1 ENOENT (No such file or directory) <0.000017> access("/var/tmp/.2438911946_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000014> access("/var/tmp/.2438911947_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/var/tmp/.2438911948_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/var/tmp/.2438911949_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/var/tmp/.2438911950_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/var/tmp/.2438911951_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000016> access("/var/tmp/.2438911952_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000014> access("/var/tmp/.2438911953_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/var/tmp/.2438911954_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/var/tmp/.2438911955_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000014> access("/var/tmp/.2438911956_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000014> access("/var/tmp/.2438911957_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/var/tmp/.2438911958_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/var/tmp/.2438911959_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000016> access("/var/tmp/.2438911960_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/var/tmp/.2438911961_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/var/tmp/.2438911962_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> access("/var/tmp/.2438911963_dOeSnotExist_.db", F_OK) = -1 ENOENT (No such file or directory) <0.000015> ...... |
NSS 是什么 ,NSS 是开源软件,和 OpenSSL 一样,是一个底层密码学库,包括 TLS 实现。网上很多反馈NSS内存泄漏,或者其他低版本的问题。
C、通过 curl –version 查看版本,会发现有个NSS
1 2 3 4 5 |
curl --version ## curl 7.20.0 (x86_64-unknown-linux-gnu) libcurl/7.53.1 NSS/3.28.4 zlib/1.2.8 libidn2/0.16 libpsl/0.6.2 (+libicu/50.1.2) libssh2/1.4.2 nghttp2/1.21.1 Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp Features: AsynchDNS IDN IPv6 Largefile NTLM SPNEGO SSL libz |
既然NSS有很多问题,那就把NSS换成opensSSL吧,重新编译curl ,编译参数加上 –without-nss 去掉NSS即可,加上 –with-ssl 修改为openSSL 。当然你要下载curl的软件包吧,自己下吧。编译参数如下
1 |
./configure --prefix=/usr/local --without-nss --with-ssl --enable-file --enable-http --enable-ftp --enable-ipv6 && make && make install |
再次curl –version 查看版本,会发现NSS被替换成了openSSL了
1 2 3 4 5 6 |
curl --version ### curl 7.20.0 (x86_64-unknown-linux-gnu) libcurl/7.20.0 OpenSSL/1.0.2k zlib/1.2.8 libidn/1.18 Protocols: dict file ftp ftps http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp Features: IDN IPv6 Largefile NTLM SSL libz |
最后检查并发curl也正常了
注意:curl都重新编译了,记得php也要重新编译哦