Posts

预处理 从一个文件中过滤 grep key file ➜ grep ERROR a.log 12:12 ERROR:core bad message 从多个文件中过滤 grep key file1 fil2 多文件搜索，指定多个文件 grep key *.log 使用正则的方式，匹配多个文件 grep -h key *.log 可以使用-h, 让结果中不出现文件名。默认文件名会出现在匹配行的前面。 ➜ grep ERROR a.log b.log a.log:12:12 ERROR:core bad message b.log:13:12 ERROR:core bad message ➜ grep ERROR *.log a.log:12:12 ERROR:core bad message b.log:13:12 ERROR:core bad message 多个关键词过滤 grep -e key1 -e key2 file 使用-e参数，可以制定多个关键词 ➜ grep -e ERROR -e INFO a.log 12:12 ERROR:core bad message 12:12 INFO:parse bad message1 正则过滤 grep -E REG file 下面例子是匹配db:后跟数字部分 ➜ grep -E "db:\d+ " a.log 12:14 WARNING:db:1 bad message 12:14 WARNING:db:21 bad message 12:14 WARNING:db:2 bad message1 12:14 WARNING:db:4 bad message 仅输出匹配字段 grep -o args 使用-o参数，可以仅仅输出匹配项，而不是整个匹配的行 ➜ go-tour grep -o -E "db:\d+ " a.log db:1 db:21 db:2 db:4 统计关键词出现的行数 例如一个nginx的access.log, 我们想统计其中的POST的个数，和OPTIONS的个数。 ...

最近遇到一个问题，WebSocket总是会在下午出现比较大的断开的量。 首先怀疑的是客户端的网络到服务端的网络出现抖动或者断开，要么就是入口的nginx有异常，或者是内部的服务出现异常。 排查下来，发现nginx的最大打开文件个数是1024 nginx master进程 nginx work进程 当进程打开文件数超过限制时，会发生什么？ 当进程超过最大打开文件限制时，会收到SIGXFSZ信号。这个信号会默认行为会杀死一个进程。进程内部也可以捕获这个信号。 我试着向nginx wrok进程发送SIGXFSZ信号, work进程会退出，然后master监听了这个事件后，会重新启动一个work进程。 kill -XFSZ work_pid 在nginx的error.log文件中，可以看到类似的日志输出。 这里的25就是XFSZ信号的整数表示。 ... [alert] ...#.: work process ... exited on signal 25 _ 参考 https://www.monitis.com/blog/6-best-practices-for-optimizing-your-nginx-performance/ https://www.cnblogs.com/shansongxian/p/9989631.html https://www.cnblogs.com/jpfss/p/9755706.html https://man7.org/linux/man-pages/man2/getrlimit.2.html https://man7.org/linux/man-pages/man5/proc.5.html

调研目的 在异常情况下，网络断开对WebSocket的影响 测试代码 测试代码没有心跳机制 心跳机制并不包含在WebSocket协议内部 var ws = new WebSocket('wss://echo.websocket.org/') ws.onopen =function(e){ console.log('onopen') } ws.onerror = function (e) { console.log('onerror: ' + e.code) console.log(e) } ws.onclose = function (e) { console.log('onclose: ' + e.code) console.log(e) } 场景1: 断网后，是否会立即触发onerror, 或者onclose事件？ 答案：不会立即触发 测试代码中没有心跳机制，断网后，并不会立即触发onerror或者onclose的回调函数。 个人测试的情况 及其 测试场景 Macbook pro chrome 83.0.4103.106 每隔10秒发送一次消息的情况下，40秒后出发onclose事件 Macbook pro chrome 83.0.4103.106 一直不发送消息，一直就不回出发onclose事件 Macbook pro chrome 83.0.4103.106 发出一个消息后？ 场景2: 断网后，使用send()发送数据，回触发事件吗？ 为什么无法准确拿到断开原因？ WebSocket关闭事件中有三个属性 code 断开原因码 reason 具体原因 wasClean 是否是正常断开 官方文档上，code字段有很多个值。但是大多数情况下，要么拿到的值是undefined, 要么是1006，基本上没有其他情况。 ...

新建一个文件 ip.list.cfg, 包含所有的带测试的ip地址。 192.168.40.20 192.168.40.21 执行命令: nohup fping -D -u -l -p 2000 -f ip.list.cfg & -D 显示时间戳 -u 显示不可达的目标 -l 持续的ping -p 每隔多少毫秒执行一次 -f 指定ip列表文件 在nohup.out中，回持续的显示到各个ip的网络状况。 [1592643928.961414] 192.168.40.20 : [0], 84 bytes, 3.22 ms (3.22 avg, 0% loss) [1592643928.969987] 192.168.40.21 : [0], 84 bytes, 1.22 ms (1.22 avg, 0% loss) [1592643930.965753] 192.168.40.20 : [1], 84 bytes, 5.25 ms (4.23 avg, 0% loss) [1592643930.972833] 192.168.40.21 : [1], 84 bytes, 1.14 ms (1.18 avg, 0% loss) [1592643932.965636] 192.168.40.20 : [2], 84 bytes, 3.45 ms (3.97 avg, 0% loss) [1592643932.978245] 192.168.40.21 : [2], 84 bytes, 4.39 ms (2.25 avg, 0% loss) [1592643934.991354] 192.168.40.20 : [3], 84 bytes, 27.9 ms (9.96 avg, 0% loss) [1592643934.991621] 192.168.40.21 : [3], 84 bytes, 14.9 ms (5.42 avg, 0% loss) [1592643936.978135] 192.168.40.20 : [4], 84 bytes, 11.3 ms (10.2 avg, 0% loss) [1592643936.979620] 192.168.40.21 : [4], 84 bytes, 1.37 ms (4.61 avg, 0% loss)

wireshark具有这个功能，但是并不适合做批量执行。 下面的方案比较适合批量执行。 # 1. 安装依赖 yum install gcc libpcap-devel libnet-devel sox -y # 2. 克隆源码 git clone https://github.com/wangduanduan/rtpsplit.git # 3. 切换目录 cd rtpsplit # 4. 编译可执行文件 make # 5. 将可执行文件复制到/usr/local/bin目录下 cp src/rtpbreak /usr/local/bin # 6. 切换到录音文件的目录，假如当前目录只有一个文件 rtpbreak -r krk9hprvin1u1laqe14g-8beffe8aaeb9bf99.pcap -g -m -d ./ audio git:(edge) ✗ rtpbreak -r krk9hprvin1u1laqe14g-8beffe8aaeb9bf99.pcap -g -m -d ./ + rtpbreak v1.3a running here! + pid: 1885, date/time: 01/05/2020#09:49:05 + Configuration + INPUT Packet source: rxfile 'krk9hprvin1u1laqe14g-8beffe8aaeb9bf99.pcap' Force datalink header length: disabled + OUTPUT Output directory: './' RTP raw dumps: enabled RTP pcap dumps: enabled Fill gaps: enabled Dump noise: disabled Logfile: './/rtp.0.txt' Logging to stdout: enabled Logging to syslog: disabled Be verbose: disabled + SELECT Sniff packets in promisc mode: enabled Add pcap filter: disabled Expecting even destination UDP port: disabled Expecting unprivileged source/destination UDP ports: disabled Expecting RTP payload type: any Expecting RTP payload length: any Packet timeout: 10.00 seconds Pattern timeout: 0.25 seconds Pattern packets: 5 + EXECUTION Running as user/group: root/root Running daemonized: disabled * You can dump stats sending me a SIGUSR2 signal * Reading packets... open di .//rtp.0.0.txt ! [rtp0] detected: pt=0(g711U) 192.168.40.192:26396 => 192.168.60.229:20000 open di .//rtp.0.1.txt ! [rtp1] detected: pt=0(g711U) 10.197.169.10:49265 => 192.168.60.229:20012 * eof reached. -- Caught SIGTERM signal (15), cleaning up... -- * [rtp1] closed: packets inbuffer=0 flushed=285 lost=0(0.00%), call_length=0m12s * [rtp0] closed: packets inbuffer=0 flushed=586 lost=0(0.00%), call_length=0m12s + Status Alive RTP Sessions: 0 Closed RTP Sessions: 2 Detected RTP Sessions: 2 Flushed RTP packets: 871 Lost RTP packets: 0 (0.00%) Noise (false positive) packets: 8 + No active RTP streams # 7. 查看输出文件 -rw-r--r--. 1 root root 185K May 1 09:22 krk9hprvin1u1laqe14g-8beffe8aaeb9bf99.pcap -rw-r--r--. 1 root root 132K May 1 09:49 rtp.0.0.pcap -rw-r--r--. 1 root root 92K May 1 09:49 rtp.0.0.raw -rw-r--r--. 1 root root 412 May 1 09:49 rtp.0.0.txt -rw-r--r--. 1 root root 52K May 1 09:49 rtp.0.1.pcap -rw-r--r--. 1 root root 33K May 1 09:49 rtp.0.1.raw -rw-r--r--. 1 root root 435 May 1 09:49 rtp.0.1.txt -rw-r--r--. 1 root root 1.7K May 1 09:49 rtp.0.txt # 8. 使用sox 转码以及合成wav文件 sox -r8000 -c1 -t ul rtp.0.0.raw -t wav 0.wav sox -r8000 -c1 -t ul rtp.0.1.raw -t wav 1.wav sox -m 0.wav 1.wav call.wav # 最终合成的 call.wav文件，就是可以放到浏览器中播放的双声道语音文件 参考 rtpbreak帮助文档 Copyright (c) 2007-2008 Dallachiesa Michele <micheleDOTdallachiesaATposteDOTit> rtpbreak v1.3a is free software, covered by the GNU General Public License. USAGE: rtpbreak (-r|-i) <source> [options] INPUT -r <str> Read packets from pcap file <str> -i <str> Read packets from network interface <str> -L <int> Force datalink header length == <int> bytes OUTPUT -d <str> Set output directory to <str> (def:.) -w Disable RTP raw dumps -W Disable RTP pcap dumps -g Fill gaps in RTP raw dumps (caused by lost packets) -n Dump noise packets -f Disable stdout logging -F Enable syslog logging -v Be verbose SELECT -m Sniff packets in promisc mode -p <str> Add pcap filter <str> -e Expect even destination UDP port -u Expect unprivileged source/destination UDP ports (>1024) -y <int> Expect RTP payload type == <int> -l <int> Expect RTP payload length == <int> bytes -t <float> Set packet timeout to <float> seconds (def:10.00) -T <float> Set pattern timeout to <float> seconds (def:0.25) -P <int> Set pattern packets count to <int> (def:5) EXECUTION -Z <str> Run as user <str> -D Run in background (option -f implicit) MISC -k List known RTP payload types -h This

简介 如果你的主机在公网上有端口暴露出去，那么总会有一些不怀好意的家伙，会尝试通过各种方式攻击你的机器。常见的服务例如ssh, nginx都会有类似的威胁。 手工将某个ip加入黑名单，这种操作太麻烦，而且效率低。而fail2ban就是一种自动化的解决方案。 fail2ban工作原理 fail2ban的工作原理是监控某个日志文件，然后根据某些关键词，提取出攻击方的IP地址，然后将其加入到黑名单。 fail2ban安装 yum install fail2ban -y # 如果找不到fail2ban包，就执行下面的命令 yum install epel-release # 安装fail2ban 完成后 systemctl enable fail2ban # 设置fail2ban开机启动 systemctl start fail2ban # 启动fail2ban systemctl status fail2ban # 查看fail2ban的运行状态 用fail2ban保护ssh fail2ban的配置文件位于/etc/fail2ban目录下。 在该目录下建立一个文件 jail.local, 内容如下 bantime 持续禁止多久 maxretry 最大多少次尝试 banaction 拦截后的操作 findtime 查找时间 看下下面的操作的意思是：监控sshd服务的最近10分钟的日志，如果某个ip在10分钟之内，有2次登录失败，就把这个ip加入黑名单, 24小时之后，这个ip才会被从黑名单中移除。 [DEFAULT] bantime = 24h banaction = iptables-multiport maxretry = 2 findtime = 10m [sshd] enabled = true 然后重启fail2ban, systemctl restart fail2ban fail2ban提供管理工具fail2ban-client **fail2ban-client status **显示fail2ban的状态 **fail2ban-client status sshd **显示某个监狱的配置。从下文的输出来看可以看出来fail2ban已经拦截了一些IP地址了 > fail2ban-client status Status |- Number of jail: 1 `- Jail list: sshd > fail2ban-client status sshd Status for the jail: sshd |- Filter | |- Currently failed: 2 | |- Total failed: 23289 | `- Journal matches: _SYSTEMD_UNIT=sshd.service + _COMM=sshd `- Actions |- Currently banned: 9 |- Total banned: 1270 `- Banned IP list: 93.174.93.10 165.22.238.92 23.231.25.234 134.255.219.207 77.202.192.113 120.224.47.86 144.91.70.139 90.3.194.84 217.182.89.87 fail2ban保护sshd的原理 fail2ban的配置文件目录下有个filter.d目录，该目录下有个sshd.conf的文件，这个文件就是对于sshd日志的过滤规则，里面有些正常时用来提取出恶意家伙的IP地址。 ...

思考题：当你用ssh登录到一个linux机器，并且执行了某个hello.sh之后，有哪些进程参与了该过程? linux系统架构 kernel mode user mode 内核态和用户态的区别 什么是进程 进程是运行的程序 process 是对 processor 虚拟化，通过时间片 进程都有uid nginx访问某个目录，Permission denied 进程都有pid $$ 进程都有父进程 准确来说，除了pid为0的进程之外，其他进程都有父进程 有时候，你用kill命令杀死了一个进程，但是立马你就发现这个进程又起来了。你就要看看，这个进程是不是有个非init进程的父进程。一般这个进程负责监控子进程，一旦子进程挂掉，就会去重新创建一个进程。所以你需要找到这个父进程的Id，先把父进程kill掉，然后在kill子进程。 进程是一棵树 #!/bin/bash echo "pid is $$" times=0 while true do sleep 2s; let times++; echo $times hello; done ➜ ~ pstree 24601 sshd─┬─3*[sshd───zsh] ├─sshd───zsh───pstree └─sshd───zsh───world.sh───sleep 进程都有生命周期 创建 销毁 进程都有状态 runing 进程占用CPU, 正在执行指令 ready 进程所有需要的资源都已经就绪，等待进入CPU执行 blocked 进程被某些事件阻断，例如IO。 进程的状态转移图 ...

相比于wireshark, RawCap非常小，仅有48kb。 使用RawCap命令需要使用管理员权限打开CMD，然后进入到RawCap.exe的目录。例如F:\Tools 显示网卡列表 输入RawCap.exe –help, 可以显示命令的使用帮助、网卡列表还有使用例子。 F:\Tools>RawCap.exe --help NETRESEC RawCap version 0.2.0.0 Usage: RawCap.exe [OPTIONS] <interface> <pcap_target> <interface> can be an interface number or IP address <pcap_target> can be filename, stdout (-) or named pipe (starting with \\.\pipe\) OPTIONS: -f Flush data to file after each packet (no buffer) -c <count> Stop sniffing after receiving <count> packets -s <sec> Stop sniffing after <sec> seconds -m Disable automatic creation of RawCap firewall entry -q Quiet, don't print packet count to standard out INTERFACES: 0. IP : 169.254.63.243 NIC Name : Local Area Connection NIC Type : Ethernet 1. IP : 192.168.1.129 NIC Name : WiFi NIC Type : Wireless80211 2. IP : 127.0.0.1 NIC Name : Loopback Pseudo-Interface 1 NIC Type : Loopback 3. IP : 10.165.240.132 NIC Name : Mobile 12 NIC Type : Wwanpp Example 1: RawCap.exe 0 dumpfile.pcap Example 2: RawCap.exe -s 60 127.0.0.1 localhost.pcap Example 3: RawCap.exe 127.0.0.1 \\.\pipe\RawCap Example 4: RawCap.exe -q 127.0.0.1 - | Wireshark.exe -i - -k :::warning 注意： ...

下文的论述都以下面的配置为例子 location ^~ /p/security { rewrite /p/security/(.*) /security/$1 break; proxy_pass http://security:8080; proxy_redirect off; proxy_set_header Host $host; add_header 'Access-Control-Allow-Origin' '*' always; add_header 'Access-Control-Allow-Credentials' 'true' always; } 如果dns无法解析，nginx则无法启动 security如果无法解析，那么nginx则无法启动 DNS缓存问题： nginx启动时，如果将security dns解析为1.2.3.4。如果security的ip地址变了。nginx不会自动解析新的ip地址，导致反向代理报错504。 反向代理的DNS缓存问题务必重视 跨域头配置的always 反向代理一般都是希望允许跨域的。如果不加always，那么只会对成功的请求加跨域头，失败的请求则不会。 关于**‘Access-Control-Allow-Origin’ ‘*’，如果后端服务本身就带有这个头，那么如果你在nginx中再添加这个头，就会在浏览器中遇到下面的报错。而解决办法就是不要在nginx中设置这个头。** Access to fetch at 'http://192.168.40.107:31088/p/security/v2/login' from origin 'http://localhost:5000' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: The 'Access-Control-Allow-Origin' header contains multiple values '*, *', but only one is allowed. Have the server send the header with a valid value, or, if an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled. 参考链接 http://nginx.org/en/docs/http/ngx_http_headers_module.html http://www.hxs.biz/html/20180425122255.html https://blog.csdn.net/xiojing825/article/details/83383524 https://cloud.tencent.com/developer/article/1470375 https://blog.csdn.net/bbg221/article/details/79886979

apt-get install sox libsox-fmt-mp3 -y sox input.vox output.mp3 sox支持命令 ➜ vox sox --help sox: SoX v14.4.1 Usage summary: [gopts] [[fopts] infile]... [fopts] outfile [effect [effopt]]... SPECIAL FILENAMES (infile, outfile): - Pipe/redirect input/output (stdin/stdout); may need -t -d, --default-device Use the default audio device (where available) -n, --null Use the `null' file handler; e.g. with synth effect -p, --sox-pipe Alias for `-t sox -' SPECIAL FILENAMES (infile only): "|program [options] ..." Pipe input from external program (where supported) http://server/file Use the given URL as input file (where supported) GLOBAL OPTIONS (gopts) (can be specified at any point before the first effect): --buffer BYTES Set the size of all processing buffers (default 8192) --clobber Don't prompt to overwrite output file (default) --combine concatenate Concatenate all input files (default for sox, rec) --combine sequence Sequence all input files (default for play) -D, --no-dither Don't dither automatically --effects-file FILENAME File containing effects and options -G, --guard Use temporary files to guard against clipping -h, --help Display version number and usage information --help-effect NAME Show usage of effect NAME, or NAME=all for all --help-format NAME Show info on format NAME, or NAME=all for all --i, --info Behave as soxi(1) --input-buffer BYTES Override the input buffer size (default: as --buffer) --no-clobber Prompt to overwrite output file -m, --combine mix Mix multiple input files (instead of concatenating) --combine mix-power Mix to equal power (instead of concatenating) -M, --combine merge Merge multiple input files (instead of concatenating) --magic Use `magic' file-type detection --multi-threaded Enable parallel effects channels processing --norm Guard (see --guard) & normalise --play-rate-arg ARG Default `rate' argument for auto-resample with `play' --plot gnuplot|octave Generate script to plot response of filter effect -q, --no-show-progress Run in quiet mode; opposite of -S --replay-gain track|album|off Default: off (sox, rec), track (play) -R Use default random numbers (same on each run of SoX) -S, --show-progress Display progress while processing audio data --single-threaded Disable parallel effects channels processing --temp DIRECTORY Specify the directory to use for temporary files -T, --combine multiply Multiply samples of corresponding channels from all input files (instead of concatenating) --version Display version number of SoX and exit -V[LEVEL] Increment or set verbosity level (default 2); levels: 1: failure messages 2: warnings 3: details of processing 4-6: increasing levels of debug messages FORMAT OPTIONS (fopts): Input file format options need only be supplied for files that are headerless. Output files will have the same format as the input file where possible and not overriden by any of various means including providing output format options. -v|--volume FACTOR Input file volume adjustment factor (real number) --ignore-length Ignore input file length given in header; read to EOF -t|--type FILETYPE File type of audio -e|--encoding ENCODING Set encoding (ENCODING may be one of signed-integer, unsigned-integer, floating-point, mu-law, a-law, ima-adpcm, ms-adpcm, gsm-full-rate) -b|--bits BITS Encoded sample size in bits -N|--reverse-nibbles Encoded nibble-order -X|--reverse-bits Encoded bit-order --endian little|big|swap Encoded byte-order; swap means opposite to default -L/-B/-x Short options for the above -c|--channels CHANNELS Number of channels of audio data; e.g. 2 = stereo -r|--rate RATE Sample rate of audio -C|--compression FACTOR Compression factor for output format --add-comment TEXT Append output file comment --comment TEXT Specify comment text for the output file --comment-file FILENAME File containing comment text for the output file --no-glob Don't `glob' wildcard match the following filename AUDIO FILE FORMATS: 8svx aif aifc aiff aiffc al amb amr-nb amr-wb anb au avr awb caf cdda cdr cvs cvsd cvu dat dvms f32 f4 f64 f8 fap flac fssd gsm gsrt hcom htk ima ircam la lpc lpc10 lu mat mat4 mat5 maud mp2 mp3 nist ogg paf prc pvf raw s1 s16 s2 s24 s3 s32 s4 s8 sb sd2 sds sf sl sln smp snd sndfile sndr sndt sou sox sph sw txw u1 u16 u2 u24 u3 u32 u4 u8 ub ul uw vms voc vorbis vox w64 wav wavpcm wv wve xa xi PLAYLIST FORMATS: m3u pls AUDIO DEVICE DRIVERS: alsa EFFECTS: allpass band bandpass bandreject bass bend biquad chorus channels compand contrast dcshift deemph delay dither divide+ downsample earwax echo echos equalizer fade fir firfit+ flanger gain highpass hilbert input# ladspa loudness lowpass mcompand mixer* noiseprof noisered norm oops output# overdrive pad phaser pitch rate remix repeat reverb reverse riaa silence sinc spectrogram speed splice stat stats stretch swap synth tempo treble tremolo trim upsample vad vol * Deprecated effect + Experimental effect # LibSoX-only effect EFFECT OPTIONS (effopts): effect dependent; see --help-effect 参考 http://sox.sourceforge.net/sox.html#OPTIONS