基于10.2.0.1 rac deadlock死锁 trace file分析_增补二
结论
1,oracle 10g rac如果在2个节点间发生死锁，会记录到告警日志中2,从告警日志可以找到对应的死锁的TRACE FILE
3，死锁的TRACE FILE内容构成
第一部分：
略去，因为和死锁内容无关
第二部分：
这是我们基于RAC死锁最为关心的信息
*** 2015-11-09 22:04:01.002
user session for deadlock lock 0x82f5c810
pid=18 serial=57 audsid=26 user: 27/TBS_ZXY --发生死锁会话对应的进程号以及用户
O/S info: user: oracle, term: pts/2, ospid: 6050, machine: jingfa1 --源于V$SESSION
program: sqlplus@jingfa1 (TNS V1-V3)
application name: SQL*Plus, hash value=3669949024
Current SQL Statement:
update t_lock set a=22 where a=2 --发生死锁会话正在执行的SQL
小结下：第二部分包括死锁会话的一些相关信息，即死锁会话SID及进程号以及终端信息，还在正在执行的SQL
第三部分：
----可见如下on [0xa0006][0x7c],[TX] 含义为[0xa0006]对应xidusn+xidslot，而0x7c]对应xidsqn
ENQUEUE DUMP REQUEST: from 1.22803 on [0xa0006][0x7c],[TX] for reason 3 mtype 0
DUMP LOCAL BLOCKER/HOLDER: block level 5 res [0xa0006][0x7c],[TX] --同上
----------resource 0x0x7e51ac20----------------------
resname : [0xa0006][0x7c],[TX]
Local node : 0
dir_node : 0
master_node : 0
hv idx : 86
hv last r.inc : 4
current inc : 4
hv status : 0
hv master : 0
open options : dd
grant_bits : KJUSERNL KJUSEREX
grant mode : KJUSERNL KJUSERCR KJUSERCW KJUSERPR KJUSERPW KJUSEREX
count : 1 0 0 0 0 1
val_state : KJUSERVS_NOVALUE
valblk : 0x00000000000000000000000000000000 .
access_node : 0
vbreq_state : 0
state : x0
resp : 0x7e51ac20
On Scan_q? : N
Total accesses: 67
Imm. accesses: 58
Granted_locks : 1
Cvting_locks : 1
value_block: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
小结：
重点关注上述的信息：ENQUEUE DUMP REQUEST: from 1.22803 on [0xa0006][0x7c],[TX] for reason 3 mtype 0，对应等待者会话的XID相关的信息，即发生死锁会话的XID
至于其它信息，过于INTERNAL，还需要研究
第四部分
可见global blockers dump start即死锁的持锁者事务的信息为如下信息：
即res [0x110022][0x11],[TX]对应持锁会话,具体为res [0x110022]对应xidusn + xidslot,而[0x11]对应xidsqn
也就是可以根据如下，把16进制转换为10进制，然后到GV$TRANSACTION去匹配对应的会话，最终由GV$SESSION找到持锁会话的信息，进行针对性处理即可
Global blockers dump start:---------------------------------
DUMP LOCAL BLOCKER/HOLDER: block level 5 res [0x110022][0x11],[TX]
----------resource 0x0x7f32f540----------------------
resname : [0x110022][0x11],[TX]
Local node : 0
dir_node : 1
master_node : 1
hv idx : 7
hv last r.inc : 2
current inc : 4
hv status : 0
hv master : 1
open options : dd
Held mode : KJUSERNL
Cvt mode : KJUSEREX
Next Cvt mode : KJUSERNL
msg_seq : 0x1
res_seq : 3
grant_bits : KJUSERNL
grant mode : KJUSERNL KJUSERCR KJUSERCW KJUSERPR KJUSERPW KJUSEREX
count : 1 0 0 0 0 0
val_state : KJUSERVS_NOVALUE
valblk : 0x00000000000000000000000000000000 .
access_node : 1
vbreq_state : 0
state : x8
resp : 0x7f32f540
On Scan_q? : N
Total accesses: 29
Imm. accesses: 22
Granted_locks : 0
Cvting_locks : 1
value_block: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
GRANTED_Q :
CONVERT_Q:
lp 0x82f5c810 gl KJUSERNL rl KJUSEREX rp 0x7f32f540 [0x110022][0x11],[TX]
master 1 gl owner 0x83b4fe90 possible pid 16602 xid 12000-0001-00000035 bast 0 rseq 3 mseq 0 history 0x1495149a
convert opt KJUSERGETVALUE
----------enqueue 0x0x82f5c810------------------------
lock version : 1207
Owner node : 0
grant_level : KJUSERNL
req_level : KJUSEREX
bast_level : KJUSERNL
notify_func : (nil)
resp : 0x7f32f540
procp : 0x82d53660
pid : 16602
proc version : 6
oprocp : (nil)
opid : 0
group lock owner : 0x83b4fe90
possible pid : 16602
xid : 12000-0001-00000035
dd_time : 74.0 secs
dd_count : 1
timeout : 0.0 secs
On_timer_q? : N
On_dd_q? : Y
lock_state : OPENING CONVERTING
Open Options : KJUSERDEADLOCK
Convert options : KJUSERGETVALUE
History : 0x1495149a
Msg_Seq : 0x0
res_seq : 3
valblk : 0x00000000000000000000000000000000 .
小结下：
这部分可以理解为引发死锁的会话及进程的相关信息，重点关注如下信息
Global blockers dump start:---------------------------------
DUMP LOCAL BLOCKER/HOLDER: block level 5 res [0x110022][0x11],[TX]
对应即res [0x110022][0x11],[TX]对应持锁会话,具体为res [0x110022]对应xidusn + xidslot,而[0x11]对应xidsqn
也就是说先在TRAC FILE查找Global blockers dump start,然后对应的res相关的信息
然后根据res的信息如下，把16进制转换为10进制，然后到GV$TRANSACTION去匹配对应的会话，最终由GV$SESSION找到持锁会话的信息，进行针对性处理即可
关于第四部分，定位到引发死锁的持锁会话的相关信息脚本如下：
SQL> select inst_id,saddr,sid from gv$session where sid in (133,147);
INST_ID SADDR SID
---------- ---------------- ----------
1 0000000083B4FE90 133
1 0000000083B619A0 147
2 0000000083B619A0 147
SQL> select inst_id,xidusn,xidslot,xidsqn,ubafil,ubablk,ubasqn,ubarec from gv$transaction where ses_addr in ('0000000083B4FE90','0000000083B619A0');
INST_ID XIDUSN XIDSLOT XIDSQN UBAFIL UBABLK UBASQN UBAREC
---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
1 10 6 124 2 890 33 56 --等待者
2 17 34 17 4 434 7 20 --持锁者
SQL> select to_char(17,'xxxxxxxx') xidusn,to_char(34,'xxxxxxxx') xidslot,to_char(17,'xxxxxxxx') xidsqn from dual;
XIDUSN XIDSLOT XIDSQN
--------- --------- ---------
11 22 11
SQL> select inst_id,ses_addr from gv$transaction where xidusn=12 and xidslot=16 and xidsqn=21;
INST_ID SES_ADDR
---------- ----------------
2 0000000083B619A0
SQL> select inst_id,sql_id from gv$session where saddr='0000000083B619A0' and inst_id=2;
INST_ID SQL_ID
---------- -------------
2 b3tmrc0st5pyb
可见最终定位到了引发死锁的所属实例及会话信息，即执行的SQL
SQL> select sql_text from gv$sql where sql_id='b3tmrc0st5pyb' and inst_id=2;
SQL_TEXT
--------------------------------------------------
update t_lock set a=11 where a=1
4，检测基于RAC的死锁是由lmd进程负责
[oracle@jingfa1 bdump]$ ps -ef|grep lmd
oracle 4001 1 0 Nov09 ? 00:00:09 ora_lmd0_jingfa1
oracle 29844 29669 0 00:59 pts/3 00:00:00 grep lmd
5,基于RAC的死锁检测在前端报错和单实例一样，但是在告警日志报销有所区别，内容如下：
Tue Nov 10 00:16:26 2015
Global Enqueue Services Deadlock detected. More info in file
/u01/app/oracle/admin/jingfa/bdump/jingfa1_lmd0_4001.trc.
引申：也就是说GES是负责RAC的全局锁相关的内容
6，如果要深入理解基于RAC的死锁，一定要研究进程LMD及GES的含义
7，感觉在RAC环境死锁检测时间将近30多秒，比较久，我分析有个参数控制，将于下文继续研究
测试
--oracle version
SQL> select * from v$version where rownum=1;
BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bi
SQL> conn tbs_zxy/system
Connected.
---node 1 session 133
SQL> select sid from v$mystat where rownum=1;
SID
----------
133
SQL> select pid,spid,addr from v$process where addr=(select paddr from v$session where sid=(select sid from v$mystat where rownum=1));
PID SPID ADDR
---------- ------------ ----------------
18 16602 0000000083A5E4A8
SQL> create table t_lock(a int,b int);
Table created.
SQL> insert into t_lock select level,level from dual connect by level<=2;
2 rows created.
SQL> commit;
Commit complete.
SQL> select * from t_lock;
A B
---------- ----------
1 1
2 2
---node 2 session 147
SQL> select sid from v$mystat where rownum=1;
SID
----------
147
SQL> select pid,spid,addr from v$process where addr=(select paddr from v$session where sid=(select sid from v$mystat where rownum=1));
PID SPID ADDR
---------- ------------ ----------------
22 16151 0000000083A60448
SQL> update t_lock set a=22 where a=2;
1 row updated.
模拟死锁的代码略
在告警日志出生死锁信息
Mon Nov 9 22:04:01 2015
Global Enqueue Services Deadlock detected. More info in file
/u01/app/oracle/admin/jingfa/bdump/jingfa1_lmd0_1450.trc.
分析死锁信息,分析及注释直接标记在TRACE FILE上面
第一部分：
这部分的内容好像和DRM有关，具体相关原理将于另一文章进行分析及测试
*** SERVICE NAME:() 2015-11-09 21:45:05.422
*** SESSION ID:(166.1) 2015-11-09 21:45:05.422
syncr inc 4 lvl 1 from 1 rcvd (my inc,lvl: 0, 0) (4/5.0.0)
s2m domain info received from 1 (4.6)
* kjxbdomps2m: domain 0 not valid according to instance 1
first hvmap received from 1 for domain 0 (4.7)
* kjxhvmaph: domain 0 valid = 0 according to instance 1
last hvmap received from 1, (4.7)
first hvmap received from 1 for enqueue mappings (4.7)
last hvmap received from 1, (4.7)
ftd received from node 1 (4/7.0.0)
all ftds received
syncr inc 4 lvl 2 from 1 rcvd (my inc,lvl: 4, 1) (4/7.0.0)
中间略
syncr inc 4 lvl 11 from 1 rcvd (my inc,lvl: 4, 10) (4/0.36.0)
ftd received from node 1 (4/0.38.0)
第二部分
这是我们基于RAC死锁最为关心的信息
*** 2015-11-09 22:04:01.002
user session for deadlock lock 0x82f5c810
pid=18 serial=57 audsid=26 user: 27/TBS_ZXY --发生死锁会话对应的进程号以及用户
O/S info: user: oracle, term: pts/2, ospid: 6050, machine: jingfa1 --源于V$SESSION
program: sqlplus@jingfa1 (TNS V1-V3)
application name: SQL*Plus, hash value=3669949024
Current SQL Statement:
update t_lock set a=22 where a=2 --发生死锁会话正在执行的SQL
SQL> select inst_id,saddr,sid from gv$session where sid in (133,147);
INST_ID SADDR SID
---------- ---------------- ----------
1 0000000083B4FE90 133
1 0000000083B619A0 147
2 0000000083B619A0 147
SQL> select inst_id,xidusn,xidslot,xidsqn,ubafil,ubablk,ubasqn,ubarec from gv$transaction where ses_addr in ('0000000083B4FE90','0000000083B619A0');
INST_ID XIDUSN XIDSLOT XIDSQN UBAFIL UBABLK UBASQN UBAREC
---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
1 10 6 124 2 890 33 56
2 17 34 17 4 434 7 20
SQL> select to_char(10,'xxxxxxxx') xidusn,to_char(6,'xxxxxxxx') xidslot,to_char(124,'xxxxxxxx') xidsqn from dual;
XIDUSN XIDSLOT XIDSQN
--------- --------- ---------
a 6 7c
----可见如下on [0xa0006][0x7c],[TX] 含义为[0xa0006]对应xidusn+xidslot，而0x7c]对应xidsqn
ENQUEUE DUMP REQUEST: from 1.22803 on [0xa0006][0x7c],[TX] for reason 3 mtype 0
DUMP LOCAL BLOCKER/HOLDER: block level 5 res [0xa0006][0x7c],[TX] --同上
----------resource 0x0x7e51ac20----------------------
resname : [0xa0006][0x7c],[TX]
Local node : 0
dir_node : 0
master_node : 0
hv idx : 86
hv last r.inc : 4
current inc : 4
hv status : 0
hv master : 0
open options : dd
grant_bits : KJUSERNL KJUSEREX
grant mode : KJUSERNL KJUSERCR KJUSERCW KJUSERPR KJUSERPW KJUSEREX
count : 1 0 0 0 0 1
val_state : KJUSERVS_NOVALUE
valblk : 0x00000000000000000000000000000000 .
access_node : 0
vbreq_state : 0
state : x0
resp : 0x7e51ac20
On Scan_q? : N
Total accesses: 67
Imm. accesses: 58
Granted_locks : 1
Cvting_locks : 1
value_block: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
GRANTED_Q :
lp 0x7e5237d0 gl KJUSEREX rp 0x7e51ac20 [0xa0006][0x7c],[TX] ---同上
master 0 gl owner 0x83b4fe90 possible pid 16602 xid 12000-0001-00000035 bast 0 rseq 8 mseq 0 history 0x4977d495 --pid为发生死锁会话的操作系统进程号
open opt KJUSERDEADLOCK
CONVERT_Q:
lp 0x7e524118 gl KJUSERNL rl KJUSEREX rp 0x7e51ac20 [0xa0006][0x7c],[TX]
master 0 owner 1 bast 1 rseq 12 mseq 0x1 history 0x97d497ad
convert opt KJUSERGETVALUE
----------enqueue 0x0x7e5237d0------------------------
lock version : 11
Owner node : 0
grant_level : KJUSEREX
req_level : KJUSEREX
bast_level : KJUSERNL
notify_func : (nil)
resp : 0x7e51ac20
procp : 0x82d4b4c8
pid : 1450
proc version : 0
oprocp : (nil)
opid : 0
group lock owner : 0x83b4fe90
possible pid : 16602
xid : 12000-0001-00000035
dd_time : 0.0 secs
dd_count : 0
timeout : 0.0 secs
On_timer_q? : N
On_dd_q? : N
lock_state : GRANTED
Open Options : KJUSERDEADLOCK
Convert options : KJUSERNOQUEUE
History : 0x4977d495
Msg_Seq : 0x0
res_seq : 8
valblk : 0x00000000000000000000000000000000 .
DUMP LOCAL BLOCKER: initiate state dump for
possible owner[18.16602]
Submitting asynchronized dump request [28]
----------enqueue 0x0x7e524118------------------------
lock version : 9
Owner node : 1
grant_level : KJUSERNL
req_level : KJUSEREX
bast_level : KJUSERNL
notify_func : 0x165d838
resp : 0x7e51ac20
procp : 0x82d4ea68
pid : 0
proc version : 0
oprocp : (nil)
opid : 0
group lock owner : (nil)
xid : 0000-0000-00000000
dd_time : 0.0 secs
dd_count : 0
timeout : 0.0 secs
On_timer_q? : N
On_dd_q? : N
lock_state : GRANTED
Open Options : KJUSERNO_XID
Convert options : KJUSERGETVALUE
History : 0x97d497ad
Msg_Seq : 0x1
res_seq : 12
valblk : 0x00000000000000000000000000000000 .
SQL> select inst_id,saddr,sid from gv$session where sid in (133,147);
INST_ID SADDR SID
---------- ---------------- ----------
1 0000000083B4FE90 133
1 0000000083B619A0 147
2 0000000083B619A0 147
SQL> select inst_id,xidusn,xidslot,xidsqn,ubafil,ubablk,ubasqn,ubarec from gv$transaction where ses_addr in ('0000000083B4FE90','0000000083B619A0');
INST_ID XIDUSN XIDSLOT XIDSQN UBAFIL UBABLK UBASQN UBAREC
---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
1 10 6 124 2 890 33 56 --等待者
2 17 34 17 4 434 7 20 --持锁者
SQL> select to_char(17,'xxxxxxxx') xidusn,to_char(34,'xxxxxxxx') xidslot,to_char(17,'xxxxxxxx') xidsqn from dual;
XIDUSN XIDSLOT XIDSQN
--------- --------- ---------
11 22 11
SQL> select inst_id,ses_addr from gv$transaction where xidusn=12 and xidslot=16 and xidsqn=21;
INST_ID SES_ADDR
---------- ----------------
2 0000000083B619A0
SQL> select inst_id,sql_id from gv$session where saddr='0000000083B619A0' and inst_id=2;
INST_ID SQL_ID
---------- -------------
2 b3tmrc0st5pyb
可见最终定位到了引发死锁的所属实例及会话信息，即执行的SQL
SQL> select sql_text from gv$sql where sql_id='b3tmrc0st5pyb' and inst_id=2;
SQL_TEXT
--------------------------------------------------
update t_lock set a=11 where a=1
可见global blockers dump start即死锁的持锁者事务的信息为如下信息：
即res [0x110022][0x11],[TX]对应持锁会话,具体为res [0x110022]对应xidusn + xidslot,而[0x11]对应xidsqn
也就是可以根据如下，把16进制转换为10进制，然后到GV$TRANSACTION去匹配对应的会话，最终由GV$SESSION找到持锁会话的信息，进行针对性处理即可
Global blockers dump start:---------------------------------
DUMP LOCAL BLOCKER/HOLDER: block level 5 res [0x110022][0x11],[TX]
----------resource 0x0x7f32f540----------------------
resname : [0x110022][0x11],[TX]
Local node : 0
dir_node : 1
master_node : 1
hv idx : 7
hv last r.inc : 2
current inc : 4
hv status : 0
hv master : 1
open options : dd
Held mode : KJUSERNL
Cvt mode : KJUSEREX
Next Cvt mode : KJUSERNL
msg_seq : 0x1
res_seq : 3
grant_bits : KJUSERNL
grant mode : KJUSERNL KJUSERCR KJUSERCW KJUSERPR KJUSERPW KJUSEREX
count : 1 0 0 0 0 0
val_state : KJUSERVS_NOVALUE
valblk : 0x00000000000000000000000000000000 .
access_node : 1
vbreq_state : 0
state : x8
resp : 0x7f32f540
On Scan_q? : N
Total accesses: 29
Imm. accesses: 22
Granted_locks : 0
Cvting_locks : 1
value_block: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
GRANTED_Q :
CONVERT_Q:
lp 0x82f5c810 gl KJUSERNL rl KJUSEREX rp 0x7f32f540 [0x110022][0x11],[TX]
master 1 gl owner 0x83b4fe90 possible pid 16602 xid 12000-0001-00000035 bast 0 rseq 3 mseq 0 history 0x1495149a
convert opt KJUSERGETVALUE
----------enqueue 0x0x82f5c810------------------------
lock version : 1207
Owner node : 0
grant_level : KJUSERNL
req_level : KJUSEREX
bast_level : KJUSERNL
notify_func : (nil)
resp : 0x7f32f540
procp : 0x82d53660
pid : 16602
proc version : 6
oprocp : (nil)
opid : 0
group lock owner : 0x83b4fe90
possible pid : 16602
xid : 12000-0001-00000035
dd_time : 74.0 secs
dd_count : 1
timeout : 0.0 secs
On_timer_q? : N
On_dd_q? : Y
lock_state : OPENING CONVERTING
Open Options : KJUSERDEADLOCK
Convert options : KJUSERGETVALUE
History : 0x1495149a
Msg_Seq : 0x0
res_seq : 3
valblk : 0x00000000000000000000000000000000 .
DUMP LOCAL BLOCKER: initiate state dump for
possible owner[18.16602]
Submitting asynchronized dump request [28]
DUMP LOCAL BLOCKER/HOLDER: block level 5 res [0xa0006][0x7c],[TX]
----------resource 0x0x7e51ac20----------------------
resname : [0xa0006][0x7c],[TX]
Local node : 0
dir_node : 0
master_node : 0
hv idx : 86
hv last r.inc : 4
current inc : 4
hv status : 0
hv master : 0
open options : dd
grant_bits : KJUSERNL KJUSEREX
grant mode : KJUSERNL KJUSERCR KJUSERCW KJUSERPR KJUSERPW KJUSEREX
count : 1 0 0 0 0 1
val_state : KJUSERVS_NOVALUE
valblk : 0x00000000000000000000000000000000 .
access_node : 0
vbreq_state : 0
state : x0
resp : 0x7e51ac20
On Scan_q? : N
Total accesses: 69
Imm. accesses: 59
Granted_locks : 1
Cvting_locks : 1
value_block: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
GRANTED_Q :
lp 0x7e5237d0 gl KJUSEREX rp 0x7e51ac20 [0xa0006][0x7c],[TX]
master 0 gl owner 0x83b4fe90 possible pid 16602 xid 12000-0001-00000035 bast 0 rseq 8 mseq 0 history 0x4977d495
open opt KJUSERDEADLOCK
CONVERT_Q:
lp 0x7e524118 gl KJUSERNL rl KJUSEREX rp 0x7e51ac20 [0xa0006][0x7c],[TX]
master 0 owner 1 bast 1 rseq 12 mseq 0x1 history 0x97d497ad
convert opt KJUSERGETVALUE
----------enqueue 0x0x7e5237d0------------------------
lock version : 11
Owner node : 0
grant_level : KJUSEREX
req_level : KJUSEREX
bast_level : KJUSERNL
notify_func : (nil)
resp : 0x7e51ac20
procp : 0x82d4b4c8
pid : 1450
proc version : 0
oprocp : (nil)
opid : 0
group lock owner : 0x83b4fe90
possible pid : 16602
xid : 12000-0001-00000035
dd_time : 0.0 secs
dd_count : 0
timeout : 0.0 secs
On_timer_q? : N
On_dd_q? : N
lock_state : GRANTED
Open Options : KJUSERDEADLOCK
Convert options : KJUSERNOQUEUE
History : 0x4977d495
Msg_Seq : 0x0
res_seq : 8
valblk : 0x00000000000000000000000000000000 .
DUMP LOCAL BLOCKER: initiate state dump for
possible owner[18.16602]
Submitting asynchronized dump request [28]
----------enqueue 0x0x7e524118------------------------
lock version : 9
Owner node : 1
grant_level : KJUSERNL
req_level : KJUSEREX
bast_level : KJUSERNL
notify_func : 0x165d838
resp : 0x7e51ac20
procp : 0x82d4ea68
pid : 0
proc version : 0
oprocp : (nil)
opid : 0
group lock owner : (nil)
xid : 0000-0000-00000000
dd_time : 0.0 secs
dd_count : 0
timeout : 0.0 secs
On_timer_q? : N
On_dd_q? : N
lock_state : GRANTED
Open Options : KJUSERNO_XID
Convert options : KJUSERGETVALUE
History : 0x97d497ad
Msg_Seq : 0x1
res_seq : 12
valblk : 0x00000000000000000000000000000000 .
Global blockers dump end:-----------------------------------
基于RAC的全局死锁的图表链（此块内容仍没有完全理解）
Global Wait-For-Graph(WFG) at ddTS[0.1] :
BLOCKED 0x82f5c810 5 [0x110022][0x11],[TX] [12000-0001-00000035] 0 --blocked表示等待者，其中[0x110022][0x11]对应持锁会话的[0x110022]对应xidusn + xidslot,而[0x11]对应xidsqn
BLOCKER 0x82eb5db0 5 [0x110022][0x11],[TX] [16000-0002-0000000A] 1 --blocker表示持锁者
BLOCKED 0x82f59468 5 [0xa0006][0x7c],[TX] [16000-0002-0000000A] 1 ---表示等待者
BLOCKER 0x7e5237d0 5 [0xa0006][0x7c],[TX] [12000-0001-00000035] 0 ---表示等待者
正文到此结束
热门推荐
相关文章
Loading...
![[HBLOG]公众号](http://www.liuhaihua.cn/img/qrcode_gzh.jpg)
