Deep Learning 在中文分词和词性标注中的应用_20131030.docx

《Deep Learning 在中文分词和词性标注中的应用_20131030.docx》由会员分享，可在线阅读，更多相关《Deep Learning 在中文分词和词性标注中的应用_20131030.docx（41页珍藏版）》请在得力文库 - 分享文档赚钱的网站上搜索。

1、1 VC_L8= Xj 4 1 2 mh $+)./(p%eghoty& q*!0/#,- O K32.1 Mapping Characters into Feature Vectors . . . . . . . . . . . . . . . . . . . . 5 2.2 Tag Scoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.3 Tag Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2、 . . . . . . . 7 2.4 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.4.1 Sentence-Level Log-Likelihood . . . . . . . . . . . . . . . . . . . . . . 8 2.5 A New Training Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3 12 3.1 Tagging Scheme

3、. 13 3.2 The Choice of Hyper-parameters . 13 3.3 Closed Test on the SIGHAN Bakeoff . 15 3.4 Combined Approach . 16 4 18 s 2 , (DL, Deep Learning) (CWS, Chinese word segm entat ioVn) ;tHj S!Z(POO-S tagging) LdGH. $ l -j S bj5 Z 2v-Il WL C_. l xmyQ. 1 3 QX=, Aj & | Ws& Ws S M K! 3. 1 NLP , l;X b P8LX

4、n i;L W KI? L2M !. : (pipelined system) , (joint solutions) LjRPOVS D + q lr;5.2 ()8 , +d6 xx SLZ. hb , 8 Lx , heO2IC;L : ?bH /t# 1. , 2. ,E KCQ K I L6zEa=Q p YjJ ; oZ8 #! (overfit); 4. , . 1.1 4 , , . PULk I 9EsLyL o DLlC;w e= !K qL ) 6r G HL Q!6L2!p LO- Kj +jL V 3 o$ D j&TT ?lEL% d2mjL r lowe= pCi

5、pel in edWsy ste m!5jnoinlt Lso lut ion s , J Ws mJ Ws Xm x q & yz Ws . . / LX Ab, xb LN: C l0P (state-of-the-art) . ., Xb i0V? ud S CEkrBKEIC Lr S+J(lginjgu5istHic kn6ow leEdge) . Fhbe, mh?e t b b Rjoint , POS joint . joint . , . 5 Od9yQ 1 5CZ2 : (1) perceptron-style , , 5X QTT L ze 4 VH L6h/ , ; D

6、eep Learning NLP . DC j !pYJ%VNM . Q&J(K rSL 2 l; ;H !L blEWD% LM9PG !. 8 !heLH 2 P , , (CRFs) (feature templates). Ybm VF k?w(labo r- intensive), (human ingenuity) (linguistic intu- ition). , , , H , : ?b 2003 , Bengio (2), 2011 5 Collobert N?- N?Cvw rS EX 5(3).X TSLTT Q6LY m j . 1 . L KI3b?Q1KIL L

7、D iNIX d r 2.1 Mapping Characters into Feature Vectors , . XVCd L LhSD , vL?; _ i X XM PR #.d ,+ d+ ;e ( ), |D| . 2.1 , . jLVk%qc1 :n, n ci, 1 i n. + ci L ; X M LQki , pOJd _ ZD (ci) = Meki R , (2.1) eki R|D| R|D| ki ( ki 1 0). ZD +gXP & (lookup tLabQle layerE), !W;df 6XQELlh& (p+ro jecltion layer).

8、 L ( M) , BP , . ;Z83gXj , $ i%0T|+.)4dEX m j L a l &,WuQD, PG+ KZq 6( name entity recognition) , : . YEL l; E . XoBd 9 L X , $U LW + x 6XgLbPo , + , ; I e; , , WddEX -o 6L uD.boundary entropy, accessor variety NW+joj Lw L: fV(:Yfm?oqc1 +a6 XL Ic=n).4Posci,VKXILl;Ed% _ f 1 ZD ( ci w ) . (ci) = ZD (c

9、i) . ZD (ci+ w ) . (2.5) , start stop. wrM2./3 :fgLcDi iCN,/,|n| $k:L6q Llp ,(g,Z w, b 1. (2.8) , $ LZT sigmoid d2 p HardTanh 6 . p 2V XRR . XLlti +pf (ti|Vi)X. 2.1 T = S, B, I, E, , (2.9), c1:5 = , , , , 2.5 3 n 2.3 Tag Inference NLP , tag LYmoH.lG;y M;|H !L tUagLlT LQli s X Qj% X|, $ LdE6 g=Ll p2

10、(transition score) Aij , , A0i tag T i $ , m? L taPg p%athdKEI/s ,_b+i QtXagpathLKI . LQ XL lp$ LTT lEX%c1:n, DKIX*lX f (c1:n). 7 f (t|i) _l i ci tag T t . tag path t1:n, ELK%l d Ls(c1: n, t1:n , L) Q= n X(Ati +(62.g9) 1 i=1 (2.9), . I zP p,L $ |J2.4.17 zN$M . zj/ (2.11) , , QpY L|+.CN:dXtUj (c. gLw

11、 Wgub , /XJDNM$ LQze . 7 Z8L6za=VK tj/xz i i i i 2.5 A New Training Method X BqL+CQuhhe.e (maximum-likelihood met hod) , $ jI 4 , L.d 5/ (c, t), f (c), , VViteXrbi zQedU I%$ c LLXTTK l +-xoLG0ta!g Lp , lD5+X7 Xt . t PXt,: ti 6= t , . 1. i F L (t, t|c) + +; L (t, t|c) . (2.15) f (ti|i) f (t |i) 2. ti

12、1 6= t1 F ti 6= t , , L ( t, t|c) + +; L (t, t|c) . (2.16) Ati1 ti At 1 ti +, ( C ). L (t, t|c) br$L 7V lm Q _U& 9(c , t&) #+.C L rpS, Ld&9f& ?co#rr.ecXt pXath LLKlqin- correct path ( G0!p/KILZm+xKlL path) 1 , 1:5 , 2 6= t3, x . LKl =L)a12.8 A0t ? (2.16), : t1 6= t rX, -D /oxz bD 7pL Joxz Q gu /6F G

13、 L (t, t|c) A + +; L (t, t|c) A . (2.17) 0t1 0t 2.2 , I !/SJb =L, !Lp ) d-UovG 0. !pLl 5 LKl 06hb F , . 1. R: N : 3. E: tagging 11 Step 1 ). X VL SZ 2. Z : NQ L!p1 2.1 ( ) 61.1 4_Q ez e= (M, W 2, b2, W 3, b3, A) (small random values). 12 (2) f (c) A, Viterbi , , _(pf N , ) e, , g 7d; , , 2.1, . 1 ,

14、4 Lhlozpp 4 BP$z e Lmhz5K oKJIo. QzeLhzHp F d K 14 NLP 2. PSA: jZlT%pgY| jdT!p L lpY ? |.3: 3 (ze Yk 1. SLL: 2.4.1 Sentence-Level Log-Likelihood . 5NLMPLer-ceoptron-Style Qze. LQze 1T. ZT 1. CWS ( SEG) 2. CWS +ZPTO S y(Z T 7y 7 JWP) Z QhX * (hyper-parameters) . training set, +pL 168 %. dezveelopHm e

15、nLt shet8. 15 . ELKZmd H 2. 3. Chinese Treebank 4 (CTB-4) . 1529 %Z. ls, ZTpYl& treebank L Sections 1-43, 144-169, 900-931, + 78023 , , 45135 . ; 1697 . % Bbq ! (double-annotaed) LQ b*Z ZT TpLY #jCBhinese Treebank (CTB) data sets from Bakeoff-3. + j om tj ;l4 ( 2.1 ). Sina news, 325MB . 18 LpY %g$U

16、l Dze LE|.4 19 ./& xG$U zeD% g:LjrJavla &Z2LSLL PSA |ze. : Intel Core i3, 2.13GHz, 2GB RAM; Linux + Java Develop- ment Kit 1.6. H *% #: #L F-score (3SZ ( LT ) ). 20 , 3.1 Tagging Scheme , 1. B: o(B eglinnlin;g) ;$L Q X% .L X !D/t tag : X2. I: (Inside) . ;L; L=+ X. 3. E: (Endding) 4. S: (Single) E; (

17、 ). 3lE /L,; 5 : ! +o x ; H l!;D +;. uHD ,!$L jD$/ t jtagl o XL W!;hr3(vekrbphrLase ): 1. B VP: 2. I VP: . WW;*LL Q=X. 3. E VP: 4. S VP: . W ; ;X*L+L WX;*. 21 : 5 , F-score . , (word X !p& /Lh X G = : LSFp , ZJr$ jLb IOBES (inside, other, beginning, end, and single) !z , O . 3.2 bT he C ho ic e oof

18、HyLpelr-;pa ra;mHet er!s (window size) w H. jLpYJ JRJ(N|MLddeTve!loppmen t Wse:t. flCoz !p jLb |e.E4sp, b 3 , (drop smmothly). Foov , w 5 , G sX H L)! pP/(wF CX!p) H QopYJGL:P;X(ovHerfitt/edF). 6g 2 5 (noise) . L:f LW / | ) |$f!rpL%Lg K9b5X L +lE G 3 5 2 F-sco)re F-score H|$:fC w .L 95, H ) CX, H Lh

19、8 m4 ( 3 ). 3F-score ) H |$e1E4 sp L 95 $L# r eD/ K rIL&ZDLT+ Jina$n ewNs Mrt jlC+ : Zjm !nptYrKeI r D;+LLh;e(character em- 3.4 Combined Approach S text)$P G *j . rS EAtj PSA lQ. 3, $ + bedding), , (the acceptability of a piece of KVQXXrXS mEax 0, 1 f9 (cL|hd) +Ifj (Hc|h), (3.18) hH c D + H: md L te

20、xt window (39). D: . c|h:Sc CL:f h. |h: h c . c f (c|hD):fc|h L lCEKI Ll5 .KILB:f f (c|h): |h . $ jLS ll&EQKIrL l, 53 Bakeoff-3 pY1, v 8000 . % . LV R2 Fb Lj%am GqZh K6P XL R VL:0 / l; ; 3. LrRS cascading, voting VL ; . ensemble; 1 Xiaoqing Zheng, Hanyang Chen, Tianyu Xu. Deep Learning for Chinese W

21、ord Segmenta- tion and POS tagging. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 647-657. 19 2 Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic langugage model. Journal of Machine Learning Research, 3:

22、 1137 1155. 3 Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12: 2493-2537. 4 Michael Collins. 2002. Discriminative training methods for hidden Markov model

23、s: Theory and experiments with perceptron algorithms. In Proceedings of the International Conference on Empirical Methods in Natural Language Processing (EMNLP 02). 5 Hwee Tou Ng, Jin Kiat Low. Chinese Part-of-Speech Tagging: One-at-a-Time or All-at-Once? Word-Based or Character-Based? In Proceedings of the International Conference on Empirical Methods in Natural Language Processing (EMNLP 04).