Traditional Chinese Parsing Task

1.Introduction

Chinese parsing has been a highly active research area in recent years, and there is a pressing need for a common evaluation platform where different approaches can be compared and progress can be gauged. The purpose of the CIPS-ParsEval campaign is to provide such a platform. The first CIPS-ParsEval was successfully held in Beijing in 2009. The second CIPS-ParsEval, jointly sponsored by CIPS and SIGHAN, was also held in Beijing in 2010. Built on these successes, Chinese parsing evaluation will be organized again and held in in conjunction with the second Chinese Language Processing Conference (CLP 2012) as bakeoff task. The hope is that through such evaluation campaigns, more advanced Chinese syntactic parsing techniques will emerge, more effective Chinese language processing resources will be built, and the state of the art will be advanced as a result.

 

This evaluation includes two sub-tasks: sentence parsing and semantic role labeling task. For each sub-task, there are two tracks. 1) In the closed track, participants can only use training data provided by the organizers. 2) In the open track the participants can use any data source in addition to the training data provided by the organizers. Entries in the two tracks will be evaluated separately. In addition, single systems and combined systems will be evaluated separately in the closed track. 1) single system: parsers that use a single parsing model to accomplish the parsing task. 2) system combination: participants are allowed to combine multiple models to improve performance. Collaborative decoding methods will be regarded as a combination method.

2.Tasks

Sub-Task 1: Sentence Parsing

Goal: Evaluate the ability of automatic parsers on complete sentences in real texts.

Task description:

Input: Complete Chinese sentences with gold standard word segmentation. The word count of each sentence should be greater than 7.

Example:

他 刊登 一則 廣告 在 報紙 上

Output: The system should assign a POS tag to each word and recognize the syntactic structure in a given sentence.

Example:

S(agent:NP(Head:Nhaa:) | Head:VC33:刊登 | theme: NP (quantifier: DM:一則 | Head: Nac: 廣告) | location: PP (Head:P21: | DUMMY: GP(DUMMY:NP(Head:Nab:報紙) | Head:Ng:)))

The criterion for judging correctness: the boundary and phrase label of a syntactic constituent should be completely identical with the gold standard. The semantic roles (agent, Head, etc.) can be ignored in the output format. The complete set of phrase labels (S, VP, NP, GP, PP, XP, and DM) is defined in the User Manual of Sinica Treebank v3.0 (http://turing.iis.sinica.edu.tw/treesearch, page 6). The complete set of POS tags is defined in http://ckipsvr.iis.sinica.edu.tw/cat.htm.

Performance metrics: Precision, Recall, F1 measure

P = # of correctly recognized constituents / # of all constituents in the automatic parse

R = # of correctly recognized constituents / # of all constituents in the gold standard parse

F1 = 2*P*R / (P + R)

 

Sub-Task 2: Semantic Role Labeling

Goal: Evaluate the ability of automatic parsers on semantic role labeling.

Task description:

Input: Complete Chinese sentences with gold standard word segmentation. The word count of each sentence should be greater than 7.

Example:

他 刊登 一則 廣告 在 報紙 上

Output: The system should assign a semantic role for each constituent in a given sentence, as shown in the following example. Since there are more than 50 semantic roles defined in the User Manual of Sinica Treebank v3.0 (http://turing.iis.sinica.edu.tw/treesearch, page 6), this task only focuses on the identification of the agent-Head-theme relation in a given sentence, where Head represents the head of the sentence. For example, the agent-Head-theme relation in the given sample sentence is “他刊登一則廣告” (agent:NP(Head:Nhaa:他) | Head:VC33:刊登 | theme: NP (quantifier: DM:一則 | Head: Nac: 廣告)).

Example:

S(agent:NP(Head:Nhaa:) | Head:VC33:刊登 | theme: NP (quantifier: DM:一則 | Head: Nac: 廣告) | location: PP (Head:P21: | DUMMY: GP(DUMMY:NP(Head:Nab: 報紙) | Head:Ng:)))

 

The criterion for judging correctness: the boundary and semantic role of a syntactic constituent should be completely identical with the gold standard. The ground truth of this sample sentence is agent (他), Head(刊登), theme(一則廣告), location(在報紙上). The semantic roles (agent, Head, etc) are defined in the User Manual of Sinica Treebank v3.0 (http://turing.iis.sinica.edu.tw/treesearch, page 6).

 

Performance metrics: Precision, Recall, F1 measure

P = # of correctly recognized roles (agent, Head, theme) / # of all roles in the recognized data

R = # of correctly recognized roles (agent, Head, theme) / # of all roles (agent, Head, theme) in the gold standard data

F1 = 2*P*R / (P + R)

3.Data Sets

Training data: the training data will be selected from Sinica Treebank according to sentence lengths and complexities.

Test data: 1000 newly developed sentences selected for both sub-tasks to cover different sentence lengths and complexities.

 

Note: Please fill in the copyright license file, send scanned version and mail paper version to corresponding organizer.

License Download

Mailing Address:

Lung-Hao Lee

NLP Lab (R301), CSIE Building, National Taiwan University

No. 1, Sec. 4, Roosevelt Rd., Taipei 10617, Taiwan

4.Important Dates

· Registration for Bakeoffs open: May 15, 2012

· Training data released: July 1, 2012

· Dry run (Submission format validation): August 1, 2012

· Test data released: September 27, 2012

· Test results submission deadline: September 30, 2012

· Test result evaluation released: October 20, 2012

· Draft evaluation report submission deadline: November 10, 2012

· Draft evaluation report reviews returned: November 20, 2012

· Final evaluation report submission deadline: December 1, 2012

· Main Conference: December 20-21, 2012

5.References

[1] Chinese Knowledge Information Processing Group (1993). Categorical Analysis of Chinese. ACLCLP Technical Report # 93-05, Academia Sinica.

[2] Chu-Ren Huang, Keh-Jiann Chen, Feng-Yi Chen, Keh-Jiann Chen, Zhao-Ming Gao and Kuang-Yu Chen (2000). Sinica Treebank: Design Criteria, Annotation Guidelines, and On-line Interface. In Proceedings of 2nd Chinese Language Processing Workshop (Held in conjunction with ACL-2000). 29-37.

[3] Keh-Jiann Chen, Chu-Ren Huang, Feng-Yi Chen, Chi-Ching Luo, Ming-Chung Chang, Chao-Jan Chen, and Zhao-Ming Gao (2003). Sinica Treebank: Design Criteria, Representational Issues and Implementation. In Anne Abeille (Ed.) Treebanks Building and Using Parsed Corpora. Language and Speech series. Dordrecht:Kluwer, 231-248.

6. Acknowledgement

Research fellow Keh-Jiann Chen, the leader of Chinese Knowledge Information Processing Group (CKIP) in IIS, Academia Sinica, is appreciated for supporting Sinica Treebank in this traditional Chinese parsing task.

 

Organizers:

Yuen-Hsien Tseng, National Taiwan Normal University

Liang-Chih Yu, Yuan Ze University

Lung-Hao Lee, National Taiwan University