例句與造句

1. for a given language, a word breaker tokenizes text based on the lexical rules of the language
   對(duì)于某種給定的語言，斷字符可基于該語言的詞匯規(guī)則對(duì)文本進(jìn)行標(biāo)記化。
2. in full-text search, for a given language, a word breaker tokenizes text based on the lexical rules of the language
   外部聯(lián)接的一種，其中包含join子句中最右側(cè)表的所有行。
3. It's difficult to find tokenizes in a sentence. 用tokenizes造句挺難的