{"title":"pandey.github.io","link":[{"@attributes":{"href":"https:\/\/pandey.github.io\/","rel":"alternate"}},{"@attributes":{"href":"https:\/\/pandey.github.io\/feeds\/all.atom.xml","rel":"self"}}],"id":"https:\/\/pandey.github.io\/","updated":"2017-02-02T00:00:00-06:00","entry":[{"title":"How old is a Unicode\u00a0character?","link":{"@attributes":{"href":"https:\/\/pandey.github.io\/posts\/unicode-codepoint-age.html","rel":"alternate"}},"published":"2017-02-02T00:00:00-06:00","updated":"2017-02-02T00:00:00-06:00","author":{"name":"Anshuman Pandey"},"id":"tag:pandey.github.io,2017-02-02:\/posts\/unicode-codepoint-age.html","summary":"

I often need to find out when a particular character was encoded in The \nUnicode Standard. The Unicode Character Database (UCD<\/span>) has \na file called DerivedAge.txt<\/code> that contains such information. This UCD<\/span> \nfile provides the Age<\/code> property for characters, which can be queried \nto obtain the version of Unicode \u2026<\/p>","content":"

I often need to find out when a particular character was encoded in The \nUnicode Standard. The Unicode Character Database (UCD<\/span>) has \na file called DerivedAge.txt<\/code> that contains such information. This UCD<\/span> \nfile provides the Age<\/code> property for characters, which can be queried \nto obtain the version of Unicode in which a codepoint was assigned. \nHowever, deriving the age of a character from DerivedAge.txt<\/code> is not \nso straightforward on account of the file’s format. Here, I will \nparse that file and define a new function in order to obtain the \nnecessary data with ease.<\/p>\n

Understand the data<\/h2>\n

I used DerivedAge.txt<\/code> in a previous post to explore the \ngrowth of Unicode<\/a> \nin terms of the number of characters added in each version. \nRecall that the DerivedAge.txt<\/code> file has the following presentation:<\/p>\n

<\/span>    # Age=V1_1\n\n    # Assigned as of Unicode 1.1.0 (June, 1993)\n    # [excluding removed Hangul Syllables]\n\n    0000..001F    ; 1.1 #  [32] <control-0000>..<control-001F>\n    0020..007E    ; 1.1 #  [95] SPACE..TILDE\n    007F..009F    ; 1.1 #  [33] <control-007F>..<control-009F>\n    00A0..00AC    ; 1.1 #  [13] NO-BREAK SPACE..NOT SIGN\n    00AD          ; 1.1 #       SOFT HYPHEN\n<\/code><\/pre><\/div>\n\n
In that previous post, I parsed this data and stored it into the \nderivedage<\/code> list of lists, which has the following structure:<\/p>\n
<\/span>derivedage<\/span>[<\/span>0<\/span>:<\/span>5<\/span>]<\/span>\n<\/code><\/pre><\/div>\n\n
<\/span>[['1.1'<\/span>, <\/span>'0000..001F'<\/span>, <\/span>'32'<\/span>, <\/span>'<control-0000>..<control-001F>'<\/span>],\n <\/span>['1.1'<\/span>, <\/span>'0020..007E'<\/span>, <\/span>'95'<\/span>, <\/span>'SPACE..TILDE'<\/span>],\n <\/span>['1.1'<\/span>, <\/span>'007F..009F'<\/span>, <\/span>'33'<\/span>, <\/span>'<control-007F>..<control-009F>'<\/span>],\n <\/span>['1.1'<\/span>, <\/span>'00A0..00AC'<\/span>, <\/span>'13'<\/span>, <\/span>'NO-BREAK SPACE..NOT SIGN'<\/span>],\n <\/span>['1.1'<\/span>, <\/span>'00AD'<\/span>, <\/span>'1'<\/span>, <\/span>'SOFT HYPHEN'<\/span>]]\n<\/code><\/pre><\/div>\n\n
Notice that the second element contains the codepoint. However, \nthere are actual two types of data masquerading as one. In some lists, \nthe second element is a single codepoint, eg. 00AD<\/span>, while in others \nit stands for a range, eg. 00A0..00AC<\/span>.<\/p>\n
Approach the problem<\/h2>\n
The issue becomes clear: searching for a particular codepoint will have \none of two results. If the query matches the exact value of the second \nelement, then the Age<\/code> property is readily available. However, if there \nis no exact match, then additional processing is necessary.<\/p>\n
There are two ways to approach the task:<\/p>\n
    \n
  1. Expand all code ranges to produce entries for each literal codepoint<\/li>\n
  2. Infer codepoint’s membership within a range of codepoints<\/li>\n<\/ol>\n
    The first approach would require generating new entries for each codepoint in a given range:<\/p>\n
    <\/span>coderange<\/span> =<\/span> [<\/span>'00A0'<\/span>,<\/span> '00AC'<\/span>]<\/span>\nhexlower<\/span> =<\/span> int<\/span>(<\/span>"0x"<\/span> +<\/span> coderange<\/span>[<\/span>0<\/span>],<\/span> 16<\/span>)<\/span>\nhexupper<\/span> =<\/span> int<\/span>(<\/span>"0x"<\/span> +<\/span> coderange<\/span>[<\/span>1<\/span>],<\/span> 16<\/span>)<\/span>\n\nfor<\/span> i<\/span> in<\/span> (<\/span>range<\/span>(<\/span>hexlower<\/span>,<\/span> hexupper<\/span> +<\/span> 1<\/span>)):<\/span>\n    print<\/span> (<\/span>hex<\/span>(<\/span>i<\/span>))<\/span>\n<\/code><\/pre><\/div>\n\n
    <\/span>0<\/span>xa0<\/span>\n0<\/span>xa1<\/span>\n0<\/span>xa2<\/span>\n0<\/span>xa3<\/span>\n0<\/span>xa4<\/span>\n0<\/span>xa5<\/span>\n0<\/span>xa6<\/span>\n0<\/span>xa7<\/span>\n0<\/span>xa8<\/span>\n0<\/span>xa9<\/span>\n0<\/span>xaa<\/span>\n0<\/span>xab<\/span>\n0<\/span>xac<\/span>\n<\/code><\/pre><\/div>\n\n
    One benefit to this approach is the ability to produce a dictionary that contains every codepoint as a key and it’s age as the value:<\/p>\n
    <\/span>derivedage = {\n    '00A0' : '1.1',\n    '00A1' : '1.1',\n    ...\n    }\n<\/code><\/pre><\/div>\n\n
    The downside is that it will significantly expand the dataset as there are more than 160k codepoints as of Unicode version 9.0.<\/p>\n
    The other approach is to search within a range at runtime and to return the associated age:<\/p>\n
    <\/span>coderange<\/span> =<\/span> [<\/span>'00A0'<\/span>,<\/span> '00AC'<\/span>]<\/span>\nhexlower<\/span> =<\/span> int<\/span>(<\/span>"0x"<\/span> +<\/span> coderange<\/span>[<\/span>0<\/span>],<\/span> 16<\/span>)<\/span>\nhexupper<\/span> =<\/span> int<\/span>(<\/span>"0x"<\/span> +<\/span> coderange<\/span>[<\/span>1<\/span>],<\/span> 16<\/span>)<\/span>\n\nchar<\/span> =<\/span> '00A3'<\/span>\n\nchar<\/span> =<\/span> int<\/span>(<\/span>"0x"<\/span> +<\/span> char<\/span>,<\/span> 16<\/span>)<\/span>\nif<\/span> hexlower<\/span> <=<\/span> char<\/span> <=<\/span> hexupper<\/span>:<\/span>\n    print<\/span> (<\/span>'found'<\/span>,<\/span> hex<\/span>(<\/span>char<\/span>),<\/span> 'within'<\/span>,<\/span> hex<\/span>(<\/span>hexlower<\/span>),<\/span> 'and'<\/span>,<\/span> hex<\/span>(<\/span>hexupper<\/span>))<\/span>\nelse<\/span>:<\/span>\n    print<\/span> (<\/span>'not found'<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
    <\/span>found 0xa3 within 0xa0 and 0xac\n<\/code><\/pre><\/div>\n\n
    Define the function<\/h2>\n
    The function relies upon the derivedage<\/code> container, which is a list of \nlists with the following structure:<\/p>\n
    <\/span>['1.1', '0020..007E', '95', 'SPACE..TILDE']\n<\/code><\/pre><\/div>\n\n
    For our present purposes, we need to perform an operation on element #2, which is \neither a single codepoint value or a range of codepoints. If it is a range, then \nwe need to split into two to define the lower and upper bounds:<\/p>\n
    <\/span>['0020', '007E']\n<\/code><\/pre><\/div>\n\n
    The function will take a list of 1 or more codepoints. The codepoint is expressed \nas a string, eg. ‘0065’. For each element in the list, the function will determine \nif the codepoint scan derivedage<\/code>:<\/p>\n
    <\/span>def<\/span> <\/span>getDerivedAge<\/span>(<\/span>inputlist_<\/span>):<\/span>\n\n    resultslist<\/span> =<\/span> {}<\/span>\n\n    for<\/span> codepoint<\/span> in<\/span> inputlist_<\/span>:<\/span>\n\n        scriptinfo<\/span> =<\/span> None<\/span>\n        char<\/span> =<\/span> int<\/span>(<\/span>"0x"<\/span> +<\/span> codepoint<\/span>,<\/span> 16<\/span>)<\/span>\n\n        for<\/span> script<\/span> in<\/span> derivedage<\/span>:<\/span>\n\n            version<\/span> =<\/span> script<\/span>[<\/span>0<\/span>]<\/span>\n            coderange<\/span> =<\/span> script<\/span>[<\/span>1<\/span>]<\/span>.<\/span>split<\/span>(<\/span>'..'<\/span>)<\/span>\n\n            if<\/span> len<\/span>(<\/span>coderange<\/span>)<\/span> ==<\/span> 2<\/span>:<\/span>\n\n                hexlower<\/span> =<\/span> int<\/span>(<\/span>"0x"<\/span> +<\/span> coderange<\/span>[<\/span>0<\/span>],<\/span> 16<\/span>)<\/span>\n                hexupper<\/span> =<\/span> int<\/span>(<\/span>"0x"<\/span> +<\/span> coderange<\/span>[<\/span>1<\/span>],<\/span> 16<\/span>)<\/span>\n\n                if<\/span> hexlower<\/span> <=<\/span> char<\/span> <=<\/span> hexupper<\/span>:<\/span>\n                    scriptinfo<\/span> =<\/span> version<\/span>\n\n            elif<\/span> len<\/span>(<\/span>coderange<\/span>)<\/span> ==<\/span> 1<\/span>:<\/span>\n\n                if<\/span> codepoint<\/span> ==<\/span> coderange<\/span>[<\/span>0<\/span>]:<\/span>\n                    scriptinfo<\/span> =<\/span> version<\/span>\n\n            else<\/span>:<\/span>\n\n                scriptinfo<\/span> =<\/span> 'Error'<\/span>\n\n        resultslist<\/span>.<\/span>update<\/span>({<\/span>codepoint<\/span> :<\/span> scriptinfo<\/span>})<\/span>\n\n    return<\/span> resultslist<\/span>\n<\/code><\/pre><\/div>\n\n
    Test the function<\/h2>\n
    <\/span>codepoints<\/span> =<\/span> [<\/span>'0035'<\/span>,<\/span> '0913'<\/span>,<\/span> 'A8F0'<\/span>,<\/span> 'BBBB'<\/span>,<\/span> '11080'<\/span>,<\/span> '14000'<\/span>]<\/span>\n<\/code><\/pre><\/div>\n\n
    <\/span>getDerivedAge<\/span>(<\/span>codepoints<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
    <\/span>{'0035': '1.1',\n '0913': '1.1',\n '11080': '5.2',\n '14000': None,\n 'A8F0': '5.2',\n 'BBBB': '2.0'}\n<\/code><\/pre><\/div>\n\n
    The function returns the versions in which each of our specified codepoints \nwas encoded. But, we got None<\/code> for 14000… is there an error somewhere? \nPerforming a search through DerivedAge.txt<\/code> for a single codepoint or \nwithinn a range yields nothing. A quick glance at the \nUnicode Roadmap shows that 14000 is an unencoded codepoint in a block \nthat is tentatively allocated for Egyptian Hieroglyphs. So, there is no \ncharacter in Unicode associated with this codepoint. The value None<\/code> \nis accurate.<\/p>\n
    Assessment<\/h1>\n
    The DerivedAge.txt<\/code> file presents the ‘age’ of a codepoint, or \nthe version of Unicode in which it was assigned in the stanard.. \nWhen this version information is aggregated with other data in the UCD<\/span>, \nit is possible to generate holistic metadata for codepoints in Unicode. \nIn a future post I will explore the Blocks.txt<\/code> and Scripts.txt<\/code> file \nin order to illustrate how codepoints are grouped within Unicode paradigms \nof ‘blocks’ and ‘scripts’.<\/p>","category":{"@attributes":{"term":"articles"}}},{"title":"Visualizing the growth of Unicode using\u00a0matplotlib","link":{"@attributes":{"href":"https:\/\/pandey.github.io\/posts\/unicode-growth-matplotlib.html","rel":"alternate"}},"published":"2017-01-30T00:00:00-06:00","updated":"2017-01-30T00:00:00-06:00","author":{"name":"Anshuman Pandey"},"id":"tag:pandey.github.io,2017-01-30:\/posts\/unicode-growth-matplotlib.html","summary":"
    In a previous post<\/a>, I presented \ninformation about the growth of Unicode in terms of the number of \ncodepoints assigned in each version. The data was displayed as text tables \nusing the PrettyTable<\/code> package. As we are visual beings, I think it would be useful \nto also present that data \u2026<\/p>","content":"
    In a previous post<\/a>, I presented \ninformation about the growth of Unicode in terms of the number of \ncodepoints assigned in each version. The data was displayed as text tables \nusing the PrettyTable<\/code> package. As we are visual beings, I think it would be useful \nto also present that data using charts. Here I use matplotlib<\/code> and the same \ndata sets to produce such visualizations.<\/p>\n
    Prerequisites<\/h2>\n
    Recall the lists of lists we produced: DA_codepoint_totals<\/code>, DA_char_totals<\/code>, and DA_other_totals<\/code>:<\/p>\n
    ** Growth of codepoints by version: **<\/p>\n
    <\/span>DA_codepoint_totals<\/span>\n<\/code><\/pre><\/div>\n\n
    <\/span>[['1.1', 33979, 33979],\n ['2.0', 144521, 178500],\n ['2.1', 2, 178502],\n ['3.0', 10307, 188809],\n ['3.1', 44978, 233787],\n ['3.2', 1016, 234803],\n ['4.0', 1226, 236029],\n ['4.1', 1273, 237302],\n ['5.0', 1369, 238671],\n ['5.1', 1624, 240295],\n ['5.2', 6648, 246943],\n ['6.0', 2088, 249031],\n ['6.1', 732, 249763],\n ['6.2', 1, 249764],\n ['6.3', 5, 249769],\n ['7.0', 2834, 252603],\n ['8.0', 7716, 260319],\n ['9.0', 7500, 267819]]\n<\/code><\/pre><\/div>\n\n
    ** Growth of characters per version: **<\/p>\n
    <\/span>DA_char_totals<\/span>\n<\/code><\/pre><\/div>\n\n
    <\/span>[['1.1', 27512, 27512],\n ['2.0', 11373, 38885],\n ['2.1', 2, 38887],\n ['3.0', 10307, 49194],\n ['3.1', 44946, 94140],\n ['3.2', 1016, 95156],\n ['4.0', 1226, 96382],\n ['4.1', 1273, 97655],\n ['5.0', 1369, 99024],\n ['5.1', 1624, 100648],\n ['5.2', 6648, 107296],\n ['6.0', 2088, 109384],\n ['6.1', 732, 110116],\n ['6.2', 1, 110117],\n ['6.3', 5, 110122],\n ['7.0', 2834, 112956],\n ['8.0', 7716, 120672],\n ['9.0', 7500, 128172]]\n<\/code><\/pre><\/div>\n\n
    ** Growth of noncharacters per version: **<\/p>\n
    <\/span>DA_other_totals<\/span>\n<\/code><\/pre><\/div>\n\n
    <\/span>[['1.1', 6467, 6467],\n ['2.0', 133148, 139615],\n ['2.1', 0, 139615],\n ['3.0', 0, 139615],\n ['3.1', 32, 139647],\n ['3.2', 0, 139647],\n ['4.0', 0, 139647],\n ['4.1', 0, 139647],\n ['5.0', 0, 139647],\n ['5.1', 0, 139647],\n ['5.2', 0, 139647],\n ['6.0', 0, 139647],\n ['6.1', 0, 139647],\n ['6.2', 0, 139647],\n ['6.3', 0, 139647],\n ['7.0', 0, 139647],\n ['8.0', 0, 139647],\n ['9.0', 0, 139647]]\n<\/code><\/pre><\/div>\n\n
    Approach<\/h2>\n
    Import the Python modules needed for this task:<\/p>\n
    <\/span>import<\/span> <\/span>numpy<\/span> <\/span>as<\/span> <\/span>np<\/span>\n<\/code><\/pre><\/div>\n\n
    <\/span>import<\/span> <\/span>matplotlib.pyplot<\/span> <\/span>as<\/span> <\/span>plt<\/span>\nimport<\/span> <\/span>matplotlib.ticker<\/span> <\/span>as<\/span> <\/span>ticker<\/span>\n<\/code><\/pre><\/div>\n\n
    <\/span>import<\/span> <\/span>seaborn<\/span>\n<\/code><\/pre><\/div>\n\n
    Use Jupyter magic to show plots within the notebook:<\/p>\n
    <\/span>%<\/span>matplotlib<\/span> inline<\/span>\n<\/code><\/pre><\/div>\n\n
    Plot line graphs<\/h3>\n
    The x<\/em> axis is the arange<\/code> of the number of versions, not the version number itself. The y<\/em> axis is the number of characters get values for x<\/em> and y<\/em> axes. Convert list of lists to numpy<\/code> array, then read columms from array into lists: x<\/em> values from column 0, y<\/em> values from column 2. Use numpy<\/code> slice [:,0]<\/code> to read the matrix by column. Convert y<\/em> values to type int<\/code> when slicing.<\/p>\n
    Line graph: number of characters per version<\/h4>\n
    <\/span>char_array<\/span> =<\/span> np<\/span>.<\/span>array<\/span>(<\/span>DA_char_totals<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
    <\/span>fig<\/span>,<\/span> ax<\/span> =<\/span> plt<\/span>.<\/span>subplots<\/span>()<\/span>\n\nplt<\/span>.<\/span>title<\/span>(<\/span>'Number of characters encoded in Unicode'<\/span>)<\/span>\nplt<\/span>.<\/span>ylabel<\/span>(<\/span>'Characters'<\/span>)<\/span>\nplt<\/span>.<\/span>xlabel<\/span>(<\/span>'Versions'<\/span>)<\/span>\n\nx_values<\/span> =<\/span> char_array<\/span>[:,<\/span>0<\/span>]<\/span>\nx_range<\/span> =<\/span> np<\/span>.<\/span>arange<\/span>(<\/span>len<\/span>(<\/span>x_values<\/span>))<\/span>\ny_values<\/span> =<\/span> char_array<\/span>[:,<\/span>2<\/span>]<\/span>.<\/span>astype<\/span>(<\/span>int<\/span>)<\/span>\n\nax<\/span>.<\/span>set_xticks<\/span>(<\/span>x_range<\/span>)<\/span>\nax<\/span>.<\/span>set_xticklabels<\/span>(<\/span>x_values<\/span>,<\/span> rotation<\/span>=<\/span>45<\/span>)<\/span>\n\nax<\/span>.<\/span>plot<\/span> (<\/span>x_range<\/span>,<\/span> y_values<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
    <\/span>[<matplotlib.lines.Line2D at 0x973c610>]\n<\/code><\/pre><\/div>\n\n
    \"image\"<\/p>\n
    As we will be plotting two more graphs of the same type and using the same type of container, it is practical to create a function for the above plot type:<\/p>\n
    <\/span>def<\/span> <\/span>plot_line_graph<\/span>(<\/span>list_<\/span>,<\/span> labels_<\/span>):<\/span>\n\n    # process parameters<\/span>\n\n    char_array<\/span> =<\/span> np<\/span>.<\/span>array<\/span>(<\/span>list_<\/span>)<\/span>\n\n    plt_title<\/span>,<\/span> plt_ylabel<\/span>,<\/span> plt_xlabel<\/span> =<\/span> labels_<\/span>\n\n    # set up plot<\/span>\n\n    fig<\/span>,<\/span> ax<\/span> =<\/span> plt<\/span>.<\/span>subplots<\/span>()<\/span>\n\n    plt<\/span>.<\/span>title<\/span>(<\/span>plt_title<\/span>)<\/span>\n    plt<\/span>.<\/span>ylabel<\/span>(<\/span>plt_ylabel<\/span>)<\/span>\n    plt<\/span>.<\/span>xlabel<\/span>(<\/span>plt_xlabel<\/span>)<\/span>\n\n    x_values<\/span> =<\/span> char_array<\/span>[:,<\/span>0<\/span>]<\/span>\n    x_range<\/span> =<\/span> np<\/span>.<\/span>arange<\/span>(<\/span>len<\/span>(<\/span>x_values<\/span>))<\/span>\n    y_values<\/span> =<\/span> char_array<\/span>[:,<\/span>2<\/span>]<\/span>.<\/span>astype<\/span>(<\/span>int<\/span>)<\/span>\n\n    ax<\/span>.<\/span>set_xticks<\/span>(<\/span>x_range<\/span>)<\/span>\n    ax<\/span>.<\/span>set_xticklabels<\/span>(<\/span>x_values<\/span>,<\/span> rotation<\/span>=<\/span>45<\/span>)<\/span>\n\n    ax<\/span>.<\/span>plot<\/span> (<\/span>x_range<\/span>,<\/span> y_values<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
    Now we can pass a container and a list of labels to the function to generate additional plots:<\/p>\n
    <\/span>labels<\/span> =<\/span> [<\/span>'Number of characters encoded in Unicode'<\/span>,<\/span> 'Characters'<\/span>,<\/span> 'Versions'<\/span>]<\/span>\nplot_line_graph<\/span>(<\/span>DA_char_totals<\/span>,<\/span> labels<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
    \"image\"<\/p>\n
    Line graph: number of other entities per version<\/h4>\n
    <\/span>labels<\/span> =<\/span> [<\/span>'Number of other entities encoded in Unicode'<\/span>,<\/span> 'Entities'<\/span>,<\/span> 'Versions'<\/span>]<\/span>\nplot_line_graph<\/span>(<\/span>DA_other_totals<\/span>,<\/span> labels<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
    \"image\"<\/p>\n
    Line graph: number of codepoints assigned per version<\/h4>\n
    <\/span>labels<\/span> =<\/span> [<\/span>'Number of codepoints encoded in Unicode'<\/span>,<\/span> 'Noncharacters'<\/span>,<\/span> 'Versions'<\/span>]<\/span>\nplot_line_graph<\/span>(<\/span>DA_codepoint_totals<\/span>,<\/span> labels<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
    \"image\"<\/p>\n
    Plot stacked bar graphs<\/h3>\n
    We can also plot the growth of number of characters using a stacked bar graph. Each bar is segmented into smaller categories in order to convey the smaller units that constitute the whole value. <\/p>\n
    The x<\/em> axis is the arange<\/code> of the number of versions, not the version number itself. We will display the actual version numbers by modifying the labels of the plot. Our stacked bar requires two y<\/em> values: the first (bottom) is the number of new characters per version, the second (top) is the value from the previous version, or, the difference between the total characters for a given version minus the new characters added.<\/p>\n
    We’ll convert a list of lists to a numpy<\/code> array, then read columms from the array into lists using numpy<\/code> slice, eg. [:,0]<\/code>.<\/p>\n
    Bar graph: growth of number of characters per version<\/h4>\n
    <\/span>char_array<\/span> =<\/span> np<\/span>.<\/span>array<\/span>(<\/span>DA_char_totals<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
    Get the difference between the total characters and the new characters for each version. This is expressed as follows:<\/p>\n
    <\/span>diff_array<\/span> =<\/span> char_array<\/span>[:,<\/span>2<\/span>]<\/span>.<\/span>astype<\/span>(<\/span>int<\/span>)<\/span> -<\/span> char_array<\/span>[:,<\/span>1<\/span>]<\/span>.<\/span>astype<\/span>(<\/span>int<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
    Set up the plot:<\/p>\n
    <\/span>fig<\/span>,<\/span> ax<\/span> =<\/span> plt<\/span>.<\/span>subplots<\/span>()<\/span>\nax<\/span> =<\/span> plt<\/span>.<\/span>axes<\/span>()<\/span>\n\nax<\/span>.<\/span>set_title<\/span>(<\/span>'Characters per version'<\/span>)<\/span>\nax<\/span>.<\/span>set_ylabel<\/span>(<\/span>'Number of characters'<\/span>)<\/span>\nax<\/span>.<\/span>set_xlabel<\/span>(<\/span>'Versions'<\/span>)<\/span>\n\nx_values<\/span> =<\/span> char_array<\/span>[:,<\/span>0<\/span>]<\/span>\nx_range<\/span> =<\/span> np<\/span>.<\/span>arange<\/span>(<\/span>len<\/span>(<\/span>x_values<\/span>))<\/span>\n\ny1_values<\/span> =<\/span> char_array<\/span>[:,<\/span>1<\/span>]<\/span>.<\/span>astype<\/span>(<\/span>int<\/span>)<\/span>\ny2_values<\/span> =<\/span> diff_array<\/span>\n\ny_total<\/span> =<\/span> char_array<\/span>[:,<\/span>2<\/span>]<\/span>.<\/span>astype<\/span>(<\/span>int<\/span>)<\/span>\n\nax<\/span>.<\/span>xaxis<\/span>.<\/span>set_major_locator<\/span>(<\/span>ticker<\/span>.<\/span>FixedLocator<\/span>(<\/span>x_range<\/span>))<\/span>\nax<\/span>.<\/span>xaxis<\/span>.<\/span>set_major_formatter<\/span>(<\/span>ticker<\/span>.<\/span>FixedFormatter<\/span>(<\/span>x_values<\/span>))<\/span>\nax<\/span>.<\/span>set_xticklabels<\/span>(<\/span>x_values<\/span>,<\/span> rotation<\/span>=<\/span>45<\/span>)<\/span>\n\nax<\/span>.<\/span>bar<\/span>(<\/span>x_range<\/span>,<\/span> y1_values<\/span>,<\/span> width<\/span>=<\/span>1<\/span>,<\/span> label<\/span> =<\/span> 'New chars'<\/span>)<\/span>\nax<\/span>.<\/span>bar<\/span>(<\/span>x_range<\/span>,<\/span> y2_values<\/span>,<\/span> width<\/span>=<\/span>1<\/span>,<\/span> bottom<\/span>=<\/span>y1_values<\/span>,<\/span> label<\/span> =<\/span> 'Total chars'<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
    \"image\"<\/p>\n
    Commentary<\/h2>\n
    This exercise helped us to convert our data from lists into graphs using matplotlib<\/code>. The text tables I produced in the previous \nexercise, but the visualizations give us another perspective \ninto the growth of Unicode.<\/p>","category":{"@attributes":{"term":"articles"}}},{"title":"Exploring the growth of Unicode using the UCD and\u00a0Python","link":{"@attributes":{"href":"https:\/\/pandey.github.io\/posts\/unicode-growth-UCD-python.html","rel":"alternate"}},"published":"2017-01-28T00:00:00-06:00","updated":"2017-01-28T00:00:00-06:00","author":{"name":"Anshuman Pandey"},"id":"tag:pandey.github.io,2017-01-28:\/posts\/unicode-growth-UCD-python.html","summary":"
    How many characters are in Unicode? How many new characters have been added for each version of Unicode? From a programmatic standpoint, figuring out the number of characters published in a given version of Unicode is not as straightforward as one might imagine. While it is true that Unicode contains \u2026<\/p>","content":"
    How many characters are in Unicode? How many new characters have been added for each version of Unicode? From a programmatic standpoint, figuring out the number of characters published in a given version of Unicode is not as straightforward as one might imagine. While it is true that Unicode contains ‘characters’ from writing systems, there are also entities that are not truly ‘characters’ in the conventional sense. There are entities such as control characters, private-use characters, surrogates, and even noncharacters. Therefore, to get a sense of the number of entities in Unicode, it is practical to think instead in terms of ‘code points’.<\/p>\n
    In this exercise, I process data from the Unicode Character Database (UCD<\/span>) in order to produce an overview of the longitudinal growth of Unicode. There are certainly other ways to solve the challenge, my goal is to do so in a programmatic fashion using available data. Specifically, I will explore a UCD<\/span> file called DerivedAge.txt<\/code>, which provides all codepoints assignments for each version, from 1.1 to 9.0.<\/p>\n
    I will process this file to extract the number of code points in each version of Unicode, and also to distinguish at a high level between characters and other entities.<\/p>\n
    Approach<\/h2>\n
    Import the Python modules needed for this task:<\/p>\n
    <\/span>import<\/span> <\/span>re<\/span>\n<\/code><\/pre><\/div>\n\n
    <\/span>import<\/span> <\/span>requests<\/span>\nfrom<\/span> <\/span>prettytable<\/span> <\/span>import<\/span> PrettyTable<\/span>\n<\/code><\/pre><\/div>\n\n
    Get the data<\/h3>\n
    Fetch the plain text file DerivedAge.txt<\/code> from the Unicode Character Database using requests<\/code>, and get the text of the requests<\/code> object using the .text<\/code> method:<\/p>\n
    <\/span>url<\/span> =<\/span> "http:\/\/www.unicode.org\/Public\/UCD\/latest\/ucd\/DerivedAge.txt"<\/span>\nr<\/span> =<\/span> requests<\/span>.<\/span>get<\/span>(<\/span>url<\/span>)<\/span>.<\/span>text<\/span>\n<\/code><\/pre><\/div>\n\n
    Inspect the data<\/h3>\n
    <\/span>len<\/span>(<\/span>r<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
    <\/span>106583<\/span>\n<\/code><\/pre><\/div>\n\n
    Split the requests<\/code> text by newline into a list named filestream<\/code>, then take a slice of the list to get a sense of the data:<\/p>\n
    <\/span>filestream<\/span> =<\/span> r<\/span>.<\/span>split<\/span>(<\/span>'<\/span>\\n<\/span>'<\/span>)<\/span>\n\nfor<\/span> line<\/span> in<\/span> filestream<\/span>[<\/span>51<\/span>:<\/span>70<\/span>]:<\/span>\n    print<\/span> (<\/span>line<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
    <\/span># Age=V1_1<\/span>\n\n# Assigned as of Unicode 1.1.0 (June, 1993)<\/span>\n# [excluding removed Hangul Syllables]<\/span>\n\n0000..001<\/span>F<\/span>    <\/span>;<\/span> <\/span>1.1<\/span> <\/span>#  [32] <control-0000>..<control-001F><\/span>\n0020..007<\/span>E<\/span>    <\/span>;<\/span> <\/span>1.1<\/span> <\/span>#  [95] SPACE..TILDE<\/span>\n007F<\/span>.<\/span>.009<\/span>F<\/span>    <\/span>;<\/span> <\/span>1.1<\/span> <\/span>#  [33] <control-007F>..<control-009F><\/span>\n00A0<\/span>.<\/span>.00<\/span>AC<\/span>    <\/span>;<\/span> <\/span>1.1<\/span> <\/span>#  [13] NO-BREAK SPACE..NOT SIGN<\/span>\n00AD<\/span>          <\/span>;<\/span> <\/span>1.1<\/span> <\/span>#       SOFT HYPHEN<\/span>\n00AE<\/span>.<\/span>.01<\/span>F5<\/span>    <\/span>;<\/span> <\/span>1.1<\/span> <\/span># [328] REGISTERED SIGN..LATIN SMALL LETTER G WITH ACUTE<\/span>\n01FA<\/span>.<\/span>.0217<\/span>    <\/span>;<\/span> <\/span>1.1<\/span> <\/span>#  [30] LATIN CAPITAL LETTER A WITH RING ABOVE AND ACUTE..LATIN SMALL LETTER U WITH INVERTED BREVE<\/span>\n0250..02<\/span>A8<\/span>    <\/span>;<\/span> <\/span>1.1<\/span> <\/span>#  [89] LATIN SMALL LETTER TURNED A..LATIN SMALL LETTER TC DIGRAPH WITH CURL<\/span>\n02B0<\/span>.<\/span>.02<\/span>DE<\/span>    <\/span>;<\/span> <\/span>1.1<\/span> <\/span>#  [47] MODIFIER LETTER SMALL H..MODIFIER LETTER RHOTIC HOOK<\/span>\n02E0<\/span>.<\/span>.02E9<\/span>    <\/span>;<\/span> <\/span>1.1<\/span> <\/span>#  [10] MODIFIER LETTER SMALL GAMMA..MODIFIER LETTER EXTRA-LOW TONE BAR<\/span>\n0300..0345<\/span>    <\/span>;<\/span> <\/span>1.1<\/span> <\/span>#  [70] COMBINING GRAVE ACCENT..COMBINING GREEK YPOGEGRAMMENI<\/span>\n0360..0361<\/span>    <\/span>;<\/span> <\/span>1.1<\/span> <\/span>#   [2] COMBINING DOUBLE TILDE..COMBINING DOUBLE INVERTED BREVE<\/span>\n0374..0375<\/span>    <\/span>;<\/span> <\/span>1.1<\/span> <\/span>#   [2] GREEK NUMERAL SIGN..GREEK LOWER NUMERAL SIGN<\/span>\n037A<\/span>          <\/span>;<\/span> <\/span>1.1<\/span> <\/span>#       GREEK YPOGEGRAMMENI<\/span>\n<\/code><\/pre><\/div>\n\n
    Observations<\/h3>\n
    The file lists the codepoints encoded in each version. It specifies the property Age<\/code>, which is indicated both as a comment before each group of codepoints<\/p>\n
    <\/span>#<\/span> Age=V1_1\n<\/code><\/pre><\/div>\n\n
    and in the data for every code point or range of codepoints:<\/p>\n
    <\/span>037<\/span>A<\/span>          <\/span>;<\/span> <\/span>1.1<\/span> <\/span>#<\/span>       <\/span>GREEK<\/span> <\/span>YPOGEGRAMMENI<\/span>\n<\/code><\/pre><\/div>\n\n
    The above record tells us that the character U+037A GREEK<\/span> YPOGRAMMENI<\/span>, was encoded in Unicode version 1.1.<\/p>\n
    The total number of code points is given as a comment after the list for each version:<\/p>\n
    <\/span># Total code points: 33979\n<\/code><\/pre><\/div>\n\n
    However, the number of codepoints<\/em> does not tell us about the number of characters<\/em>, or other entities. Such details are available in DerivedAge.txt<\/code>, but we will need to obtain them programmatically from the data. First, let’s take a look at the format and contents of the file.<\/p>\n
    Description of the data format<\/h3>\n
    A sample of data in DerivedAge.txt<\/code>:<\/p>\n
    <\/span>00<\/span>A0<\/span>..00<\/span>AC<\/span>    <\/span>;<\/span> <\/span>1.1<\/span> <\/span>#<\/span>  <\/span>[<\/span>13<\/span>]<\/span> <\/span>NO<\/span>-<\/span>BREAK<\/span> <\/span>SPACE<\/span>..<\/span>NOT<\/span> <\/span>SIGN<\/span>\n00<\/span>AD<\/span>          <\/span>;<\/span> <\/span>1.1<\/span> <\/span>#<\/span>       <\/span>SOFT<\/span> <\/span>HYPHEN<\/span>\n4<\/span>E00<\/span>..9<\/span>FA5<\/span>    <\/span>;<\/span> <\/span>1.1<\/span> <\/span>#<\/span> <\/span>[<\/span>20902<\/span>]<\/span> <\/span>CJK<\/span> <\/span>UNIFIED<\/span> <\/span>IDEOGRAPH<\/span>-<\/span>4<\/span>E00<\/span>..<\/span>CJK<\/span> <\/span>UNIFIED<\/span> <\/span>IDEOGRAPH<\/span>-<\/span>9<\/span>FA5<\/span>\nE000<\/span>..<\/span>F8FF<\/span>    <\/span>;<\/span> <\/span>1.1<\/span> <\/span>#<\/span> <\/span>[<\/span>6400<\/span>]<\/span> <\/span><<\/span>private<\/span>-<\/span>use<\/span>-<\/span>E000<\/span>><\/span>..<\/span><<\/span>private<\/span>-<\/span>use<\/span>-<\/span>F8FF<\/span>><\/span>\n<\/code><\/pre><\/div>\n\n
    As shown DerivedAge.txt<\/code> is a CSV<\/span>-like flat file. There are two primary fields, separated by a ;<\/code> semi-colon. The information after #<\/code> may be considered comments, but may be separated into two fields:<\/p>\n
      \n
    • field 0: codepoint or codepoint range for a character or group of contiguous characters<\/li>\n
    • field 1: version of Unicode in which the character(s) was\/were encoded<\/li>\n
    • field 2: number of codepoints in a range, in square braces, empty if a single codepoint<\/li>\n
    • field 3: name or names of first and last item in a contiguous group; characters are UPPERCASE<\/span>, while noncharacters are lowercase within <<\/code> … ><\/code> delimiters<\/li>\n<\/ul>\n
      The format is generally uniform, apart from inconsistencies in the spacing of fields after #<\/code>. Field 3 is alloted a length of seven spaces between the # and field 4. In most cases, the [<\/code>…]<\/code> fits within the 7 columns. When the codepoint count is present, there is 1 space between the #<\/code> and [<\/code>…]<\/code>, and 1 space between [<\/code>…]<\/code> and field 4. If the [<\/code>…]<\/code> takes more than four spaces, then the starting column for the character names are shifted right.<\/p>\n
      Define regex for extracting fields<\/h4>\n
      The fields can be extracted using a regex:<\/p>\n
      <\/span>^([\\w\\.]*)\\s*\\; (.*) \\#\\s+(?:\\[(\\d*)\\])?\\s+(.*)\n\n  ^^^^^^^        ^^          ^^^^^^^^^      ^^\n     1            2              3           4\n<\/code><\/pre><\/div>\n\n
      Four capture groups are defined:<\/p>\n
        \n
      • [\\w\\.]*<\/code> for the codepoint range or single code point<\/li>\n
      • (.*)<\/code> for the version number<\/li>\n
      • (?:\\[(\\d*)\\])<\/code> is a lookahead for […] to handle single code point<\/li>\n
      • (.*)<\/code> for the character name(s)<\/li>\n<\/ul>\n
        Wrangle the data<\/h3>\n
        Now wrangle the data from its native format into a more usable container:<\/p>\n
        <\/span>derivedage<\/span> =<\/span> []<\/span>\n<\/code><\/pre><\/div>\n\n
        derivedage<\/code> will be a list of lists with a structure of:<\/p>\n
        <\/span>[\n  ['version number', 'code point range', 'count', 'character names']]\n]\n<\/code><\/pre><\/div>\n\n
        Also, let’s use a dictionary to capture the Age<\/code> property values that have been explicitly specified. These values can be used for validating the data that we will derive from the file.<\/p>\n
        <\/span>derivedage_properties<\/span> =<\/span> {}<\/span>\n<\/code><\/pre><\/div>\n\n
        Now we’ll read the file line by line, and extract information from lines:<\/p>\n
        <\/span>for<\/span> line<\/span> in<\/span> filestream<\/span>:<\/span>\n\n    # use the regex defined above:<\/span>\n\n    m<\/span> =<\/span> re<\/span>.<\/span>search<\/span>(<\/span>'^([\\w\\.]*)\\s*\\; (.*) \\#\\s+(?:\\[(\\d*)\\])?\\s+(.*)'<\/span>,<\/span> line<\/span>)<\/span>\n\n    if<\/span> m<\/span>:<\/span>\n\n        m_coderange<\/span> =<\/span> m<\/span>.<\/span>group<\/span>(<\/span>1<\/span>)<\/span>\n        m_version<\/span> =<\/span> m<\/span>.<\/span>group<\/span>(<\/span>2<\/span>)<\/span>\n\n        # m.group(3) == None if the (?:\\[(\\d*)\\])? segment of the regex is empty<\/span>\n\n        if<\/span> m<\/span>.<\/span>group<\/span>(<\/span>3<\/span>)<\/span> ==<\/span> None<\/span>:<\/span>\n            m_count<\/span> =<\/span> '1'<\/span>\n        else<\/span>:<\/span>\n            m_count<\/span> =<\/span> m<\/span>.<\/span>group<\/span>(<\/span>3<\/span>)<\/span>\n\n        m_charname<\/span> =<\/span> m<\/span>.<\/span>group<\/span>(<\/span>4<\/span>)<\/span>\n\n        derivedage<\/span>.<\/span>append<\/span>([<\/span>m_version<\/span>,<\/span> m_coderange<\/span>,<\/span> m_count<\/span>,<\/span> m_charname<\/span>])<\/span>\n\n\n    # now capture the age property details:<\/span>\n\n    age_property_match<\/span> =<\/span> re<\/span>.<\/span>search<\/span>(<\/span>'\\# Age=V(.*)'<\/span>,<\/span> line<\/span>)<\/span>\n    age_count_match<\/span> =<\/span> re<\/span>.<\/span>search<\/span>(<\/span>'\\# Total code points: (.*)'<\/span>,<\/span> line<\/span>)<\/span>\n\n    if<\/span> age_property_match<\/span>:<\/span>\n        age_property<\/span> =<\/span> age_property_match<\/span>.<\/span>group<\/span>(<\/span>1<\/span>)<\/span>\n        age_property<\/span> =<\/span> re<\/span>.<\/span>sub<\/span>(<\/span>'\\_'<\/span>,<\/span> '.'<\/span>,<\/span> age_property<\/span>)<\/span>\n        derivedage_properties<\/span>.<\/span>update<\/span>({<\/span>age_property<\/span> :<\/span> 0<\/span>})<\/span>\n\n    if<\/span> age_count_match<\/span>:<\/span>\n        age_count<\/span> =<\/span> age_count_match<\/span>.<\/span>group<\/span>(<\/span>1<\/span>)<\/span>\n        derivedage_properties<\/span>[<\/span>age_property<\/span>]<\/span> =<\/span> int<\/span>(<\/span>age_count<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
        Verify derivedage<\/code>:<\/p>\n
        <\/span>derivedage<\/span>[<\/span>0<\/span>:<\/span>10<\/span>]<\/span>\n<\/code><\/pre><\/div>\n\n
        <\/span>[['1.1'<\/span>, <\/span>'0000..001F'<\/span>, <\/span>'32'<\/span>, <\/span>'<control-0000>..<control-001F>'<\/span>],\n <\/span>['1.1'<\/span>, <\/span>'0020..007E'<\/span>, <\/span>'95'<\/span>, <\/span>'SPACE..TILDE'<\/span>],\n <\/span>['1.1'<\/span>, <\/span>'007F..009F'<\/span>, <\/span>'33'<\/span>, <\/span>'<control-007F>..<control-009F>'<\/span>],\n <\/span>['1.1'<\/span>, <\/span>'00A0..00AC'<\/span>, <\/span>'13'<\/span>, <\/span>'NO-BREAK SPACE..NOT SIGN'<\/span>],\n <\/span>['1.1'<\/span>, <\/span>'00AD'<\/span>, <\/span>'1'<\/span>, <\/span>'SOFT HYPHEN'<\/span>],\n <\/span>['1.1'<\/span>,\n  <\/span>'00AE..01F5'<\/span>,\n  <\/span>'328'<\/span>,\n  <\/span>'REGISTERED SIGN..LATIN SMALL LETTER G WITH ACUTE'<\/span>],\n <\/span>['1.1'<\/span>,\n  <\/span>'01FA..0217'<\/span>,\n  <\/span>'30'<\/span>,\n  <\/span>'LATIN CAPITAL LETTER A WITH RING ABOVE AND ACUTE..LATIN SMALL LETTER U WITH INVERTED BREVE'<\/span>],\n <\/span>['1.1'<\/span>,\n  <\/span>'0250..02A8'<\/span>,\n  <\/span>'89'<\/span>,\n  <\/span>'LATIN SMALL LETTER TURNED A..LATIN SMALL LETTER TC DIGRAPH WITH CURL'<\/span>],\n <\/span>['1.1'<\/span>,\n  <\/span>'02B0..02DE'<\/span>,\n  <\/span>'47'<\/span>,\n  <\/span>'MODIFIER LETTER SMALL H..MODIFIER LETTER RHOTIC HOOK'<\/span>],\n <\/span>['1.1'<\/span>,\n  <\/span>'02E0..02E9'<\/span>,\n  <\/span>'10'<\/span>,\n  <\/span>'MODIFIER LETTER SMALL GAMMA..MODIFIER LETTER EXTRA-LOW TONE BAR'<\/span>]]\n<\/code><\/pre><\/div>\n\n
        Verify derivedage_summary<\/code>:<\/p>\n
        <\/span>derivedage_properties<\/span>\n<\/code><\/pre><\/div>\n\n
        <\/span>{'1.1': 33979,\n '2.0': 144521,\n '2.1': 2,\n '3.0': 10307,\n '3.1': 44978,\n '3.2': 1016,\n '4.0': 1226,\n '4.1': 1273,\n '5.0': 1369,\n '5.1': 1624,\n '5.2': 6648,\n '6.0': 2088,\n '6.1': 732,\n '6.2': 1,\n '6.3': 5,\n '7.0': 2834,\n '8.0': 7716,\n '9.0': 7500}\n<\/code><\/pre><\/div>\n\n
        Process the data<\/h2>\n
        Define dictionaries for capturing the data:<\/p>\n
        <\/span>DA_codepoints<\/span> =<\/span> {}<\/span>\nDA_chars<\/span> =<\/span> {}<\/span>\nDA_other<\/span> =<\/span> {}<\/span>\n\nDA_entities<\/span> =<\/span> {}<\/span>\n<\/code><\/pre><\/div>\n\n
        Now iterate over the derivedage<\/code> list of lists:<\/p>\n
        <\/span>for<\/span> entry<\/span> in<\/span> derivedage<\/span>:<\/span>\n\n    version<\/span> =<\/span> entry<\/span>[<\/span>0<\/span>]<\/span>\n    coderange<\/span> =<\/span> entry<\/span>[<\/span>1<\/span>]<\/span>.<\/span>split<\/span>(<\/span>'..'<\/span>)<\/span>\n    count<\/span> =<\/span> entry<\/span>[<\/span>2<\/span>]<\/span>\n    charname<\/span> =<\/span> entry<\/span>[<\/span>3<\/span>]<\/span>\n\n    # 1. get codepoint range. if coderange ==2, get number of codepoints in <\/span>\n    # range by subtracting coderange[1] and coderange[0]. the values in <\/span>\n    # these lists are string represents of hex codes, so convert them <\/span>\n    # to actual hex numbers:<\/span>\n\n    if<\/span> len<\/span>(<\/span>coderange<\/span>)<\/span> ==<\/span> 2<\/span>:<\/span>\n        range_length<\/span> =<\/span> int<\/span>(<\/span>coderange<\/span>[<\/span>1<\/span>],<\/span> 16<\/span>)<\/span> -<\/span> int<\/span>(<\/span>coderange<\/span>[<\/span>0<\/span>],<\/span> 16<\/span>)<\/span> +<\/span> 1<\/span>\n    elif<\/span> len<\/span>(<\/span>coderange<\/span>)<\/span> ==<\/span> 1<\/span>:<\/span>\n        range_length<\/span> =<\/span> 1<\/span>\n\n    # 2. store number of total codepoints per version:<\/span>\n\n    if<\/span> version<\/span> in<\/span> DA_codepoints<\/span>:<\/span>\n        DA_codepoints<\/span>[<\/span>version<\/span>]<\/span> +=<\/span> range_length<\/span>\n    else<\/span>:<\/span>\n        DA_codepoints<\/span>[<\/span>version<\/span>]<\/span> =<\/span> range_length<\/span>\n\n\n    # 3. store number of characters and other codepoints.<\/span>\n    # check for codepoint types. noncharacters are identified in <\/span>\n    # DerivedAge.txt using descriptions in <..> brackets<\/span>\n\n    nonchar_match<\/span> =<\/span> re<\/span>.<\/span>search<\/span>(<\/span>'^\\<'<\/span>,<\/span> charname<\/span>)<\/span>\n\n    # if entry is not a character:<\/span>\n    if<\/span> nonchar_match<\/span>:<\/span>\n\n        # keep a record of the number of noncharacters per version:<\/span>\n\n        if<\/span> version<\/span> in<\/span> DA_other<\/span>:<\/span>\n            DA_other<\/span>[<\/span>version<\/span>]<\/span> +=<\/span> range_length<\/span>\n        else<\/span>:<\/span>\n            DA_other<\/span>[<\/span>version<\/span>]<\/span> =<\/span> range_length<\/span>\n\n    # if entry is a character:<\/span>\n    else<\/span>:<\/span>\n\n        # keep a record of the number of characters per version:<\/span>\n\n        if<\/span> version<\/span> in<\/span> DA_chars<\/span>:<\/span>\n            DA_chars<\/span>[<\/span>version<\/span>]<\/span> +=<\/span> range_length<\/span>\n        else<\/span>:<\/span>\n            DA_chars<\/span>[<\/span>version<\/span>]<\/span> =<\/span> range_length<\/span>\n\n    # 4. store number of codepoint types per version:<\/span>\n\n    nonchar_type_match<\/span> =<\/span> re<\/span>.<\/span>search<\/span>(<\/span>'^\\<(\\w*)\\-'<\/span>,<\/span> charname<\/span>)<\/span>\n\n    if<\/span> nonchar_type_match<\/span>:<\/span>\n        entity_type<\/span> =<\/span> nonchar_type_match<\/span>.<\/span>group<\/span>(<\/span>1<\/span>)<\/span>\n    else<\/span>:<\/span>\n        entity_type<\/span> =<\/span> 'character'<\/span>\n\n    if<\/span> version<\/span> in<\/span> DA_entities<\/span>:<\/span>               \n        if<\/span> entity_type<\/span> in<\/span> DA_entities<\/span>[<\/span>version<\/span>]:<\/span>\n            DA_entities<\/span>[<\/span>version<\/span>][<\/span>entity_type<\/span>]<\/span> +=<\/span> range_length<\/span>\n        else<\/span>:<\/span>\n            DA_entities<\/span>[<\/span>version<\/span>]<\/span>.<\/span>update<\/span>({<\/span>entity_type<\/span> :<\/span> range_length<\/span>})<\/span>\n    else<\/span>:<\/span>\n        DA_entities<\/span>[<\/span>version<\/span>]<\/span> =<\/span> {<\/span>entity_type<\/span> :<\/span> range_length<\/span>}<\/span>            \n<\/code><\/pre><\/div>\n\n
        Review generated data<\/h2>\n
        ** Get number of codepoints by version:**<\/p>\n
        <\/span>DA_codepoints<\/span>\n<\/code><\/pre><\/div>\n\n
        <\/span>{'1.1': 33979,\n '2.0': 144521,\n '2.1': 2,\n '3.0': 10307,\n '3.1': 44978,\n '3.2': 1016,\n '4.0': 1226,\n '4.1': 1273,\n '5.0': 1369,\n '5.1': 1624,\n '5.2': 6648,\n '6.0': 2088,\n '6.1': 732,\n '6.2': 1,\n '6.3': 5,\n '7.0': 2834,\n '8.0': 7716,\n '9.0': 7500}\n<\/code><\/pre><\/div>\n\n
        Does our codepoint count match the Age<\/code> property values explicitly given in DerivedAge.txt<\/code>?:<\/p>\n
        <\/span>DA_codepoints<\/span> ==<\/span> derivedage_properties<\/span>\n<\/code><\/pre><\/div>\n\n
        <\/span>True\n<\/code><\/pre><\/div>\n\n
        ** Number of new characters per version: **<\/p>\n
        <\/span>DA_chars<\/span>\n<\/code><\/pre><\/div>\n\n
        <\/span>{'1.1': 27512,\n '2.0': 11373,\n '2.1': 2,\n '3.0': 10307,\n '3.1': 44946,\n '3.2': 1016,\n '4.0': 1226,\n '4.1': 1273,\n '5.0': 1369,\n '5.1': 1624,\n '5.2': 6648,\n '6.0': 2088,\n '6.1': 732,\n '6.2': 1,\n '6.3': 5,\n '7.0': 2834,\n '8.0': 7716,\n '9.0': 7500}\n<\/code><\/pre><\/div>\n\n
        ** Number of new noncharacters per version: **<\/p>\n
        <\/span>DA_other<\/span>\n<\/code><\/pre><\/div>\n\n
        <\/span>{'1.1': 6467, '2.0': 133148, '3.1': 32}\n<\/code><\/pre><\/div>\n\n
        ** Breakdown of codepoint types by version **<\/p>\n
        <\/span>DA_entities<\/span>\n<\/code><\/pre><\/div>\n\n
        <\/span>{'1.1': {'character': 27512,\n  'control': 65,\n  'noncharacter': 2,\n  'private': 6400},\n '2.0': {'character': 11373,\n  'noncharacter': 32,\n  'private': 131068,\n  'surrogate': 2048},\n '2.1': {'character': 2},\n '3.0': {'character': 10307},\n '3.1': {'character': 44946, 'noncharacter': 32},\n '3.2': {'character': 1016},\n '4.0': {'character': 1226},\n '4.1': {'character': 1273},\n '5.0': {'character': 1369},\n '5.1': {'character': 1624},\n '5.2': {'character': 6648},\n '6.0': {'character': 2088},\n '6.1': {'character': 732},\n '6.2': {'character': 1},\n '6.3': {'character': 5},\n '7.0': {'character': 2834},\n '8.0': {'character': 7716},\n '9.0': {'character': 7500}}\n<\/code><\/pre><\/div>\n\n
        Present the data<\/h2>\n
        We will use the PrettyTable<\/code> module to present the data using text tables<\/p>\n
        New codepoints and codepoint types by version<\/h3>\n
        <\/span>x<\/span> =<\/span> PrettyTable<\/span>([<\/span>'version'<\/span>,<\/span> 'new codepoints'<\/span>,<\/span> 'new chars'<\/span>,<\/span> 'new other'<\/span>])<\/span>\n\nfor<\/span> k<\/span>,<\/span> v<\/span> in<\/span> DA_codepoints<\/span>.<\/span>items<\/span>():<\/span>\n    x<\/span>.<\/span>add_row<\/span>([<\/span>k<\/span>,<\/span> v<\/span>,<\/span> DA_chars<\/span>.<\/span>get<\/span>(<\/span>k<\/span>,<\/span> '-'<\/span>),<\/span> DA_other<\/span>.<\/span>get<\/span>(<\/span>k<\/span>,<\/span> '-'<\/span>)])<\/span>\n\nprint<\/span> (<\/span>x<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
        <\/span>+---------+----------------+-----------+-----------+\n| version | new codepoints | new chars | new other |\n+---------+----------------+-----------+-----------+\n|   1.1   |     33979      |   27512   |    6467   |\n|   2.0   |     144521     |   11373   |   133148  |\n|   2.1   |       2        |     2     |     -     |\n|   3.0   |     10307      |   10307   |     -     |\n|   3.1   |     44978      |   44946   |     32    |\n|   3.2   |      1016      |    1016   |     -     |\n|   4.0   |      1226      |    1226   |     -     |\n|   4.1   |      1273      |    1273   |     -     |\n|   5.0   |      1369      |    1369   |     -     |\n|   5.1   |      1624      |    1624   |     -     |\n|   5.2   |      6648      |    6648   |     -     |\n|   6.0   |      2088      |    2088   |     -     |\n|   6.1   |      732       |    732    |     -     |\n|   6.2   |       1        |     1     |     -     |\n|   6.3   |       5        |     5     |     -     |\n|   7.0   |      2834      |    2834   |     -     |\n|   8.0   |      7716      |    7716   |     -     |\n|   9.0   |      7500      |    7500   |     -     |\n+---------+----------------+-----------+-----------+\n<\/code><\/pre><\/div>\n\n
        Growth of codepoints, characters, noncharacters by version<\/h3>\n
        Declare three lists:<\/p>\n
        <\/span>DA_char_totals<\/span> =<\/span> []<\/span>\nDA_other_totals<\/span> =<\/span> []<\/span>\nDA_codepoint_totals<\/span> =<\/span> []<\/span>\n<\/code><\/pre><\/div>\n\n
        <\/span>count_codepoints<\/span> =<\/span> 0<\/span>\ncount_chars<\/span> =<\/span> 0<\/span>\ncount_other<\/span> =<\/span> 0<\/span>\n\nz<\/span> =<\/span> PrettyTable<\/span>([<\/span>'version'<\/span>,<\/span> 'total codepoints'<\/span>,<\/span> 'total chars'<\/span>,<\/span> 'total other'<\/span>])<\/span>\n\n# for each version in DA_codepoints, get the corresponding values <\/span>\n# from DA_chars and DA_other. Produce running count of the values <\/span>\n# for each version.<\/span>\n\nfor<\/span> k<\/span>,<\/span> v<\/span> in<\/span> DA_codepoints<\/span>.<\/span>items<\/span>():<\/span>\n\n    count_codepoints<\/span> +=<\/span> DA_codepoints<\/span>.<\/span>get<\/span>(<\/span>k<\/span>,<\/span> 0<\/span>)<\/span>\n    count_chars<\/span> +=<\/span> DA_chars<\/span>.<\/span>get<\/span>(<\/span>k<\/span>,<\/span> 0<\/span>)<\/span>\n    count_other<\/span> +=<\/span> DA_other<\/span>.<\/span>get<\/span>(<\/span>k<\/span>,<\/span> 0<\/span>)<\/span>\n\n    DA_char_totals<\/span>.<\/span>append<\/span>([<\/span>k<\/span>,<\/span> DA_chars<\/span>.<\/span>get<\/span>(<\/span>k<\/span>,<\/span> 0<\/span>),<\/span> count_chars<\/span>])<\/span>\n    DA_other_totals<\/span>.<\/span>append<\/span>([<\/span>k<\/span>,<\/span> DA_other<\/span>.<\/span>get<\/span>(<\/span>k<\/span>,<\/span> 0<\/span>),<\/span> count_other<\/span>])<\/span>\n    DA_codepoint_totals<\/span>.<\/span>append<\/span>([<\/span>k<\/span>,<\/span> DA_codepoints<\/span>.<\/span>get<\/span>(<\/span>k<\/span>,<\/span> 0<\/span>),<\/span> count_codepoints<\/span>])<\/span>\n\n    z<\/span>.<\/span>add_row<\/span>([<\/span>k<\/span>,<\/span> count_codepoints<\/span>,<\/span> count_chars<\/span>,<\/span> count_other<\/span>])<\/span>\n\nprint<\/span> (<\/span>z<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
        <\/span>+---------+------------------+-------------+-------------+\n| version | total codepoints | total chars | total other |\n+---------+------------------+-------------+-------------+\n|   1.1   |      33979       |    27512    |     6467    |\n|   2.0   |      178500      |    38885    |    139615   |\n|   2.1   |      178502      |    38887    |    139615   |\n|   3.0   |      188809      |    49194    |    139615   |\n|   3.1   |      233787      |    94140    |    139647   |\n|   3.2   |      234803      |    95156    |    139647   |\n|   4.0   |      236029      |    96382    |    139647   |\n|   4.1   |      237302      |    97655    |    139647   |\n|   5.0   |      238671      |    99024    |    139647   |\n|   5.1   |      240295      |    100648   |    139647   |\n|   5.2   |      246943      |    107296   |    139647   |\n|   6.0   |      249031      |    109384   |    139647   |\n|   6.1   |      249763      |    110116   |    139647   |\n|   6.2   |      249764      |    110117   |    139647   |\n|   6.3   |      249769      |    110122   |    139647   |\n|   7.0   |      252603      |    112956   |    139647   |\n|   8.0   |      260319      |    120672   |    139647   |\n|   9.0   |      267819      |    128172   |    139647   |\n+---------+------------------+-------------+-------------+\n<\/code><\/pre><\/div>\n\n
        <\/span>z<\/span> =<\/span> PrettyTable<\/span>([<\/span>'version'<\/span>,<\/span> 'new codepoints'<\/span>,<\/span> 'total codepoints'<\/span>])<\/span>\n\nfor<\/span> e<\/span> in<\/span> DA_codepoint_totals<\/span>:<\/span>\n    z<\/span>.<\/span>add_row<\/span>([<\/span>x<\/span> for<\/span> x<\/span> in<\/span> e<\/span>])<\/span>\n\nprint<\/span> (<\/span>z<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
        <\/span>+---------+----------------+------------------+\n| version | new codepoints | total codepoints |\n+---------+----------------+------------------+\n|   1.1   |     33979      |      33979       |\n|   2.0   |     144521     |      178500      |\n|   2.1   |       2        |      178502      |\n|   3.0   |     10307      |      188809      |\n|   3.1   |     44978      |      233787      |\n|   3.2   |      1016      |      234803      |\n|   4.0   |      1226      |      236029      |\n|   4.1   |      1273      |      237302      |\n|   5.0   |      1369      |      238671      |\n|   5.1   |      1624      |      240295      |\n|   5.2   |      6648      |      246943      |\n|   6.0   |      2088      |      249031      |\n|   6.1   |      732       |      249763      |\n|   6.2   |       1        |      249764      |\n|   6.3   |       5        |      249769      |\n|   7.0   |      2834      |      252603      |\n|   8.0   |      7716      |      260319      |\n|   9.0   |      7500      |      267819      |\n+---------+----------------+------------------+\n<\/code><\/pre><\/div>\n\n
        <\/span>z<\/span> =<\/span> PrettyTable<\/span>([<\/span>'version'<\/span>,<\/span> 'new chars'<\/span>,<\/span> 'total chars'<\/span>])<\/span>\n\nfor<\/span> e<\/span> in<\/span> DA_char_totals<\/span>:<\/span>\n    z<\/span>.<\/span>add_row<\/span>([<\/span>x<\/span> for<\/span> x<\/span> in<\/span> e<\/span>])<\/span>\n\nprint<\/span> (<\/span>z<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
        <\/span>+---------+-----------+-------------+\n| version | new chars | total chars |\n+---------+-----------+-------------+\n|   1.1   |   27512   |    27512    |\n|   2.0   |   11373   |    38885    |\n|   2.1   |     2     |    38887    |\n|   3.0   |   10307   |    49194    |\n|   3.1   |   44946   |    94140    |\n|   3.2   |    1016   |    95156    |\n|   4.0   |    1226   |    96382    |\n|   4.1   |    1273   |    97655    |\n|   5.0   |    1369   |    99024    |\n|   5.1   |    1624   |    100648   |\n|   5.2   |    6648   |    107296   |\n|   6.0   |    2088   |    109384   |\n|   6.1   |    732    |    110116   |\n|   6.2   |     1     |    110117   |\n|   6.3   |     5     |    110122   |\n|   7.0   |    2834   |    112956   |\n|   8.0   |    7716   |    120672   |\n|   9.0   |    7500   |    128172   |\n+---------+-----------+-------------+\n<\/code><\/pre><\/div>\n\n
        <\/span>z<\/span> =<\/span> PrettyTable<\/span>([<\/span>'version'<\/span>,<\/span> 'new other'<\/span>,<\/span> 'total other'<\/span>])<\/span>\n\nfor<\/span> e<\/span> in<\/span> DA_other_totals<\/span>:<\/span>\n    z<\/span>.<\/span>add_row<\/span>([<\/span>x<\/span> for<\/span> x<\/span> in<\/span> e<\/span>])<\/span>\n\nprint<\/span> (<\/span>z<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
        <\/span>+---------+-----------+-------------+\n| version | new other | total other |\n+---------+-----------+-------------+\n|   1.1   |    6467   |     6467    |\n|   2.0   |   133148  |    139615   |\n|   2.1   |     0     |    139615   |\n|   3.0   |     0     |    139615   |\n|   3.1   |     32    |    139647   |\n|   3.2   |     0     |    139647   |\n|   4.0   |     0     |    139647   |\n|   4.1   |     0     |    139647   |\n|   5.0   |     0     |    139647   |\n|   5.1   |     0     |    139647   |\n|   5.2   |     0     |    139647   |\n|   6.0   |     0     |    139647   |\n|   6.1   |     0     |    139647   |\n|   6.2   |     0     |    139647   |\n|   6.3   |     0     |    139647   |\n|   7.0   |     0     |    139647   |\n|   8.0   |     0     |    139647   |\n|   9.0   |     0     |    139647   |\n+---------+-----------+-------------+\n<\/code><\/pre><\/div>\n\n
        Commentary<\/h2>\n
        This exercise helped us to analyze the number of codepoints added to Unicode over time using \nDerivedAge.txt<\/code>. The data provides a high-level overview of the number and types of \ncodepoints that have been assigned. The UCD<\/span> contains other data that provide more details on \nthe characters, noncharacters, and other entities in the standard. I will explore these files \nin subsequent posts in order to expand our understanding of Unicode data through the \nUnicode Character Database.<\/p>","category":{"@attributes":{"term":"articles"}}},{"title":"Python function to transliterate\u00a0Devanagari","link":{"@attributes":{"href":"https:\/\/pandey.github.io\/posts\/transliterate-devanagari-to-latin.html","rel":"alternate"}},"published":"2016-09-15T00:00:00-05:00","updated":"2016-09-15T00:00:00-05:00","author":{"name":"Anshuman Pandey"},"id":"tag:pandey.github.io,2016-09-15:\/posts\/transliterate-devanagari-to-latin.html","summary":"
        Transliteration between complex scripts and Latin often requires \nmore than a one-to-one mapping table. The underlying difficulty \narises from the typology of complex scripts, such as those of the \nIndic family, which are alpha-syllabic. One issue with transliterating \nIndic scripts to Latin is handling the inherent a<\/em> of consonant letters \u2026<\/p>","content":"
        Transliteration between complex scripts and Latin often requires \nmore than a one-to-one mapping table. The underlying difficulty \narises from the typology of complex scripts, such as those of the \nIndic family, which are alpha-syllabic. One issue with transliterating \nIndic scripts to Latin is handling the inherent a<\/em> of consonant letters. \nIn some contexts, a consonant letter should be transliterated using only \nthe bare consonant, ie. k<\/em>, kh<\/em>, g<\/em>, gh<\/em>, \u1e45<\/em>, while in others \nthe inherent a<\/em> is required, ie. ka<\/em>, kha<\/em>, ga<\/em>, gha<\/em>, \u1e45a<\/em>.<\/p>\n
        <\/span>import<\/span> <\/span>re<\/span>\n<\/code><\/pre><\/div>\n\n
        Define conversion maps<\/h2>\n
        Define mapping table as dictionary:<\/p>\n
        <\/span>conversiontable<\/span> =<\/span> {<\/span>\n    '\u0950'<\/span> :<\/span> 'o\u1e41'<\/span>,<\/span>\n    '\u0900'<\/span> :<\/span> '\u1e41'<\/span>,<\/span>\n    '\u0901'<\/span> :<\/span> '\u1e43'<\/span>,<\/span>\n    '\u0902'<\/span> :<\/span> '\u1e43'<\/span>,<\/span>\n    '\u0903'<\/span> :<\/span> '\u1e25'<\/span>,<\/span>\n    '\u0905'<\/span> :<\/span> 'a'<\/span>,<\/span>\n    '\u0906'<\/span> :<\/span> '\u0101'<\/span>,<\/span>\n    '\u0907'<\/span> :<\/span> 'i'<\/span>,<\/span>\n    '\u0908'<\/span> :<\/span> '\u012b'<\/span>,<\/span>\n    '\u0909'<\/span> :<\/span> 'u'<\/span>,<\/span>\n    '\u090a'<\/span> :<\/span> '\u016b'<\/span>,<\/span>\n    '\u090b'<\/span> :<\/span> 'r\u0325'<\/span>,<\/span>\n    '\u0960'<\/span> :<\/span> ' r\u0325\u0304'<\/span>,<\/span>\n    '\u090c'<\/span> :<\/span> 'l\u0325'<\/span>,<\/span>\n    '\u0961'<\/span> :<\/span> ' l\u0325\u0304'<\/span>,<\/span>\n    '\u090d'<\/span> :<\/span> '\u00ea'<\/span>,<\/span>\n    '\u090e'<\/span> :<\/span> 'e'<\/span>,<\/span>\n    '\u090f'<\/span> :<\/span> 'e'<\/span>,<\/span>\n    '\u0910'<\/span> :<\/span> 'ai'<\/span>,<\/span>\n    '\u0911'<\/span> :<\/span> '\u00f4'<\/span>,<\/span>\n    '\u0912'<\/span> :<\/span> 'o'<\/span>,<\/span>\n    '\u0913'<\/span> :<\/span> 'o'<\/span>,<\/span>\n    '\u0914'<\/span> :<\/span> 'au'<\/span>,<\/span>\n    '\u093e'<\/span> :<\/span> '\u0101'<\/span>,<\/span>\n    '\u093f'<\/span> :<\/span> 'i'<\/span>,<\/span>\n    '\u0940'<\/span> :<\/span> '\u012b'<\/span>,<\/span>\n    '\u0941'<\/span> :<\/span> 'u'<\/span>,<\/span>\n    '\u0942'<\/span> :<\/span> '\u016b'<\/span>,<\/span>\n    '\u0943'<\/span> :<\/span> 'r\u0325'<\/span>,<\/span>\n    '\u0944'<\/span> :<\/span> ' r\u0325\u0304'<\/span>,<\/span>\n    '\u0962'<\/span> :<\/span> 'l\u0325'<\/span>,<\/span>\n    '\u0963'<\/span> :<\/span> ' l\u0325\u0304'<\/span>,<\/span>\n    '\u0945'<\/span> :<\/span> '\u00ea'<\/span>,<\/span>\n    '\u0947'<\/span> :<\/span> 'e'<\/span>,<\/span>\n    '\u0948'<\/span> :<\/span> 'ai'<\/span>,<\/span>\n    '\u0949'<\/span> :<\/span> '\u00f4'<\/span>,<\/span>\n    '\u094b'<\/span> :<\/span> 'o'<\/span>,<\/span>\n    '\u094c'<\/span> :<\/span> 'au'<\/span>,<\/span>\n    '\u0915\u093c'<\/span> :<\/span> 'q'<\/span>,<\/span>\n    '\u0915'<\/span> :<\/span> 'k'<\/span>,<\/span>\n    '\u0916\u093c'<\/span> :<\/span> 'x'<\/span>,<\/span>\n    '\u0916'<\/span> :<\/span> 'kh'<\/span>,<\/span>\n    '\u0917\u093c'<\/span> :<\/span> '\u0121'<\/span>,<\/span>\n    '\u0917'<\/span> :<\/span> 'g'<\/span>,<\/span>\n    '\u097b'<\/span> :<\/span> 'g'<\/span>,<\/span>\n    '\u0918'<\/span> :<\/span> 'gh'<\/span>,<\/span>\n    '\u0919'<\/span> :<\/span> '\u1e45'<\/span>,<\/span>\n    '\u091a'<\/span> :<\/span> 'c'<\/span>,<\/span>\n    '\u091b'<\/span> :<\/span> 'ch'<\/span>,<\/span>\n    '\u091c\u093c'<\/span> :<\/span> 'z'<\/span>,<\/span>\n    '\u091c'<\/span> :<\/span> 'j'<\/span>,<\/span>\n    '\u097c'<\/span> :<\/span> 'j'<\/span>,<\/span>\n    '\u091d'<\/span> :<\/span> 'jh'<\/span>,<\/span>\n    '\u091e'<\/span> :<\/span> '\u00f1'<\/span>,<\/span>\n    '\u091f'<\/span> :<\/span> '\u1e6d'<\/span>,<\/span>\n    '\u0920'<\/span> :<\/span> '\u1e6dh'<\/span>,<\/span>\n    '\u0921\u093c'<\/span> :<\/span> '\u1e5b'<\/span>,<\/span>\n    '\u0921'<\/span> :<\/span> '\u1e0d'<\/span>,<\/span>\n    '\u0978'<\/span> :<\/span> '\u1e0d'<\/span>,<\/span>\n    '\u097e'<\/span> :<\/span> 'd'<\/span>,<\/span>\n    '\u0922\u093c'<\/span> :<\/span> '\u1e5bh'<\/span>,<\/span>\n    '\u0922'<\/span> :<\/span> '\u1e0dh'<\/span>,<\/span>\n    '\u0923'<\/span> :<\/span> '\u1e47'<\/span>,<\/span>\n    '\u0924'<\/span> :<\/span> 't'<\/span>,<\/span>\n    '\u0925'<\/span> :<\/span> 'th'<\/span>,<\/span>\n    '\u0926'<\/span> :<\/span> 'd'<\/span>,<\/span>\n    '\u0927'<\/span> :<\/span> 'dh'<\/span>,<\/span>\n    '\u0928'<\/span> :<\/span> 'n'<\/span>,<\/span>\n    '\u092a'<\/span> :<\/span> 'p'<\/span>,<\/span>\n    '\u092b\u093c'<\/span> :<\/span> 'f'<\/span>,<\/span>\n    '\u092b'<\/span> :<\/span> 'ph'<\/span>,<\/span>\n    '\u092c'<\/span> :<\/span> 'b'<\/span>,<\/span>\n    '\u097f'<\/span> :<\/span> 'b'<\/span>,<\/span>\n    '\u092d'<\/span> :<\/span> 'bh'<\/span>,<\/span>\n    '\u092e'<\/span> :<\/span> 'm'<\/span>,<\/span>\n    '\u092f'<\/span> :<\/span> 'y'<\/span>,<\/span>\n    '\u0930'<\/span> :<\/span> 'r'<\/span>,<\/span>\n    '\u0932'<\/span> :<\/span> 'l'<\/span>,<\/span>\n    '\u0933'<\/span> :<\/span> '\u1e37'<\/span>,<\/span>\n    '\u0935'<\/span> :<\/span> 'v'<\/span>,<\/span>\n    '\u0936'<\/span> :<\/span> '\u015b'<\/span>,<\/span>\n    '\u0937'<\/span> :<\/span> '\u1e63'<\/span>,<\/span>\n    '\u0938'<\/span> :<\/span> 's'<\/span>,<\/span>\n    '\u0939'<\/span> :<\/span> 'h'<\/span>,<\/span>\n    '\u093d'<\/span> :<\/span> '<\/span>\\'<\/span>'<\/span>,<\/span>\n    '\u094d'<\/span> :<\/span> ''<\/span>,<\/span>\n    '\u093c'<\/span> :<\/span> ''<\/span>,<\/span>\n    '\u0966'<\/span> :<\/span> '0'<\/span>,<\/span>\n    '\u0967'<\/span> :<\/span> '1'<\/span>,<\/span>\n    '\u0968'<\/span> :<\/span> '2'<\/span>,<\/span>\n    '\u0969'<\/span> :<\/span> '3'<\/span>,<\/span>\n    '\u096a'<\/span> :<\/span> '4'<\/span>,<\/span>\n    '\u096b'<\/span> :<\/span> '5'<\/span>,<\/span>\n    '\u096c'<\/span> :<\/span> '6'<\/span>,<\/span>\n    '\u096d'<\/span> :<\/span> '7'<\/span>,<\/span>\n    '\u096e'<\/span> :<\/span> '8'<\/span>,<\/span>\n    '\u096f'<\/span> :<\/span> '9'<\/span>,<\/span>\n    '\ua8f3'<\/span> :<\/span> '\u1e41'<\/span>,<\/span>\n    '\u0964'<\/span> :<\/span> '.'<\/span>,<\/span>\n    '\u0965'<\/span> :<\/span> '..'<\/span>,<\/span>\n    ' '<\/span> :<\/span> ' '<\/span>,<\/span>\n    }<\/span>\n<\/code><\/pre><\/div>\n\n
        Define character classes:<\/p>\n
        <\/span>consonants<\/span> =<\/span> '<\/span>\\u0915<\/span>-<\/span>\\u0939\\u0958<\/span>-<\/span>\\u095F\\u0978<\/span>-<\/span>\\u097C\\u097E<\/span>-<\/span>\\u097F<\/span>'<\/span>\nvowelsigns<\/span> =<\/span> '<\/span>\\u093E<\/span>-<\/span>\\u094C\\u093A<\/span>-<\/span>\\u093B\\u094E<\/span>-<\/span>\\u094F\\u0955<\/span>-<\/span>\\u0957<\/span>'<\/span>\nnukta<\/span> =<\/span> '<\/span>\\u093C<\/span>'<\/span>\nvirama<\/span> =<\/span> '<\/span>\\u094D<\/span>'<\/span>\n\ndevanagarichars<\/span> =<\/span> '<\/span>\\u0900<\/span>-<\/span>\\u097F\\u1CD0<\/span>-<\/span>\\u1CFF\\uA8E0<\/span>-<\/span>\\uA8FF<\/span>'<\/span>\n<\/code><\/pre><\/div>\n\n
        Define functions<\/h2>\n
        deva_to_latn()<\/h3>\n
        Primary transliteration function. Performs the following general operations:<\/p>\n
          \n
        1. Loops over each character in a word<\/li>\n
        2. If character is a Devanagari character, then continue processing, otherwise go to next character<\/li>\n
        3. Most characters can be transliterated by mapping to Latin characters in the conversationtable<\/code> dictionary. Consonants need additional attention.<\/li>\n
        4. Perform lookahead check to determine if an ‘a<\/code>‘<\/span> needs to be appended after a consonant.<\/li>\n<\/ol>\n
          <\/span>def<\/span> <\/span>deva_to_latn<\/span>(<\/span>text<\/span>):<\/span>\n\n    word<\/span> =<\/span> text<\/span>.<\/span>strip<\/span>()<\/span>\n\n    # define a buffer to store the transliteration<\/span>\n    curr<\/span> =<\/span> ''<\/span>\n\n    for<\/span> index<\/span>,<\/span> char<\/span> in<\/span> enumerate<\/span>(<\/span>word<\/span>):<\/span>\n\n        # check if char is a Devanagari character. if true then continue processing.<\/span>\n        # otherwise, output char to curr<\/span>\n\n        if<\/span> re<\/span>.<\/span>match<\/span>(<\/span>'['<\/span> +<\/span> devanagarichars<\/span> +<\/span> ']'<\/span>,<\/span> char<\/span>):<\/span>\n\n            # if char = consonant, then its transliteration is dependent upon various<\/span>\n            # factors. need to check if next char = nukta, virama, vowel sign.<\/span>\n\n            if<\/span> re<\/span>.<\/span>match<\/span>(<\/span>'['<\/span> +<\/span> consonants<\/span> +<\/span> ']'<\/span>,<\/span> char<\/span>):<\/span>\n\n                # check next char<\/span>\n                nextchar<\/span> =<\/span> word<\/span>[(<\/span>index<\/span> +<\/span> 1<\/span>)<\/span> %<\/span> len<\/span>(<\/span>word<\/span>)]<\/span>\n\n                if<\/span> nextchar<\/span>:<\/span>\n\n                    # if next char = nukta, then add present char and nukta <\/span>\n                    # to 'cons'. else just add present char. set variable<\/span>\n                    # to test for nukta when processing next char<\/span>\n\n                    if<\/span> re<\/span>.<\/span>match<\/span>(<\/span>'['<\/span> +<\/span> nukta<\/span> +<\/span> ']'<\/span>,<\/span> nextchar<\/span>):<\/span>\n                        cons<\/span> =<\/span> char<\/span> +<\/span> nextchar<\/span>\n                        nukta_present<\/span> =<\/span> 1<\/span>\n                    else<\/span>:<\/span>\n                        cons<\/span> =<\/span> char<\/span>\n                        nukta_present<\/span> =<\/span> 0<\/span>\n\n                    # if present char is nukta, then check next char<\/span>\n\n                    if<\/span> nukta_present<\/span>:<\/span>\n                        nextchar<\/span> =<\/span> word<\/span>[(<\/span>index<\/span> +<\/span> 2<\/span>)<\/span> %<\/span> len<\/span>(<\/span>word<\/span>)]<\/span>\n\n                    # if next char = combining sign or virama, convert consonant <\/span>\n                    # without "a". else if next char != combining sign or virama, <\/span>\n                    # add "a" to consonant<\/span>\n\n                    if<\/span> re<\/span>.<\/span>match<\/span>(<\/span>'['<\/span> +<\/span> vowelsigns<\/span> +<\/span> virama<\/span> +<\/span>']'<\/span>,<\/span> nextchar<\/span>):<\/span>\n                        trans<\/span> =<\/span> conversiontable<\/span>.<\/span>get<\/span>(<\/span>cons<\/span>,<\/span> ''<\/span>)<\/span>\n                        curr<\/span> =<\/span> curr<\/span> +<\/span> trans<\/span>\n                    else<\/span>:<\/span>\n                        trans<\/span> =<\/span> conversiontable<\/span>.<\/span>get<\/span>(<\/span>cons<\/span>,<\/span> ''<\/span>)<\/span>\n                        trans<\/span> =<\/span> trans<\/span> +<\/span> "a"<\/span>\n                        curr<\/span> =<\/span> curr<\/span> +<\/span> trans<\/span>\n\n            # transliterate all other chars<\/span>\n\n            else<\/span>:<\/span>\n                trans<\/span> =<\/span> conversiontable<\/span>.<\/span>get<\/span>(<\/span>char<\/span>,<\/span> ''<\/span>)<\/span>\n                curr<\/span> =<\/span> curr<\/span> +<\/span> trans<\/span>\n\n        # char is not Devanagari. output char to curr<\/span>\n\n        else<\/span>:<\/span>\n             curr<\/span> =<\/span> curr<\/span> +<\/span> char<\/span>\n\n    return<\/span> curr<\/span>\n<\/code><\/pre><\/div>\n\n
          getLatin()<\/h3>\n
          Function that preprocesses Devanagari input and passes it to deva_to_latn()<\/code><\/p>\n
          <\/span>def<\/span> <\/span>getLatin<\/span>(<\/span>inputtext<\/span>):<\/span>\n\n    word_syllables<\/span> =<\/span> []<\/span>\n    all_words<\/span> =<\/span> []<\/span>\n\n    for<\/span> word<\/span> in<\/span> inputtext<\/span>.<\/span>split<\/span>():<\/span>\n\n        latin_output<\/span> =<\/span> deva_to_latn<\/span>(<\/span>word<\/span>)<\/span>\n        all_words<\/span>.<\/span>append<\/span>(<\/span>latin_output<\/span>)<\/span>\n        joined_all_words<\/span> =<\/span> ' '<\/span>.<\/span>join<\/span>(<\/span>all_words<\/span>)<\/span>\n\n    return<\/span> joined_all_words<\/span>\n<\/code><\/pre><\/div>\n\n
          Test transliteration routines<\/h2>\n
          Test some Sanskrit text in Devanagari with Vedic ‘accents’. Accents are ignored at present. <\/p>\n
          <\/span>text<\/span> =<\/span> "\u0905\u0952\u0917\u094d\u0928\u093f\u092e\u0940\u0951\u0933\u0947 \u092a\u0941\u0952\u0930\u094b\u0939\u093f\u0951\u0924\u0902 \u092f\u0952\u091c\u094d\u091e\u0938\u094d\u092f\u0951 \u0926\u0947\u0952\u0935\u092e\u0943\u0952\u0924\u094d\u0935\u093f\u091c\u0951\u092e\u094d\u0964 \u0939\u094b\u0924\u093e\u0951\u0930\u0902 \u0930\u0924\u094d\u0928\u0952\u0927\u093e\u0924\u0951\u092e\u092e\u094d\u0965"<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>getLatin<\/span>(<\/span>text<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>'agnim\u012b\u1e37e purohita\u1e43 yaj\u00f1asya devamr\u0325tvijam. hot\u0101ra\u1e43 ratnadh\u0101tamam..'\n<\/code><\/pre><\/div>\n\n
          Test some Hindi text. Word-final inherent ‘a<\/code>‘<\/span> is generally not transliterated in modern Hindi.<\/p>\n
          <\/span>text<\/span> =<\/span> "\u0909\u0928\u094d\u0939\u0947\u0902 \u092c\u0941\u0926\u094d\u0927\u093f \u0914\u0930 \u0905\u0928\u094d\u0924\u0930\u093e\u0924\u094d\u092e\u093e \u0915\u0940 \u0926\u0947\u0928 \u092a\u094d\u0930\u093e\u092a\u094d\u0924 \u0939\u0948 \u0914\u0930 \u092a\u0930\u0938\u094d\u092a\u0930 \u0909\u0928\u094d\u0939\u0947\u0902 \u092d\u093e\u0908\u091a\u093e\u0930\u0947 \u0915\u0947 \u092d\u093e\u0935 \u0938\u0947 \u092c\u0930\u094d\u0924\u093e\u0935 \u0915\u0930\u0928\u093e \u091a\u093e\u0939\u093f\u090f"<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>getLatin<\/span>(<\/span>text<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>'unhe\u1e43 buddhi aura antar\u0101tm\u0101 k\u012b dena pr\u0101pta hai aura paraspara unhe\u1e43 bh\u0101\u012bc\u0101re ke bh\u0101va se bart\u0101va karan\u0101 c\u0101hie'\n<\/code><\/pre><\/div>\n\n
          Test some mixed script text.<\/p>\n
          <\/span>text<\/span> =<\/span> "This is in the \u0926\u0947\u0935\u0928\u093e\u0917\u0930\u0940 script"<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>getLatin<\/span>(<\/span>text<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>'This is in the devan\u0101gar\u012b script'\n<\/code><\/pre><\/div>\n\n
          Next steps<\/h1>\n
          The deva_to_latn()<\/code> function can be adapted for transliterating other Indic scripts.<\/p>","category":{"@attributes":{"term":"articles"}}},{"title":"Tokenizing Indic strings by syllables in\u00a0Python","link":{"@attributes":{"href":"https:\/\/pandey.github.io\/posts\/tokenize-indic-syllables-python.html","rel":"alternate"}},"published":"2016-09-05T00:00:00-05:00","updated":"2016-09-05T00:00:00-05:00","author":{"name":"Anshuman Pandey"},"id":"tag:pandey.github.io,2016-09-05:\/posts\/tokenize-indic-syllables-python.html","summary":"
          Processing text in non-Latin scripts in Python has become much easier on account \nof the native support for Unicode in version 3. There are still some challenges, \nbut these are issues related more to individual writing systems than to Python. \nOne such issue is the manner in which strings of \u2026<\/p>","content":"
          Processing text in non-Latin scripts in Python has become much easier on account \nof the native support for Unicode in version 3. There are still some challenges, \nbut these are issues related more to individual writing systems than to Python. \nOne such issue is the manner in which strings of various script typologies are \nhandled. While handling strings in alphabetic<\/em> systems, such as the Latin \nscript in which this post is written, is relatively straightforward, scripts \nthat have an alpha-syllabic<\/em> structure often require additional processing. <\/p>\n
          For instance, the string “English” has the following number of characters:<\/p>\n
          <\/span>len<\/span>(<\/span>'English'<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>7<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>list<\/span>(<\/span>'English'<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>['E', 'n', 'g', 'l', 'i', 's', 'h']\n<\/code><\/pre><\/div>\n\n
          Now consider the string ‘\u0926\u0947\u0935\u0928\u093e\u0917\u0930\u0940’, which contains characters from the Devanagari script, \nwhich is part of the large ‘Indic’ family of writing systems:<\/p>\n
          <\/span>len<\/span>(<\/span>'\u0926\u0947\u0935\u0928\u093e\u0917\u0930\u0940'<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>8<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>list<\/span>(<\/span>'\u0926\u0947\u0935\u0928\u093e\u0917\u0930\u0940'<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>['\u0926', '\u0947', '\u0935', '\u0928', '\u093e', '\u0917', '\u0930', '\u0940']\n<\/code><\/pre><\/div>\n\n
          Seems reasonable, doesn’t it? At some level: yes. For practical purposes: no.<\/p>\n
          For Indic scripts, parsing a string by its characters is not as meaningful as it is in Latin. The reason is that Indic scripts are alphasyllabic \nwriting systems, while Latin is an alphabetic system. For alphasyllabic types, \ninstead of individual characters, the basic orthographic unit that carries meaning \nis the syllable. For effective text processing, the Devanagari string should be analyzed as:<\/p>\n
          <\/span>['\u0926\u0947', '\u0935', '\u0928\u093e', '\u0917', '\u0930\u0940']\n<\/code><\/pre><\/div>\n\n
          Let’s find an approach to tokenize Indic strings by syllables in Python.<\/p>\n
          Approach<\/h2>\n
          Import the re<\/code> module for processing regular expressions.<\/p>\n
          <\/span>import<\/span> <\/span>re<\/span>\n<\/code><\/pre><\/div>\n\n
          Define character classes<\/h3>\n
          An orthographic syllable may be defined in terms of character classes. Each syllable has a base, which is either an independent vowel letter or consonant letter. One or more combining signs may attach to a base. Analyze the Unicode Devanagari charts and classify characters.<\/p>\n
          <\/span>vowels<\/span> =<\/span> '<\/span>\\u0904<\/span>-<\/span>\\u0914\\u0960<\/span>-<\/span>\\u0961\\u0972<\/span>-<\/span>\\u0977<\/span>'<\/span>\nconsonants<\/span> =<\/span> '<\/span>\\u0915<\/span>-<\/span>\\u0939\\u0958<\/span>-<\/span>\\u095F\\u0978<\/span>-<\/span>\\u097C\\u097E<\/span>-<\/span>\\u097F<\/span>'<\/span>\nglottal<\/span> =<\/span> '<\/span>\\u097D<\/span>'<\/span>\n\nvowel_signs<\/span> =<\/span> '<\/span>\\u093E<\/span>-<\/span>\\u094C\\u093A<\/span>-<\/span>\\u093B\\u094E<\/span>-<\/span>\\u094F\\u0955<\/span>-<\/span>\\u0957\\u1CF8<\/span>-<\/span>\\u1CF9<\/span>'<\/span>\nnasals<\/span> =<\/span> '<\/span>\\u0900<\/span>-<\/span>\\u0902\\u1CF2<\/span>-<\/span>\\u1CF6<\/span>'<\/span>\nvisarga<\/span> =<\/span> '<\/span>\\u0903<\/span>'<\/span>\nnukta<\/span> =<\/span> '<\/span>\\u093C<\/span>'<\/span>\navagraha<\/span> =<\/span> '<\/span>\\u093D<\/span>'<\/span>\nvirama<\/span> =<\/span> '<\/span>\\u094D<\/span>'<\/span>\n\nvedic_signs<\/span> =<\/span> '<\/span>\\u0951<\/span>-<\/span>\\u0952\\u1CD0<\/span>-<\/span>\\u1CE1\\u1CED<\/span>'<\/span>\nvisarga_modifiers<\/span> =<\/span> '<\/span>\\u1CE2<\/span>-<\/span>\\u1CE8<\/span>'<\/span>\ncombining<\/span> =<\/span> '<\/span>\\uA8E0<\/span>-<\/span>\\uA8F1<\/span>'<\/span>\n\nom<\/span> =<\/span> '<\/span>\\u0950<\/span>'<\/span>\n\naccents<\/span> =<\/span> '<\/span>\\u0953<\/span>-<\/span>\\u0954<\/span>'<\/span>\ndandas<\/span> =<\/span> '<\/span>\\u0964<\/span>-<\/span>\\u0965<\/span>'<\/span>\ndigits<\/span> =<\/span> '<\/span>\\u0966<\/span>-<\/span>\\u096F<\/span>'<\/span>\nabbreviation<\/span> =<\/span> '<\/span>\\u0970<\/span>'<\/span>\nspacing<\/span> =<\/span> '<\/span>\\u0971<\/span>'<\/span>\n\nvedic_nasals<\/span> =<\/span> '<\/span>\\uA8F2<\/span>-<\/span>\\uA8F7\\u1CE9<\/span>-<\/span>\\u1CEC\\u1CEE<\/span>-<\/span>\\u1CF1<\/span>'<\/span>\nfillers<\/span> =<\/span> '<\/span>\\uA8F8<\/span>-<\/span>\\uA8F9<\/span>'<\/span>\ncaret<\/span> =<\/span> '<\/span>\\uA8FA<\/span>'<\/span>\nheadstroke<\/span> =<\/span> '<\/span>\\uA8FB<\/span>'<\/span>\n\nspace<\/span> =<\/span> '<\/span>\\u0020<\/span>'<\/span>\njoiners<\/span> =<\/span> '<\/span>\\u200C<\/span>-<\/span>\\u200D<\/span>'<\/span>\n<\/code><\/pre><\/div>\n\n
          Define functions<\/h3>\n
          syllabify()<\/strong> : the basic syllable tokenization function.<\/p>\n
          <\/span>def<\/span> <\/span>syllabify<\/span>(<\/span>inputtext<\/span>):<\/span>\n\n    syllables<\/span> =<\/span> []<\/span>\n    curr<\/span> =<\/span> ''<\/span>\n\n    # iterate over each character in the input. if a char belongs to a <\/span>\n    # class that can be part of a syllable, then add it to the curr <\/span>\n    # buffer. otherwise, output it to syllables[] right away.<\/span>\n\n    for<\/span> char<\/span> in<\/span> inputtext<\/span>:<\/span>\n\n        if<\/span> re<\/span>.<\/span>match<\/span>(<\/span>'['<\/span> +<\/span> vowels<\/span> +<\/span> avagraha<\/span> +<\/span> glottal<\/span> +<\/span> om<\/span> +<\/span> ']'<\/span>,<\/span> char<\/span>):<\/span>\n\n            # need to handle non-initial independent vowel letters,<\/span>\n            # avagraha, and om<\/span>\n\n            if<\/span> curr<\/span> !=<\/span> ''<\/span>:<\/span>\n                syllables<\/span>.<\/span>append<\/span>(<\/span>curr<\/span>)<\/span>\n                curr<\/span> =<\/span> char<\/span>\n            else<\/span>:<\/span>\n                curr<\/span> =<\/span> curr<\/span> +<\/span> char<\/span>\n\n        elif<\/span> re<\/span>.<\/span>match<\/span>(<\/span>'['<\/span> +<\/span> consonants<\/span> +<\/span> ']'<\/span>,<\/span> char<\/span>):<\/span>\n\n            # if last in curr is not virama, output curr as syllable<\/span>\n            # else add present consonant to curr<\/span>\n\n            if<\/span> len<\/span>(<\/span>curr<\/span>)<\/span> ><\/span> 0<\/span> and<\/span> curr<\/span>[<\/span>-<\/span>1<\/span>]<\/span> !=<\/span> virama<\/span>:<\/span>\n                syllables<\/span>.<\/span>append<\/span>(<\/span>curr<\/span>)<\/span>\n                curr<\/span> =<\/span> char<\/span>\n            else<\/span>:<\/span>\n                curr<\/span> =<\/span> curr<\/span> +<\/span> char<\/span>\n\n        elif<\/span> re<\/span>.<\/span>match<\/span>(<\/span>'['<\/span> +<\/span> vowel_signs<\/span> +<\/span> visarga<\/span> +<\/span> vedic_signs<\/span> +<\/span> ']'<\/span>,<\/span> char<\/span>):<\/span>\n            curr<\/span> =<\/span> curr<\/span> +<\/span> char<\/span>\n\n        elif<\/span> re<\/span>.<\/span>match<\/span>(<\/span>'['<\/span> +<\/span> visarga_modifiers<\/span> +<\/span> ']'<\/span>,<\/span> char<\/span>):<\/span>\n\n            if<\/span> len<\/span>(<\/span>curr<\/span>)<\/span> ><\/span> 0<\/span> and<\/span> curr<\/span>[<\/span>-<\/span>1<\/span>]<\/span> ==<\/span> visarga<\/span>:<\/span>\n                curr<\/span> =<\/span> curr<\/span> +<\/span> char<\/span>\n                syllables<\/span>.<\/span>append<\/span>(<\/span>curr<\/span>)<\/span>\n                curr<\/span> =<\/span> ''<\/span>\n            else<\/span>:<\/span>\n                syllables<\/span>.<\/span>append<\/span>(<\/span>curr<\/span>)<\/span>\n                curr<\/span> =<\/span> ''<\/span>\n\n        elif<\/span> re<\/span>.<\/span>match<\/span>(<\/span>'['<\/span> +<\/span> nasals<\/span> +<\/span> vedic_nasals<\/span> +<\/span> ']'<\/span>,<\/span> char<\/span>):<\/span>\n\n            # if last in curr is a vowel sign, output curr as syllable<\/span>\n            # else add present vowel modifier to curr and output as syllable<\/span>\n\n            vowelsign<\/span> =<\/span> re<\/span>.<\/span>match<\/span>(<\/span>'['<\/span> +<\/span> vowel_signs<\/span> +<\/span> ']$'<\/span>,<\/span> curr<\/span>)<\/span>\n            if<\/span> vowelsign<\/span>:<\/span>\n                syllables<\/span>.<\/span>append<\/span>(<\/span>curr<\/span>)<\/span>\n                curr<\/span> =<\/span> ''<\/span>\n            else<\/span>:<\/span>\n                curr<\/span> =<\/span> curr<\/span> +<\/span> char<\/span>\n                syllables<\/span>.<\/span>append<\/span>(<\/span>curr<\/span>)<\/span>\n                curr<\/span> =<\/span> ''<\/span>\n\n        elif<\/span> re<\/span>.<\/span>match<\/span>(<\/span>'['<\/span> +<\/span> nukta<\/span> +<\/span> ']'<\/span>,<\/span> char<\/span>):<\/span>\n            curr<\/span> =<\/span> curr<\/span> +<\/span> char<\/span>\n\n        elif<\/span> re<\/span>.<\/span>match<\/span>(<\/span>'['<\/span> +<\/span> virama<\/span> +<\/span> ']'<\/span>,<\/span> char<\/span>):<\/span>\n            curr<\/span> =<\/span> curr<\/span> +<\/span> char<\/span>\n\n        elif<\/span> re<\/span>.<\/span>match<\/span>(<\/span>'['<\/span> +<\/span> digits<\/span> +<\/span> ']'<\/span>,<\/span> char<\/span>):<\/span>\n            curr<\/span> =<\/span> curr<\/span> +<\/span> char<\/span>\n\n        elif<\/span> re<\/span>.<\/span>match<\/span>(<\/span>'['<\/span> +<\/span> fillers<\/span> +<\/span> headstroke<\/span> +<\/span> ']'<\/span>,<\/span> char<\/span>):<\/span>\n            syllables<\/span>.<\/span>append<\/span>(<\/span>char<\/span>)<\/span>\n\n        elif<\/span> re<\/span>.<\/span>match<\/span>(<\/span>'['<\/span> +<\/span> joiners<\/span> +<\/span> ']'<\/span>,<\/span> char<\/span>):<\/span>\n            curr<\/span> =<\/span> curr<\/span> +<\/span> char<\/span>\n\n        else<\/span>:<\/span>\n            pass<\/span>\n            #print ("unhandled: " + char + " ", char.encode('unicode_escape'))<\/span>\n\n    # handle remaining curr<\/span>\n    if<\/span> curr<\/span> !=<\/span> ''<\/span>:<\/span>\n        syllables<\/span>.<\/span>append<\/span>(<\/span>curr<\/span>)<\/span>\n        curr<\/span> =<\/span> ''<\/span>\n\n    # return each syllable as item in a list<\/span>\n    return<\/span> syllables<\/span>\n<\/code><\/pre><\/div>\n\n
          getSyllables()<\/strong> : splits input string into words and passes them to syllabify().<\/p>\n
          <\/span>def<\/span> <\/span>getSyllables<\/span>(<\/span>inputtext<\/span>):<\/span>\n\n    word_syllables<\/span> =<\/span> []<\/span>\n    all_words<\/span> =<\/span> []<\/span>\n\n    for<\/span> word<\/span> in<\/span> inputtext<\/span>.<\/span>split<\/span>():<\/span>\n\n        word<\/span> =<\/span> word<\/span>.<\/span>strip<\/span>()<\/span>\n        word<\/span> =<\/span> re<\/span>.<\/span>sub<\/span>(<\/span>'[\\s<\/span>\\n\\u0964\\u0965<\/span>\\.]'<\/span>,<\/span> ''<\/span>,<\/span> word<\/span>)<\/span>\n\n        word_syllables<\/span> =<\/span> syllabify<\/span>(<\/span>word<\/span>)<\/span>\n        #number_syllables = len(word_syllables)<\/span>\n\n        #joined_syllables = '\\u00B7'.join(word_syllables)<\/span>\n        joined_syllables<\/span> =<\/span> word_syllables<\/span>\n\n        # make list of lists containing each word<\/span>\n        #all_words.append([word, joined_syllables, number_syllables])<\/span>\n        all_words<\/span>.<\/span>append<\/span>([<\/span>word<\/span>,<\/span> joined_syllables<\/span>])<\/span>\n\n    return<\/span> all_words<\/span>\n<\/code><\/pre><\/div>\n\n
          getSyllableStats()<\/strong> : get number of unique words and syllables in input text.<\/p>\n
          <\/span>def<\/span> <\/span>getSyllableStats<\/span>(<\/span>inputtext<\/span>):<\/span>\n\n    syllcount<\/span> =<\/span> {}<\/span>\n    wordcount<\/span> =<\/span> {}<\/span>\n    word<\/span> =<\/span> ''<\/span>\n    syllable<\/span> =<\/span> ''<\/span>\n    count<\/span> =<\/span> ''<\/span>\n    syllablestatus<\/span> =<\/span> {}<\/span>\n\n    words<\/span> =<\/span> getSyllables<\/span>(<\/span>inputtext<\/span>)<\/span>\n\n    for<\/span> entry<\/span> in<\/span> words<\/span>:<\/span>\n\n        word<\/span> =<\/span> entry<\/span>[<\/span>0<\/span>]<\/span>\n        syllables<\/span> =<\/span> entry<\/span>[<\/span>1<\/span>]<\/span>\n        #count = entry[2]<\/span>\n\n        #word_syllables = syllables.split('\\u00B7')<\/span>\n        word_syllables<\/span> =<\/span> syllables<\/span>\n\n        # count all words in input<\/span>\n        if<\/span> word<\/span> in<\/span> wordcount<\/span>:<\/span>\n            wordcount<\/span>[<\/span>word<\/span>]<\/span> +=<\/span> 1<\/span>\n        else<\/span>:<\/span>\n            wordcount<\/span>[<\/span>word<\/span>]<\/span> =<\/span> 1<\/span>\n\n        # count all syllables in input<\/span>\n        for<\/span> syll<\/span> in<\/span> word_syllables<\/span>:<\/span>\n            if<\/span> syll<\/span> in<\/span> syllcount<\/span>:<\/span>\n                syllcount<\/span>[<\/span>syll<\/span>]<\/span> +=<\/span> 1<\/span>\n            else<\/span>:<\/span>\n                syllcount<\/span>[<\/span>syll<\/span>]<\/span> =<\/span> 1<\/span>\n\n    syllablestatus<\/span>.<\/span>update<\/span>({<\/span>'words'<\/span> :<\/span> wordcount<\/span>})<\/span>\n    syllablestatus<\/span>.<\/span>update<\/span>({<\/span>'syllables'<\/span> :<\/span> syllcount<\/span>})<\/span>\n\n    return<\/span> (<\/span>syllablestatus<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
          Test the functions<\/h2>\n
          Some text in Sanskrit<\/p>\n
          <\/span>text<\/span> =<\/span> "\u0905\u0952\u0917\u094d\u0928\u093f\u092e\u0940\u0951\u0933\u0947 \u092a\u0941\u0952\u0930\u094b\u0939\u093f\u0951\u0924\u0902 \u092f\u0952\u091c\u094d\u091e\u0938\u094d\u092f\u0951 \u0926\u0947\u0952\u0935\u092e\u0943\u0952\u0924\u094d\u0935\u093f\u091c\u0951\u092e\u094d\u0964 \u0939\u094b\u0924\u093e\u0951\u0930\u0902 \u0930\u0924\u094d\u0928\u0952\u0927\u093e\u0924\u0951\u092e\u092e\u094d\u0965"<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>syllabify<\/span>(<\/span>text<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>['\u0905\u0952',\n '\u0917\u094d\u0928\u093f',\n '\u092e\u0940\u0951',\n '\u0933\u0947',\n '\u092a\u0941\u0952',\n '\u0930\u094b',\n '\u0939\u093f\u0951',\n '\u0924\u0902',\n '\u092f\u0952',\n '\u091c\u094d\u091e',\n '\u0938\u094d\u092f\u0951',\n '\u0926\u0947\u0952',\n '\u0935',\n '\u092e\u0943\u0952',\n '\u0924\u094d\u0935\u093f',\n '\u091c\u0951',\n '\u092e\u094d\u0939\u094b',\n '\u0924\u093e\u0951',\n '\u0930\u0902',\n '\u0930',\n '\u0924\u094d\u0928\u0952',\n '\u0927\u093e',\n '\u0924\u0951',\n '\u092e',\n '\u092e\u094d']\n<\/code><\/pre><\/div>\n\n
          <\/span>results<\/span> =<\/span> getSyllables<\/span>(<\/span>text<\/span>)<\/span>\n\nfor<\/span> x<\/span> in<\/span> results<\/span>:<\/span>\n    print<\/span> (<\/span>x<\/span>[<\/span>0<\/span>],<\/span> ":"<\/span>,<\/span> '<\/span>\\u00B7<\/span>'<\/span>.<\/span>join<\/span>(<\/span>x<\/span>[<\/span>1<\/span>]),<\/span> ":"<\/span>,<\/span> len<\/span>(<\/span>x<\/span>[<\/span>1<\/span>]))<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>\u0905\u0952\u0917\u094d\u0928\u093f\u092e\u0940\u0951\u0933\u0947 : \u0905\u0952\u00b7\u0917\u094d\u0928\u093f\u00b7\u092e\u0940\u0951\u00b7\u0933\u0947 : 4\n\u092a\u0941\u0952\u0930\u094b\u0939\u093f\u0951\u0924\u0902 : \u092a\u0941\u0952\u00b7\u0930\u094b\u00b7\u0939\u093f\u0951\u00b7\u0924\u0902 : 4\n\u092f\u0952\u091c\u094d\u091e\u0938\u094d\u092f\u0951 : \u092f\u0952\u00b7\u091c\u094d\u091e\u00b7\u0938\u094d\u092f\u0951 : 3\n\u0926\u0947\u0952\u0935\u092e\u0943\u0952\u0924\u094d\u0935\u093f\u091c\u0951\u092e\u094d : \u0926\u0947\u0952\u00b7\u0935\u00b7\u092e\u0943\u0952\u00b7\u0924\u094d\u0935\u093f\u00b7\u091c\u0951\u00b7\u092e\u094d : 6\n\u0939\u094b\u0924\u093e\u0951\u0930\u0902 : \u0939\u094b\u00b7\u0924\u093e\u0951\u00b7\u0930\u0902 : 3\n\u0930\u0924\u094d\u0928\u0952\u0927\u093e\u0924\u0951\u092e\u092e\u094d : \u0930\u00b7\u0924\u094d\u0928\u0952\u00b7\u0927\u093e\u00b7\u0924\u0951\u00b7\u092e\u00b7\u092e\u094d : 6\n<\/code><\/pre><\/div>\n\n
          <\/span>from<\/span> <\/span>IPython.display<\/span> <\/span>import<\/span> HTML<\/span>,<\/span> display<\/span>\nimport<\/span> <\/span>tabulate<\/span>\ndisplay<\/span>(<\/span>HTML<\/span>(<\/span>tabulate<\/span>.<\/span>tabulate<\/span>(<\/span>results<\/span>,<\/span> tablefmt<\/span>=<\/span>'html'<\/span>)))<\/span>\n<\/code><\/pre><\/div>\n\n\n\n
          \u0905\u0952\u0917\u094d\u0928\u093f\u092e\u0940\u0951\u0933\u0947 <\/td>[‘\u0905\u0952’, ‘\u0917\u094d\u0928\u093f’, ‘\u092e\u0940\u0951’, ‘\u0933\u0947’]         <\/td><\/tr>\n
          \u092a\u0941\u0952\u0930\u094b\u0939\u093f\u0951\u0924\u0902  <\/td>[‘\u092a\u0941\u0952’, ‘\u0930\u094b’, ‘\u0939\u093f\u0951’, ‘\u0924\u0902’]          <\/td><\/tr>\n
          \u092f\u0952\u091c\u094d\u091e\u0938\u094d\u092f\u0951   <\/td>[‘\u092f\u0952’, ‘\u091c\u094d\u091e’, ‘\u0938\u094d\u092f\u0951’]               <\/td><\/tr>\n
          \u0926\u0947\u0952\u0935\u092e\u0943\u0952\u0924\u094d\u0935\u093f\u091c\u0951\u092e\u094d<\/td>[‘\u0926\u0947\u0952’, ‘\u0935’, ‘\u092e\u0943\u0952’, ‘\u0924\u094d\u0935\u093f’, ‘\u091c\u0951’, ‘\u092e\u094d’]<\/td><\/tr>\n
          \u0939\u094b\u0924\u093e\u0951\u0930\u0902   <\/td>[‘\u0939\u094b’, ‘\u0924\u093e\u0951’, ‘\u0930\u0902’]               <\/td><\/tr>\n
          \u0930\u0924\u094d\u0928\u0952\u0927\u093e\u0924\u0951\u092e\u092e\u094d<\/td>[‘\u0930’, ‘\u0924\u094d\u0928\u0952’, ‘\u0927\u093e’, ‘\u0924\u0951’, ‘\u092e’, ‘\u092e\u094d’]<\/td><\/tr>\n<\/tbody>\n<\/table>\n\n
          <\/span>stats<\/span> =<\/span> getSyllableStats<\/span>(<\/span>text<\/span>)<\/span>\nfor<\/span> k<\/span>,<\/span> v<\/span> in<\/span> stats<\/span>.<\/span>items<\/span>():<\/span>\n    print<\/span> (<\/span>k<\/span>)<\/span>\n    for<\/span> x<\/span>,<\/span> y<\/span> in<\/span> v<\/span>.<\/span>items<\/span>():<\/span>\n        print<\/span> (<\/span>"<\/span>\\t<\/span>"<\/span>,<\/span> x<\/span>,<\/span> ":"<\/span>,<\/span> y<\/span>)<\/span>\n    print<\/span> (<\/span>"total"<\/span>,<\/span> k<\/span>,<\/span> "="<\/span>,<\/span> sum<\/span>(<\/span>v<\/span>.<\/span>values<\/span>()))<\/span>\n    print<\/span> (<\/span>"unique"<\/span>,<\/span> k<\/span>,<\/span> "="<\/span>,<\/span> len<\/span>(<\/span>v<\/span>),<\/span> end<\/span>=<\/span>'<\/span>\\n\\n<\/span>'<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>words\n     \u0905\u0952\u0917\u094d\u0928\u093f\u092e\u0940\u0951\u0933\u0947 : 1\n     \u092a\u0941\u0952\u0930\u094b\u0939\u093f\u0951\u0924\u0902 : 1\n     \u092f\u0952\u091c\u094d\u091e\u0938\u094d\u092f\u0951 : 1\n     \u0926\u0947\u0952\u0935\u092e\u0943\u0952\u0924\u094d\u0935\u093f\u091c\u0951\u092e\u094d : 1\n     \u0939\u094b\u0924\u093e\u0951\u0930\u0902 : 1\n     \u0930\u0924\u094d\u0928\u0952\u0927\u093e\u0924\u0951\u092e\u092e\u094d : 1\ntotal words = 6\nunique words = 6\n\nsyllables\n     \u0905\u0952 : 1\n     \u0917\u094d\u0928\u093f : 1\n     \u092e\u0940\u0951 : 1\n     \u0933\u0947 : 1\n     \u092a\u0941\u0952 : 1\n     \u0930\u094b : 1\n     \u0939\u093f\u0951 : 1\n     \u0924\u0902 : 1\n     \u092f\u0952 : 1\n     \u091c\u094d\u091e : 1\n     \u0938\u094d\u092f\u0951 : 1\n     \u0926\u0947\u0952 : 1\n     \u0935 : 1\n     \u092e\u0943\u0952 : 1\n     \u0924\u094d\u0935\u093f : 1\n     \u091c\u0951 : 1\n     \u092e\u094d : 2\n     \u0939\u094b : 1\n     \u0924\u093e\u0951 : 1\n     \u0930\u0902 : 1\n     \u0930 : 1\n     \u0924\u094d\u0928\u0952 : 1\n     \u0927\u093e : 1\n     \u0924\u0951 : 1\n     \u092e : 1\ntotal syllables = 26\nunique syllables = 25\n<\/code><\/pre><\/div>\n\n
          Some text in Hindi<\/p>\n
          <\/span>text<\/span> =<\/span> "\u0909\u0928\u094d\u0939\u0947\u0902 \u092c\u0941\u0926\u094d\u0927\u093f \u0914\u0930 \u0905\u0928\u094d\u0924\u0930\u093e\u0924\u094d\u092e\u093e \u0915\u0940 \u0926\u0947\u0928 \u092a\u094d\u0930\u093e\u092a\u094d\u0924 \u0939\u0948 \u0914\u0930 \u092a\u0930\u0938\u094d\u092a\u0930 \u0909\u0928\u094d\u0939\u0947\u0902 \u092d\u093e\u0908\u091a\u093e\u0930\u0947 \u0915\u0947 \u092d\u093e\u0935 \u0938\u0947 \u092c\u0930\u094d\u0924\u093e\u0935 \u0915\u0930\u0928\u093e \u091a\u093e\u0939\u093f\u090f"<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>results<\/span> =<\/span> getSyllables<\/span>(<\/span>text<\/span>)<\/span>\n\nfor<\/span> x<\/span> in<\/span> results<\/span>:<\/span>\n    print<\/span> (<\/span>x<\/span>[<\/span>0<\/span>],<\/span> ":"<\/span>,<\/span> '<\/span>\\u00B7<\/span>'<\/span>.<\/span>join<\/span>(<\/span>x<\/span>[<\/span>1<\/span>]),<\/span> ":"<\/span>,<\/span> len<\/span>(<\/span>x<\/span>[<\/span>1<\/span>]))<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>\u0909\u0928\u094d\u0939\u0947\u0902 : \u0909\u00b7\u0928\u094d\u0939\u0947\u0902 : 2\n\u092c\u0941\u0926\u094d\u0927\u093f : \u092c\u0941\u00b7\u0926\u094d\u0927\u093f : 2\n\u0914\u0930 : \u0914\u00b7\u0930 : 2\n\u0905\u0928\u094d\u0924\u0930\u093e\u0924\u094d\u092e\u093e : \u0905\u00b7\u0928\u094d\u0924\u00b7\u0930\u093e\u00b7\u0924\u094d\u092e\u093e : 4\n\u0915\u0940 : \u0915\u0940 : 1\n\u0926\u0947\u0928 : \u0926\u0947\u00b7\u0928 : 2\n\u092a\u094d\u0930\u093e\u092a\u094d\u0924 : \u092a\u094d\u0930\u093e\u00b7\u092a\u094d\u0924 : 2\n\u0939\u0948 : \u0939\u0948 : 1\n\u0914\u0930 : \u0914\u00b7\u0930 : 2\n\u092a\u0930\u0938\u094d\u092a\u0930 : \u092a\u00b7\u0930\u00b7\u0938\u094d\u092a\u00b7\u0930 : 4\n\u0909\u0928\u094d\u0939\u0947\u0902 : \u0909\u00b7\u0928\u094d\u0939\u0947\u0902 : 2\n\u092d\u093e\u0908\u091a\u093e\u0930\u0947 : \u092d\u093e\u00b7\u0908\u00b7\u091a\u093e\u00b7\u0930\u0947 : 4\n\u0915\u0947 : \u0915\u0947 : 1\n\u092d\u093e\u0935 : \u092d\u093e\u00b7\u0935 : 2\n\u0938\u0947 : \u0938\u0947 : 1\n\u092c\u0930\u094d\u0924\u093e\u0935 : \u092c\u00b7\u0930\u094d\u0924\u093e\u00b7\u0935 : 3\n\u0915\u0930\u0928\u093e : \u0915\u00b7\u0930\u00b7\u0928\u093e : 3\n\u091a\u093e\u0939\u093f\u090f : \u091a\u093e\u00b7\u0939\u093f\u00b7\u090f : 3\n<\/code><\/pre><\/div>\n\n
          <\/span>stats<\/span> =<\/span> getSyllableStats<\/span>(<\/span>text<\/span>)<\/span>\nfor<\/span> k<\/span>,<\/span> v<\/span> in<\/span> stats<\/span>.<\/span>items<\/span>():<\/span>\n    print<\/span> (<\/span>k<\/span>)<\/span>\n    for<\/span> x<\/span>,<\/span> y<\/span> in<\/span> v<\/span>.<\/span>items<\/span>():<\/span>\n        print<\/span> (<\/span>"<\/span>\\t<\/span>"<\/span>,<\/span> x<\/span>,<\/span> ":"<\/span>,<\/span> y<\/span>)<\/span>\n    print<\/span> (<\/span>"total"<\/span>,<\/span> k<\/span>,<\/span> "="<\/span>,<\/span> sum<\/span>(<\/span>v<\/span>.<\/span>values<\/span>()))<\/span>\n    print<\/span> (<\/span>"unique"<\/span>,<\/span> k<\/span>,<\/span> "="<\/span>,<\/span> len<\/span>(<\/span>v<\/span>),<\/span> end<\/span>=<\/span>'<\/span>\\n\\n<\/span>'<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>words\n     \u0909\u0928\u094d\u0939\u0947\u0902 : 2\n     \u092c\u0941\u0926\u094d\u0927\u093f : 1\n     \u0914\u0930 : 2\n     \u0905\u0928\u094d\u0924\u0930\u093e\u0924\u094d\u092e\u093e : 1\n     \u0915\u0940 : 1\n     \u0926\u0947\u0928 : 1\n     \u092a\u094d\u0930\u093e\u092a\u094d\u0924 : 1\n     \u0939\u0948 : 1\n     \u092a\u0930\u0938\u094d\u092a\u0930 : 1\n     \u092d\u093e\u0908\u091a\u093e\u0930\u0947 : 1\n     \u0915\u0947 : 1\n     \u092d\u093e\u0935 : 1\n     \u0938\u0947 : 1\n     \u092c\u0930\u094d\u0924\u093e\u0935 : 1\n     \u0915\u0930\u0928\u093e : 1\n     \u091a\u093e\u0939\u093f\u090f : 1\ntotal words = 18\nunique words = 16\n\nsyllables\n     \u0909 : 2\n     \u0928\u094d\u0939\u0947\u0902 : 2\n     \u092c\u0941 : 1\n     \u0926\u094d\u0927\u093f : 1\n     \u0914 : 2\n     \u0930 : 5\n     \u0905 : 1\n     \u0928\u094d\u0924 : 1\n     \u0930\u093e : 1\n     \u0924\u094d\u092e\u093e : 1\n     \u0915\u0940 : 1\n     \u0926\u0947 : 1\n     \u0928 : 1\n     \u092a\u094d\u0930\u093e : 1\n     \u092a\u094d\u0924 : 1\n     \u0939\u0948 : 1\n     \u092a : 1\n     \u0938\u094d\u092a : 1\n     \u092d\u093e : 2\n     \u0908 : 1\n     \u091a\u093e : 2\n     \u0930\u0947 : 1\n     \u0915\u0947 : 1\n     \u0935 : 2\n     \u0938\u0947 : 1\n     \u092c : 1\n     \u0930\u094d\u0924\u093e : 1\n     \u0915 : 1\n     \u0928\u093e : 1\n     \u0939\u093f : 1\n     \u090f : 1\ntotal syllables = 41\nunique syllables = 31\n<\/code><\/pre><\/div>\n\n
          Edge cases<\/h3>\n
          The presence of ZWJ<\/span> or ZWNJ<\/span> after virama might suggest the termination of a grapheme cluster.<\/p>\n
          <\/span>text<\/span> =<\/span> '\u0915\u094d\u0915 \u0915\u094d\u200d\u0915 \u0915\u094d\u200c\u0915'<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>syllabify<\/span>(<\/span>text<\/span>)<\/span>\n<\/code><\/pre><\/div>\n\n
          <\/span>['\u0915\u094d\u0915', '\u0915\u094d\\u200d', '\u0915', '\u0915\u094d\\u200c', '\u0915']\n<\/code><\/pre><\/div>\n\n
          Next steps<\/h2>\n
          The issue with the above approach is that it requires manual definition of character classes. This is suitable for a single script. But, if a use case requires analysis of, say, all twenty-two official languages of India, then there will be a need to handle not just Devanagari, but up to ten additional scripts. The upside is that these additional scripts are based upon the same orthographic structure as Devanagari, but the downside is that each script has its own features. In an upcoming post, I will show how to derive character classes programmatically from the Unicode Character Database.<\/p>","category":{"@attributes":{"term":"articles"}}}]}