Indic Scripts Including Hindi, Telugu, Tamil and many more not rendering as they supposed to be.

• [ ] make unit test for pygame to check if rendering works.
• [x] SDL_TTF is released with patch (2.0.16 should have patch included).
• [ ] expose TTF_SetDirection TTF_SetScript functions.
• [x] windows pygame build that includes harfbuzz.
• [ ] manylinux pygame build that includes harfbuzz.
• [ ] mac wheel build that includes harfbuzz.

---

When I am trying to render Hindi text using render_to(). It is rendering inappropriately. for eg. which should be written like . Also like which should be displayed like . Also there is a stackoverflow question that raise the similar issue. https://stackoverflow.com/questions/44254171/devanagari-text-rendering-improperly-in-pygame Moreover there is a bugzilla post that discuss similar issue.. Can someone fix this bug ??

Related Docs: https://www.pygame.org/docs/ref/font.html

Hi, Every font file comes with a gsub file which is a set of rules that define substituting consegutive glyphs with a joint glyph for example. Pygame ignores such gsub files while rendering font. Can anyone tell how to force pygame to use gsub rules?

When gsub rules are not used, in Devanagari "म्ह" looks like "म् ह"(ignore space) as shown in the image:-

Ah ok. So it seems the patch on the libsdl bugzilla for SDL_ttf (posted in 2015) is for using HarfBuzz. I haven't looked at the patch, but it seems getting someone to work on it in libsdl would be one way to get support there.

Has anyone been able to get this working?

I can confirm that the issue still exists in pygame-1.9.6 as of today (and that PR #706 does not actually fix this issue).

Code:

import pygame, sys
from pygame.locals import *

pygame.init()
DISPLAYSURF = pygame.display.set_mode((400, 300))
pygame.display.set_caption('Hello World!')

WHITE = (255, 255, 255)
GREEN = (0, 255, 0)
BLUE = (0, 0, 128)

text = "प्रफुल्ल"

fontObj = pygame.font.SysFont('Lohit Devanagari', 32)
textSurfaceObj = fontObj.render(text, True, GREEN, BLUE)
textRectObj = textSurfaceObj.get_rect()
textRectObj.center = (200, 150)

while True:
    DISPLAYSURF.fill(WHITE)
    DISPLAYSURF.blit(textSurfaceObj, textRectObj)
    for event in pygame.event.get():
        if event.type == QUIT:
            pygame.quit()
            sys.exit()
    pygame.display.update()

Testing Notes: 'Lohit Devanagari' font can be installed on any ubuntu-based distro by installing the package fonts-lohit-deva.

@dlon, @husano896: Esteemed developers, please pardon my pinging you, but according to the stackoverflow discussion:

I think is a SDL_ttf problem (the underlying component which actually renders the text).

Could you please look into it and make sure whether or not that is the case and let me know so that I can perhaps raise an issue over there? Please forgive me I do not have the knowledge to debug this further.

If fixed, it would be a massive help to us around here in the Indian-subcontinent. And we would be very very grateful. I hope to get your guidance in this.

Thank you for your time.

@ilario-pierbattista - Please pardon my abrupt pinging you, good sir, but because I saw this gist by you, I thought that, perhaps, you might also have something that you could bring to the discussion here. I am sorry I do not have the knowledge to understand the inner details of SDL, or to even understand the gist.

@illume @takluyver @sublimator @bitcraft @dlon - Please excuse my impatience, esteemed developers, but since this was not getting any attention, I am taking the liberty of pinging the pygame project members as listed on GitHub. I hope that perhaps some of you might be able to shed some light into this. Thank you

@peanutbutterandcrackers Please respect that people have other things going on, and this is a volunteer project. If no one is able to answer right away it's usually because they don't have time at the moment.

@illume - Yes, sir. I understand. I am sorry. I just wanted to get some information regarding whether I should be actually reporting this issue to the SDL devs instead of pygame devs, and if I could get the confirmation, I could go over there.

Hi,

No worries. I understand the feature is important for you(And others!).

The issue is reported, but really the next step is figuring out the best way to get it done.

As mentioned in an earlier thread, there is an open bug with an old patch for SDL_ttf on the SDL bug tracker. Maybe this is the best way forward... I don’t know.

For pygame we need to get a windows build, and various other builds of “harfuzz”, and then configure them for every platform. But maybe this isn’t the best library to use in 2019. We’d need feedback from developers familiar with the font rendering landscape. Like, is this library available and packaged on all platforms?

But who can do this work, or how can we get this patch into SDL_ttf? Probably a good step is seeing if that patch still applies, and then seeing if it compiles and works. This would reduce the work of the SDL people. Then asking for feedback from them on what further work the patch needs.

There’s lots of small steps in getting this into a build and working correctly left to do. We need a volunteer for this however. I can help guide someone in how to get this done, but have too many things I need to do at the moment to take on the work myself.

One small step that only requires python knowledge would be to write a test case, as a pygame unit test. This way it will be easier for pygame developers (and SDL devs) to see that it is working. Would you be interested in helping out with this? I could guide you on the way.

cheers,

On Thursday, August 29, 2019, peanutbutterandcrackers < [email protected]> wrote:

@illume https://github.com/illume - Yes, sir. I understand. I am sorry. I just wanted to get some information regarding whether I should be actually reporting this issue to the SDL devs instead of pygame devs, and if I could get the confirmation, I could go over there.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pygame/pygame/issues/464?email_source=notifications&email_token=AAACKRKDGNM2SBO3B2DGTV3QG5CYBA5CNFSM4FHTG572YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5NEY7Y#issuecomment-526011519, or mute the thread https://github.com/notifications/unsubscribe-auth/AAACKRKUZVUERYH7C3U5MU3QG5CYBANCNFSM4FHTG57Q .

@illume - Yes, sir! I would be honoured! Any little thing I could do to help, I would like to do it. However, I must admit that my knowledge regarding the matter is next-to-zero and I am not the very quickest of learners. Yet, I will try my best: even if it takes a lot of time. :)

I left a note about this issue in the stackoverflow post about this issue.

@peanutbutterandcrackers cool! Thanks. For a test we need an assertion to pass or fail if the code is correct.

Can you modify the code you posted so that it would pass/fail if the rendering is correct?

Something like this where you have to modify the is_correct code for your test:

is_correct == surf.get_at((x, y)) == (255,255,255,0)
print("is correct:%s" % is_correct)

I just checked the SDL issue, and it has been merged into SDL_ttf (but disabled by default). See https://bugzilla.libsdl.org/show_bug.cgi?id=3046

However it hasn't made it into a release of SDL_ttf yet. The last release(release-2.0.15) is 7 months ago and the patch was merged 4 months ago.

I linked this issue to the libsdl one.

Here are the new C functions exposed by SDL_ttf (from this link).

/* Set Direction and Script to be used for text shaping.
  - direction is of type hb_direction_t
  - script is of type hb_script_t
This functions returns always 0, or -1 if SDL_ttf is not compiled with HarfBuzz
*/
extern DECLSPEC int SDLCALL TTF_SetDirection(int direction); /* hb_direction_t */
extern DECLSPEC int SDLCALL TTF_SetScript(int script); /* hb_script_t */

• Mac harfbuzz exists in homebrew.
• Debian linux https://packages.debian.org/search?keywords=harfbuzz is available from oldstable (jessie).
• Windows, the harfbuzz website says it has windows binaries. https://www.freedesktop.org/wiki/Software/HarfBuzz/

I made a todo list at the top of the issue with some of the things that need to be done.

@illume - Please pardon my delayed reply, good sir. Great! I am working (right now) on something that I hope will be even more versatile (the test-script, I mean). I will probably upload it to GitHub sometime today and will post the link here. Thank you for your patience.

@illume - I have just pushed the project to GitHub. You can find it here. It is horribly ugly. But I will be re-iterating through it in the coming days. I hope the README.md file is clear-enough as to it's usage.

An Update: The test script now works with GIMP 2.8.x as well as GIMP 2.10.x. I have been updating/polishing it up sporadically. Please do check it out and let me know if there is anything I should update. It would be nice, too, if some of the reporters could add other Indic-scripts in the text.json file (Telugu, Tamil, etc.); for now it only tests Latin and Devanagari. Issues and PRs are welcome. If there is anything that is not understood, please do let me know so that I can update the README.md with further documentation/explanation.

Now we only need to wait for SDL_TTF 2.0.16 before this issue-resolution catches momentum.

P. S: Code-reviews are most welcome!

Thanks @peanutbutterandcrackers

I had a quick look at it, but haven't had a chance to play around with it yet. But it does seem like it is in a good shape to help whoever wants to work on this.

cheers!

@illume - Great!

Regarding "whoever wants to work on this": if there isn't one (and we're counting on sdl_ttf 2.16 to be released as the next step in this flow, anyways), could you recommend me some books that I could study to be able to work on this? As in, all the books/documentations/videos/lectures/etc. that an intermediate python programmer (who is unfamiliar with C) should go through in order to become competent enough to contribute in this task. And pygame in general. I would love to have a road-map that I could follow. In order words, how does one become as competent as you?

Hi @peanutbutterandcrackers,

sorry I didn't respond earlier.

I couldn't think of a good answer to your question. But here is a list of good learning resources...

• https://www.reddit.com/r/C_Programming/ (good forum of C developers)
• https://www.pygame.org/wiki/Contribute (here to get started on pygame things)
• https://www.pygame.org/wiki/Hacking (some tips for working with pygame)
• https://renesd.blogspot.com/2019/09/post-modern-c-tooling.html (some newer C based tools)
• Modern C book https://gforge.inria.fr/frs/?group_id=6881 This book apparently is good if you haven't encountered C before. I'm not sure how successful it is at it, but I hear from a few others that it's pretty good. It's also available for a free download.

Depending on what OS you are on, I could try and find other links. Feel free to ask questions, and I'll try and answer.

cheers!

@illume - Thank you! I am on Linux Mint 19, currently. I will be looking at the resources that you have provided. Thank you very much!

For Devanagari I have posted the solution in last part of Readme here (follow 7.0 and 7.1):

https://github.com/rohitsaluja22/OCR-On-the-go

SDL has moved issue trackers: and the new related bug is here: https://github.com/libsdl-org/SDL_ttf/issues/62

I think I've managed to get something working on this, thanks to SDL_ttf 2.0.18

Rendering the string "प्रफुल्ल"

pygame 2.1.3.dev4

pygame with SDL_ttf 2.0.18 and TTF_SetScript hardcoded to devanagari

Not quite sure how to expose the functionality though. It needs to boil down to these eventually: https://harfbuzz.github.io/harfbuzz-hb-common.html#hb-script-t. They all have a 4 character code that can be put into a 32 bit int. It would be a lot of constants to add to pygame for every supported script, we could have people input the 4 byte code manually.

font = pygame.font.Font("Something that supports hindi characters.ttf", 20)
font.set_script("Deva")

@Starbuck5 Can you explain a bit more on how you achieved this? "set_script" is not a method of "pygame.font.Font" is the error I get when I tried your solution.

Edit: I think your comment was more of a proposal, I realise that now. How do I hardcode TTF_SetScript to Devanagari? Appreciate any help you can offer. Thanks!

Edit: I think your comment was more of a proposal, I realise that now. How do I hardcode TTF_SetScript to Devanagari? Appreciate any help you can offer. Thanks!

Yeah, @prajwalgatti, I did a custom build of pygame with code changes to get this to work. I thought I was going to be able to hook you up with a custom build tonight, but I've run into a big issue, which I raised with the relevant authorities. Even my patch would only work on Windows, for now, we're behind on the dependencies on other platforms.

What OS and python version are you on, out of curiousity?

@prajwalgatti If you need this feature right now, I've put up a pull request implementing it (only on Windows though, because dependencies)

You can download a wheel from the artifacts tab of one of these runs: https://ci.appveyor.com/project/pygame/pygame/builds/44257337

If you're on Mac or Linux you'll have to build off my branch and update the dependencies locally to SDL_ttf 2.20.0.

@Starbuck5 Sorry I just saw your replies today. Thanks a ton for the help! :) I'm on a Linux machine and use python3.7. Will try to get my hands on a Windows machine, if not I'll do as you suggested for the Linux build.

Will update here after trying!