How to fix the certificate displaying question marks '???' instead of characters

How to fix the certificate displaying question marks '???' instead of characters

by Jack Wright -
Number of replies: 43

Before I begin, this is for the newer custom_certificate plugin and not the old certificate plugin.

This post is for those that have downloaded their certificate only to find something like this:

https://moodle.org/mod/forum/discuss.php?d=361331

Why is this happening?

The custom certificate plugin uses the TCPDF PHP library to create PDF's, and the TCPDF library has some free to use default fonts that can be provided to custom certificate. This library is great with english characters, but isn't so great with most non-english characters. The issue arises when you use unrecognised characters in your certificate.

To fix the issue, you need to upload a custom font to the library directory. The custom certificate plugin isn't at fault here. How to do this is explained below:

How to fix

Download a font that you are licensed to use. The file downloaded will usually be a TTF file.

Then, find a tool online that can convert that TTF file into the three files TCPDF uses (usually a PHP file, .z and ctg.z). I used this website: http://fonts.snm-portal.com/ to do the conversion.

When you have all of them, upload them to the moodle directory under lib/tcpdf/fonts and then you should see your new font appear in the custom certificate plugin on next refresh!

It really is that easy. smile


Average of ratings: Useful (6)
In reply to Jack Wright

Re: How to fix the certificate displaying question marks '???' instead of characters

by Mark Nelson -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers
HI Jack, very useful, thanks for that. This is a very common question so I am pinning this thread so others can see it for future sake. Cheers!
Average of ratings: -
In reply to Mark Nelson

Re: How to fix the certificate displaying question marks '???' instead of characters

by Jack Wright -
Thank you! That's great to see! smile
Average of ratings: -
In reply to Jack Wright

Re: How to fix the certificate displaying question marks '???' instead of characters

by Aaron Johnson -
Hi,
I was using this solution when we hosted our own site, but after moving to a hosting service, I no longer have access to the tcpdf/fonts directory, and my hosting service doesn't want to add the font files there for update maintenance reasons. Any ideas how to make other TCPDF fonts available without access to the moodle directory? I see several references to using PDF_CUSTOM_FONT_PATH, but not an explanation on how to actually implement that.
Thanks
Average of ratings: -
In reply to Aaron Johnson

Re: How to fix the certificate displaying question marks '???' instead of characters

by Mark Nelson -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers
The constant PDF_CUSTOM_FONT_PATH points to "$CFG->dataroot.'/fonts/'" which means it is pointing to a folder called 'fonts' in your Moodle data root. You can also override this if you want, see https://github.com/moodle/moodle/blob/master/lib/pdflib.php#L68.
Average of ratings: -
In reply to Mark Nelson

Re: How to fix the certificate displaying question marks '???' instead of characters

by Aaron Johnson -

Thanks, Mark. Is it possible to override this somewhere other than in the core Moodle code? My hosting service is hesitant to make such changes saying it makes Moodle updates more time consuming and expensive. If this variable were set in my moodle/config.php file (which has to be copied as part of an update anyway), would that be enough? Before setting PDF_CUSTOM_FONT_PATH, pdflib.php is checking that it isn't already defined, which makes me think this could/should work.

Average of ratings: -
In reply to Aaron Johnson

Re: How to fix the certificate displaying question marks '???' instead of characters

by Mark Nelson -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers
Yep, that should do the trick Aaron.
Average of ratings: -
In reply to Aaron Johnson

Re: How to fix the certificate displaying question marks '???' instead of characters

by Mark Nelson -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers
However, you will have to put the new fonts somewhere so why not the default location?
Average of ratings: Useful (1)
In reply to Aaron Johnson

Re: How to fix the certificate displaying question marks '???' instead of characters

by Mark Nelson -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers
The dataroot is not the same as the webroot, so it wont affect your hosts when they do upgrades. The dataroot is your moodledata directory.
Average of ratings: Useful (1)
In reply to Mark Nelson

Re: How to fix the certificate displaying question marks '???' instead of characters

by Aaron Johnson -
Thanks, Mark. That's a good point. My hosting company is being pretty inflexible on this, plus slow to develop their own solutions. Hopefully this gives me enough info/options to give them something they'll do.
Average of ratings: -
In reply to Mark Nelson

Re: How to fix the certificate displaying question marks '???' instead of characters

by Aaron Johnson -
Hi Mark,
It finally just clicked for me. I was making it way too complicated. Correct me if I'm wrong, but the point here is not that you have to change the PDF_CUSTOM_FONT_PATH, you just have to use it. By default, this folder ("moodledata/fonts") does not exist, Moodle sees that and instead uses the standard "moodle/lib/tcpdf/fonts".

To use custom fonts without changing the webroot:
  1. Copy the folder moodle/lib/tcpdf/fonts from your webroot to your dataroot (i.e. the new folder is probably moodledata/fonts).
    1. If for whatever reason you don't want to use "moodledata/fonts", set the desired path for PDF_CUSTOM_FONT_PATH in your moodle/config.php file.
  2. Add the new fonts to the dataroot folder according to the instructions from Jack Wright above.
  3. That's it. The new fonts should now be available in the font selector when editing text of a Custom Certificate.
This might be a better option for many people, as dataroot is not overwritten during a Moodle update, and so you shouldn't ever have to reinstall the additional fonts. Let me know if I've missed anything.

Thanks,
Aaron
Average of ratings: -
In reply to Aaron Johnson

Re: How to fix the certificate displaying question marks '???' instead of characters

by Mark Nelson -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers
Yes Aaron, that sounds correct! smile
Average of ratings: Useful (1)
In reply to Jack Wright

Re: How to fix the certificate displaying question marks '???' instead of characters

by Hanan Alsahafi -
I am using this solution. The problem now some Arabic letter printing correctly but some of them printing space or blank. I do not know where is the probelm
Average of ratings: -
In reply to Hanan Alsahafi

Re: How to fix the certificate displaying question marks '???' instead of characters

by Aaron Johnson -
Hi Hanan, have you double checked that the characters that are having problems are in the font you are using? If some characters are working, but others aren't, I would suspect the font set is incomplete.
Average of ratings: -
In reply to Aaron Johnson

رد: Re: How to fix the certificate displaying question marks '???' instead of characters

by Hanan Alsahafi -
Which file I can edit so the letter be printing correctly? a PHP file, .z or ctg.z
Average of ratings: -
In reply to Hanan Alsahafi

Re: رد: Re: How to fix the certificate displaying question marks '???' instead of characters

by Aaron Johnson -
Hi Hanan,

I don't know how, or even if, those files can be edited. You need to find the TTF file for a font type that includes the characters you need. Perhaps something from here: https://arabicfonts.net/. Thatsite allows you to type in test text, so you can ensure that the problematic characters are included in the font.

Then follow the instructions above to create the three files needed for Moodle from that TTF file and add them to your Moodle site.

Hope that helps,
Aaron
Average of ratings: -
In reply to Aaron Johnson

رد: Re: رد: Re: How to fix the certificate displaying question marks '???' instead of characters

by Hanan Alsahafi -
The TTF alone looks very good with all character. I used in Microsoft world without facing ANY problem only when I converted to tcpdf
Average of ratings: -
In reply to Aaron Johnson

رد: Re: رد: Re: How to fix the certificate displaying question marks '???' instead of characters

by Hanan Alsahafi -
This is not working website so I can do the conversion http://fonts.snm-portal.com/
Average of ratings: Useful (1)
In reply to Hanan Alsahafi

Re: رد: Re: رد: Re: How to fix the certificate displaying question marks '???' instead of characters

by Aaron Johnson -
Yep, that online conversion tool doesn't seem to be working anymore. I've been searching for another site that does the same and am coming up empty. That's going to make this harder than I know how to deal with. Sorry. I'm all out of ideas.
Average of ratings: -
In reply to Jack Wright

Re: How to fix the certificate displaying question marks '???' instead of characters

by James Peter -
Hi
what is the alternative for http://fonts.snm-portal.com/ as it is down for many days!
I found https://www.xml-convert.com/en/convert-tff-font-to-afm-pfa-fpdf-tcpdf but its is not generating .ctg.z files so the pdf get corrupted

thank you
Average of ratings: -
In reply to James Peter

Re: How to fix the certificate displaying question marks '???' instead of characters

by Aaron Johnson -
It seems that online tool is really gone. Have you tried following the instructions for doing the conversion locally? See https://docs.moodle.org/311/en/Add_fonts_for_embedding#Converting_locally
I tried this method and it seems to work. Let me know if you have trouble with it.
Average of ratings: -
In reply to Aaron Johnson

Re: How to fix the certificate displaying question marks '???' instead of characters

by James Peter -
Thank you Aaron
i tried that , but failed to execute
i created a php file and copied the below code in to that with <?php opening and closing tag and placed it in /lib/tcpdf/fonts
code : include('../tcpdf.php'); $pdf = new TCPDF_FONTS('P', 'mm', 'A4', true, 'UTF-8', false); $pdf->addTTFfont('./MyFont.ttf', 'TrueTypeUnicode');

Coplied the TTF font in to /lib/tcpdf/fonts folder and changed the MyFont.ttf name on the above php code to the real font name example tahoma.ttf

Run the file though web url
As mentioned its showing blank but no fonts generated in /lib/tcpdf 
The file permission for the php file is 644
Am doing any thing wrong?

I am using tcpdf Version : 6.3.2 which is includedin the moodle code


Please advice

Thanks a lot
Average of ratings: -
In reply to Aaron Johnson

Re: How to fix the certificate displaying question marks '???' instead of characters

by James Peter -
Update...
Thanks a lot
It was generated in font folder not /lib/tcpdf as mentioned in the document

Average of ratings: -
In reply to James Peter

Re: How to fix the certificate displaying question marks '???' instead of characters

by Aaron Johnson -
Glad it worked. I've submitted an edit to the documentation to show the correct output folder. Also note that Moodle doesn't use php closing tags. So you need "<?php" at the start of your file, but nothing at the end.
Average of ratings: -
In reply to Aaron Johnson

Re: How to fix the certificate displaying question marks '???' instead of characters

by James Peter -
Thanks again Mr. Aaron
Hmmm is there any easiest method to convert fronts by batch
Average of ratings: -
In reply to James Peter

Re: How to fix the certificate displaying question marks '???' instead of characters

by Aaron Johnson -
Not that I'm aware of.
Average of ratings: -
In reply to Jack Wright

Re: How to fix the certificate displaying question marks '???' instead of characters

by Khurram Jutt Official -

very well explained i am facing the same issue from last few days when i am trying to show my product vegidrill

thanks soo much .

Average of ratings: -
In reply to Jack Wright

Re: How to fix the certificate displaying question marks '???' instead of characters

by Jaifar Al Shizawi -
Thank you for your help,

I tried for major of Arabic language fonts, and got same issue at the begging, but after researching, i noticed that I have to search for our language font that support "Unicode" like Amiri font type in this site https://www.easyarabictyping.com/arabic/fonts

also I did it by generating the other three types (.z, .php, .czn.z) adding new php file in /lib/tcpdf/fonts/convertfonts.php

and added the following script

<?php include('../tcpdf.php'); $pdf = new TCPDF_FONTS('P', 'mm', 'A4', true, 'UTF-8', false); $pdf->addTTFfont('./Amiri-Bold.ttf', 'TrueTypeUnicode', '',96); 

Then run it using https://mymoodlesite.fo/lib/tcpdf/fonts/convertfonts.php

Average of ratings: Useful (1)
In reply to Jaifar Al Shizawi

Re: How to fix the certificate displaying question marks '???' instead of characters

by Stefano Krister -
This solution and the script helped me with my custom fonts.
A big thank you!

I tried using:
https://www.xml-convert.com/en/convert-tff-font-to-afm-pfa-fpdf-tcpdf
http://fonts.palettize.me
http://fonts.snm-portal.com

The first link didn't produce a .ctg.z-file which garbled the final .pdf.
The second link produced all required files (with an error) but it failed to display special characters.
The third one is just dead (404).

I guess the documentation here needs to be fixed:
https://docs.moodle.org/311/en/Add_fonts_for_embedding

Local conversion seems to be the best way.
Average of ratings: -
In reply to Stefano Krister

Ri: Re: How to fix the certificate displaying question marks '???' instead of characters

by Angelica Lagorara -
Hello everyone!
I've tried the font local convertion but it didn't seem working.

After i've run https://myhostname/moodle30/lib/tcpdf/fonts/convertfont.php with the browser FIREFOX (with google chrome seem not working) and it works!
Average of ratings: -
In reply to Jack Wright

Re: How to fix the certificate displaying question marks '???' instead of characters

by Stan Klapauszak -
Hi Jack - I am trying to display certificates in several different languages requiring different fonts, starting with Burmese. I have done exactly what you suggested - found a Burmese TTF font, converted to TCPDF and uploaded into the correct Moodle directory. It shows up as an available font in the Custom Certificate library, and when I insert a Burmese string into the certificate and look at it via "Reposition elements" it looks just fine:
Burmese text in Custom Certificate

However, when I look at this same certificate via "View Certificate" all the Burmese characters are replaced with question marks:
Burmese text replaced with question marks

How can I fix that?

Average of ratings: -
In reply to Stan Klapauszak

Re: How to fix the certificate displaying question marks '???' instead of characters

by Aaron Johnson -
Hi, what method did you use for generating the three font files? What font are you using? Maybe I can try it in my system.
Average of ratings: -
In reply to Aaron Johnson

Re: How to fix the certificate displaying question marks '???' instead of characters

by Stan Klapauszak -
Hi Aaron - I used the Noto Serif Myanmar - Medium TTF font and an online TTF=> TCPDF converter. I forget which converter but it may have been http://fonts.palettize.me/ (or perhaps https://www.xml-convert.com/en/convert-tff-font-to-afm-pfa-fpdf-tcpdf).

Stan
Average of ratings: Useful (1)
In reply to Stan Klapauszak

Re: How to fix the certificate displaying question marks '???' instead of characters

by Aaron Johnson -
Hi Stan, Thanks for the link to http://fonts.palettize.me/. That appears to be working and could replace the previous site we linked to that has since stopped. I just happened to be setting up a new 3.11 test site, so I tried your font there. I get the same behavior as you on a fresh install. I checked adding a couple other optional fonts, and those all worked using the files generated from that website. I'm all out of ideas at this point. Sorry.

Mark, do you have any ideas?
Average of ratings: -
In reply to Aaron Johnson

Re: How to fix the certificate displaying question marks '???' instead of characters

by Stan Klapauszak -
Thanks Aaron. If it works with some fonts and not others, I'll experiment with different Burmese fonts to see if I can find one that works.

What could cause the certificate to display correctly in the "reposition elements" view but not in the "view certificate" view? Are there different viewers being used for those two functions, that may be drawing on different font components?

Also, is there any chance this has something to do with character encoding? I kind of doubt it, because that text string displays correctly in several different applications (email, Google Translate, the "Reposition Elements" view of the Custom Certificate). I only ask because we've had prior issues with Burmese text encoding - apparently the local standard is Zawgyi encoding, whereas Moodle uses Unicode.
Average of ratings: -
In reply to Stan Klapauszak

Re: How to fix the certificate displaying question marks '???' instead of characters

by Aaron Johnson -
Hi Stan, I'd try other fonts. I used Arial UNICODE MS and go a bunch of little boxes instead of question marks, so that's something. I too have noticed differences between the positioning page and the actual PDF, so maybe it is just a question of finding the right font.
Average of ratings: -
In reply to Aaron Johnson

Re: How to fix the certificate displaying question marks '???' instead of characters

by Stan Klapauszak -
Hi Aaron - I tried several other Burmese fonts but got exactly the same problem with each of them - the "Reposition Elements" display looks perfect (and I can see the differences in the fonts, so am confident that they are actually using the different fonts) but the "View Certificate" display just shows ???s. It sure would be helpful to understand the difference between those two forms of display - there must be different processing behind them.

You said you tried a few optional fonts and they worked - were any of those non-Latin characters? If so, which fonts were they?
Average of ratings: -
In reply to Stan Klapauszak

Re: How to fix the certificate displaying question marks '???' instead of characters

by Mark Nelson -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers
When you use the 'Reposition elements' page it uses HTML to generate the font, not the TCPDF library. The issue is the TCPDF library (which generates PDFs) and the font you have selected do not support all characters. You want a UTF-8 font. The only thing to do is trial and error until you find one that works.
Average of ratings: Useful (2)
In reply to Stan Klapauszak

Re: How to fix the certificate displaying question marks '???' instead of characters

by Aaron Johnson -
Hi Stan, I meant that in general, adding additional fonts appeared to be working. I use Arial UNICODE MS to display Chinese characters (bottom row in attached pic), but it just gives boxes for Burmese (top row in pic). Sorry. I also tried TharLon and Padauk for you, but they didn't work either.
Attachment PDF Font test.PNG
Average of ratings: -
In reply to Stan Klapauszak

Re: How to fix the certificate displaying question marks '???' instead of characters

by Aaron Johnson -
So I just discovered a new plugin for adding new TCPDF fonts: https://moodle.org/plugins/tool_tcpdffonts

Using this tool, I added the TharLon font and it half worked. Maybe spend some time playing with the font import settings and you can get it to work.
Attachment PDF Font test 2.PNG
Average of ratings: -
In reply to Aaron Johnson

Re: How to fix the certificate displaying question marks '???' instead of characters

by Stan Klapauszak -
Hi Aaron - that's a good find. It has the ability to upload TTF / OTF files directly rather than having to convert them to TCPDF files first, plus has the ability to preview the font directly within the tool. This is progress, although I can't declare "victory" yet:

- first of all, I replaced a couple of the fonts that I had previously installed with the same fonts by uploading the TTF files, I got different (better) results in the "preview" screen. That would suggest that the conversion tool I was using earlier (http://fonts.palettize.me/) wasn't converting correctly (?), or at least that the conversion tool in this plugin converts differently than that website.
- I uploaded one font (NotoSansMyanmar) via TTF and in the plugin's "PDF preview" tool it looks PERFECT, and I can download a PDF that looks exactly the same:
Screen print of NotoSansMyanmar preview
    However, when I use that font in my Certificate (with the exact same Burmese text) and View Certificate, I get this bizarre image:
Custom certificate using NotoSansMyanmar

So it seems I have 2 different "PDF viewer" capabilities on the same Moodle system, one of which seems to deal with that font perfectly, and the other of which fails. How does that happen?

I'll continue experimenting with other fonts, but this makes me wonder about the "View Certificate" capability of the Custom Certificate plugin...

Average of ratings: -
In reply to Stan Klapauszak

Re: How to fix the certificate displaying question marks '???' instead of characters

by Aaron Johnson -
Hi Stan, have you tried that font on a simple certificate? I just tried it on a certificate with just one sentence, and it looks fine (see attached). Could you maybe put the text here that you are trying to render for me to try? What settings did you use when importing the font? I used these:
Average of ratings: -
In reply to Aaron Johnson

Re: How to fix the certificate displaying question marks '???' instead of characters

by Stan Klapauszak -
Hi Aaron - thanks for your feedback.

I did try Noto Sans Myanmar on simple certificates, and (although it always looked just fine in the PDF preview of the TCPDF Fonts plugin) every time when viewed via View Certificate it produced the "black triangle" shape I posted earlier . I used the exact upload parameters that you have used above, and the test Burmese text that I'm using is:

ကျွန်ုပ်တို့ iTEE တက္ကသိုလ် ဆရာ/ဆရာမများမှ အောက်ပါကျောင်းသား/ကျောင်းသူအား သင်ရိုးကို အောင်မြင်စွာ ပြီးဆုံးခဲ့ခြင်းကြောင့် ဤသင်တန်းပြီးမြောက်ကြောင်း

I'd be interested if you get different results on your system - it would raise questions about why the View Certificate would work differently on different systems.

BUT - I also have good news to report. After installing / testing at least 10 different fonts, I found a couple that (at least to my non-Burmese-speaking eyes) appear correct via View Certificate, those being MMR Text and Pyidaungsu. I've sent test certificates using those fonts to my Burmese translator for verification, and am hoping that at least one is correct.

Stan
Average of ratings: -
In reply to Stan Klapauszak

Re: How to fix the certificate displaying question marks '???' instead of characters

by Aaron Johnson -
Hi Stan, I confirm your findings. Using your text above, Noto Sans Myanmar just puts a big black triangle in the upper right corner. MMR Text however renders legible text. That's weird, but I'm glad you found a font that works.
Aaron
Average of ratings: Useful (1)