Hello,

We seem to have a case where some characters entered in the address form during checkout or on the clientlogin.asp page are getting misformatted. This only occurs sometimes.

There was a "’" in the company name field which was most likely ISO misinterpreting something that was saved as 0xE2 0x80 0x99 in UTF-8 format.

My question is why would some part of the code be trying to interpret something as ISO when all pages of our website have:
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
and the cart specifically has the newer version:
<meta charset="UTF-8">
and adminencoding = "utf-8"

Any idea on what may be causing this issue?

Thanks

Hi Dermontti
Is it possible that your customers may have addresses saved in their login account that were created before a switch to UTF-8 ? Could you check the account for the customer who had this issue?

Vince

Click Here for Shopping Cart Software
Click Here to sign up for our newsletter
Click Here for the latest updater

Hello Vince,

Yes, I believe both customer had their accounts prior. The one this morning for sure has been with us for years and had the name something like:
John#65533;s Creations ABC, Inc.

We are also on SQL Server 19 with this collation: SQL_Latin1_General_CP1_CI_AS
I tried putting in illegal characters such as : ’ ‘ ” “ ™ € • – — ° and they were fine except on the email screen. where they turned into: €™ ‘ â€� “ â„¢ € • – — °
When I set email encoding specifically to ISO, I got Company Name: macâ?Tguire Company Name: macâ?Tguire'â?T â?~ â?� â?o â"¢ â,¬ â?¢ â?" â?" ° which suggests the data is being properly saved into the database as UTF-8 and somewhere in the output for the email it is changing.

Other than the email screen, all new orders seem fine in admin and I COULD NOT reproduce the bad characters through any of the other customer inputs.

Any idea why the email had the bad characters? I inspected the code and it looks like it inherits admin encoding if left blank. I tested and Persits ASP and CDO both encountered the issue.
Is there a way to handle legacy data that may have been ISO encoded?

Hi, check to see if you have this set in includes
emailencoding="UTF-8"

Thanks,
David

Hello Dave,

Thanks, but we do have that set per my first post.

It seems to be probably:
1. legacy data like Vince mentioned
2. output for the email as the characters are UTF8 converted bad into the old ISO

Weird - I don't see emailencoding="UTF-8" in your first post at all.

David

Hello Dave,

Sorry, I misread.

I tried:
1. nothing (inheritance)
2. explicit utf-8
3. explicit ISO
4. both with persits and CDO
5. setting the below to UTF-8 also for persits email body:
if emailencoding<> "iso-8859-1" then
EmailObj.ContentTransferEncoding="Quoted-Printable"
end if

Hi Dermontti

quote:

---

Yes, I believe both customer had their accounts prior. The one this morning for sure has been with us for years and had the name something like:
John#65533;s Creations ABC, Inc.

---

If these characters were entered into the database using a different encoding then yes, they will show up as garbled characters using the UTF-8 encoding. You really need to find these and replace them.

Vince

Click Here for Shopping Cart Software
Click Here to sign up for our newsletter
Click Here for the latest updater

Hello Vince,

ECT had to change over from ISO to UTF at some point. The ASP code supports both.

I think any customer that changes over would have this issue.

Also, any idea on what would cause the output issue only for emails?

Thanks

Hi Dermontti
We've always supported both UTF-8 and ISO and other non-western encodings. As long as you are consistent with the use in the front end and back end everything will be fine. But yes, if you have data saved in one encoding and switch to another that will cause problems.

Vince

Click Here for Shopping Cart Software
Click Here to sign up for our newsletter
Click Here for the latest updater

Hello Vince,

Thank you for the response, however, we have been using your platform for about 20 years and started with ISO.

We just did a huge upgrade to version 7.6.4 which was UTF-8 by default as it should be because the net standard is now too and has been for many years.

I cannot imagine an e-commerce platform not having a way to transition over and also not have some type of form validation or sanitation to prevent illegal characters from being entered.

It is mostly an issue with international orders, but some people have the old ISO ' which can cause issues.

1. Is there no support for a switchover?
2. Any ideas on what the email issue could be?

Thanks in advance.

Hi Dermontti
The updaters haven't changed with regards to UTF-8 or iso-8859-1 and you can still use iso-8859-1 no problem.
I'm not sure to what extent it's for the cart to provide ways of converting between encodings, but as it's an open system you can use any solution and a quick search on Google should give you a few ideas...
https://www.google.com/search?client=firefox-b-d&q=sql server convert iso-8859-1 to utf-8

Vince


Click Here for Shopping Cart Software
Click Here to sign up for our newsletter
Click Here for the latest updater

Hello Vince,

Thank you for the response, but could you please shed light on what the issue with the email may be?

Also, do you provide any form validation that is editable? such as a regex for ASCII only for form input?

Hi Dermontti

quote:

---

but could you please shed light on what the issue with the email may be?

---

But is the issue with the email that if you have a customer account with address data that was saved in ISO format then that is malformed in the email? If that's the case, is the solution to convert the ISO data to UTF-8?

Vince

Click Here for Shopping Cart Software
Click Here to sign up for our newsletter
Click Here for the latest updater

Hello Vince,

Thank you for the response.

Somewhere in the code is still using ISO with the UTF-8.

I input drew péüåñ in UTF-8 and it works in admin, user login, cart and thank you page, but the email to us and the customer shows mojibake. The below output from the email is what happens in ISO when those 4 characters were created in UTF-8 format.

In other words, somewhere is still trying to interpret them as ISO. I tried tracing it down, but was unable to find it.

Order ID: 1361006
Order Date: 6/21/2024
Company Name: mac'guire Company Name: mac'guire
Billing Address: drew péüåñ
1234 street
toronto, Ontario
M5H2N2
Canada
Shipping Address: drew péüåñ
1234 street
toronto, Ontario
M5H2N2
Canada

Hi Dermontti

Could you add these two lines to your includes.asp file and that should hopefully sort things out for you...

Response.CodePage = 65001
Response.CharSet = "utf-8"

Vince

Click Here for Shopping Cart Software
Click Here to sign up for our newsletter
Click Here for the latest updater

Thanks Vince.

That seems to have solved the issue with the emails. Now I just need to find something to do about old data.

I now have this at the top of the includes:
' HTML options ----------------------------------------
Response.CodePage = 65001
Response.CharSet = "utf-8"
adminencoding = "utf-8"
emailencoding = "utf-8"

I had tried putting the response on thanks/email etc., but hadn't tried that recently with the other variables also set.


I traced the email body text all the way through the incemail/incthanks and it was not manipulated.

I also hardcoded a UTF8 text body into the Persits type 2 email function and only got the mojibake there.

Perhaps the issue was the overall encoding for incfunctions was interfering with the specific email function.

I will perform more thorough testing on our dev site and report back the final results.

Thanks again :)