A place where we write words Ignition Blog

Welcome to the Ignition Development blog, where we talk about a wide range of technical and non-technical topics.

PayPal Instant Payment Notification and UTF-8 encoding

Posted on Friday, April 9, 2010 10:06 PM

I spent almost an entire day last week fighting with UTF-8 encoding and PayPal IPN integration with the Tentacle Software store.

From a development perspective, I really like IPN. It lets your order processing workflow accept payment notifications from PayPal when a transaction is completed (not just standard purchases, but recurring payments and even any refunds that you issue), and it’s been pretty much rock solid since we launched Disk Management a couple of months ago.

That was, of course, until the ordering process happened to send a non-ASCII character in a payment request to PayPal (an accented vowel, from a customer’s first name). My order processor started throwing nasty unhandled exceptions, and the license key wasn’t emailed out to the customer.

When you take someone’s money and they don’t get what they paid for, that’s a bad (bad) thing. Something you try to take care of immediately.

After much (much) debugging and hair-pulling, it turns out that you have to do a few things to get valid UTF-8 IPN responses from PayPal. There wasn’t much that Google had to say about the issue, and PayPal’s own documentation wasn’t very clear, so here it is for posterity.

Some of these steps were fixes to my code, but some of them should be things that PayPal does by default.

Tell the IPN gateway that you want to use UTF-8, please

None of the ASP.NET PayPal IPN integration examples I could find mentioned this one. You need to specify the character set you’re going to be sending to PayPal explicitly, when you build the initial purchase request URL.

public string GeneratePurchaseRequest(string merchantReference, decimal amount)
	string paypalReturnMethod = "2";
	string paypalCharset = "UTF-8";
	Encoding encoding = Encoding.UTF8;  

	StringBuilder url = new StringBuilder();

    url.Append(paypalAccessUrl + "cmd=_xclick&business=" + HttpUtility.UrlEncode(paypalEmailAddress, encoding));
    if (!string.IsNullOrEmpty(paypalCharset)) { url.AppendFormat("&charset={0}", paypalCharset); }
    if (amount != 0.00M) { url.AppendFormat("&amount={0:f2}", amount); }
    if (!string.IsNullOrEmpty(transactionCurrency)) { url.AppendFormat("&currency_code={0}", transactionCurrency); }
    if (!string.IsNullOrEmpty(siteTitle) url.AppendFormat("&item_name={0}", HttpUtility.UrlEncode(string.Format("{0} Invoice #{1}", siteTitle, merchantReference), encoding));
    if (!string.IsNullOrEmpty(merchantReference)) { url.AppendFormat("&invoice={0}", HttpUtility.UrlEncode(merchantReference, encoding)); }
    if (!string.IsNullOrEmpty(successPage)) { url.AppendFormat("&return={0}", HttpUtility.UrlEncode(successPage, encoding)); }
    if (!string.IsNullOrEmpty(successPage)) { url.AppendFormat("&rm={0}", HttpUtility.UrlEncode(paypalReturnMethod, encoding)); }
    if (!string.IsNullOrEmpty(paypalNotifyUrl)) { url.AppendFormat("&notify_url={0}", HttpUtility.UrlEncode(paypalNotifyUrl, encoding)); }

    return url.ToString();

Pay attention to line 9 above, that’s your winner. Plus all the UTF-8 UrlEncode() stuff.

Really tell the IPN gateway that you want to use UTF-8

This is the bit that got me. I was specifying the character set I wanted PayPal to use in my request, but when I went to verify the transaction with PayPal the non-ASCII character always got mangled, and the verification failed. Both request and response forms were set to UTF-8 encoding, and everything I looked at said I was doing the right thing.

Except, I wasn’t doing the right thing that mattered most. You have to explicitly set UTF-8 encoding in your PayPal profile, while you’re logged into the PayPal site.

No, as far as I can tell, you can’t set this programmatically – you have to actually log into their management interface and tell them no, I’m serious, I really really want to use UTF-8. Here’s how (because it’s buried in the user interface):

  • Log into PayPal
  • Click the ‘Profile’ link in the menu bar under ‘My Account’
  • Click the ‘Language Encoding’ link under the ‘Selling Preferences’ column
  • Click ‘More Options’
  • Set ‘Encoding’ to ‘UTF-8’
  • Click ‘Save’


Handle the UTF-8 payment notification verification

Now PayPal is actually sending through UTF-8 responses, and your verification won’t get mangled. Well, it won’t get mangled as long as you use the raw form response.

I’ve had mixed success doing this other ways (Form.ToString() or pulling out the individual query parameters and rebuilding the query string), but your mileage may vary. BinaryRead() seems to give me the most consistent results.

public bool VerifyIpn(HttpContext context)
		Encoding encoding = Encoding.UTF8;          

		HttpWebRequest req = (HttpWebRequest)WebRequest.Create(paypalAccessUrl);

        req.Method = "POST";
        req.ContentType = "application/x-www-form-urlencoded";
        byte[] param = context.Request.BinaryRead(HttpContext.Current.Request.ContentLength);
        string strRequest = encoding.GetString(param);
        strRequest += "&cmd=_notify-validate";
        req.ContentLength = strRequest.Length;

        //Send the request to PayPal and get the response
        StreamWriter streamOut = new StreamWriter(req.GetRequestStream());

        StreamReader streamIn = new StreamReader(req.GetResponse().GetResponseStream());
        string strResponse = streamIn.ReadToEnd();

        if (strResponse == "VERIFIED")
            //PayPal says we got paid, so let's post-process this order
            //Do something else, like display "Pending payment" to the user
    catch (Exception error)
        log.Fatal(error.Message, error);
        return null;

Line 12 is where the magic UTF-8 decoding happens.

Basic IPN plumbing

I’ve skipped a bunch of the basic IPN handling and setup steps, because I’m assuming you’ve already got that working and you’ve made it to this page because someone from Germany tried to order from your store. There are plenty of examples out there about how to write your own IPN handler, so I won’t repeat those.

Hopefully this will save someone else hours of debugging. And swearing.




No comments posted yet.

Would you like to post a comment?

Post title
Your name
Your email (optional)  
Website (optional)

What do you want to say?


Please add 3 and 6 and type the answer here: