A place where we write words Ignition Blog

Welcome to the Ignition Development blog, where we talk about a wide range of technical and non-technical topics.

April 2010 Entries

PayPal Instant Payment Notification and UTF-8 encoding

I spent almost an entire day last week fighting with UTF-8 encoding and PayPal IPN integration with the Tentacle Software store.

From a development perspective, I really like IPN. It lets your order processing workflow accept payment notifications from PayPal when a transaction is completed (not just standard purchases, but recurring payments and even any refunds that you issue), and it’s been pretty much rock solid since we launched Disk Management a couple of months ago.

That was, of course, until the ordering process happened to send a non-ASCII character in a payment request to PayPal (an accented vowel, from a customer’s first name). My order processor started throwing nasty unhandled exceptions, and the license key wasn’t emailed out to the customer.

When you take someone’s money and they don’t get what they paid for, that’s a bad (bad) thing. Something you try to take care of immediately.

After much (much) debugging and hair-pulling, it turns out that you have to do a few things to get valid UTF-8 IPN responses from PayPal. There wasn’t much that Google had to say about the issue, and PayPal’s own documentation wasn’t very clear, so here it is for posterity.

Some of these steps were fixes to my code, but some of them should be things that PayPal does by default.

Tell the IPN gateway that you want to use UTF-8, please

None of the ASP.NET PayPal IPN integration examples I could find mentioned this one. You need to specify the character set you’re going to be sending to PayPal explicitly, when you build the initial purchase request URL.

public string GeneratePurchaseRequest(string merchantReference, decimal amount)
	string paypalReturnMethod = "2";
	string paypalCharset = "UTF-8";
	Encoding encoding = Encoding.UTF8;  

	StringBuilder url = new StringBuilder();

    url.Append(paypalAccessUrl + "cmd=_xclick&business=" + HttpUtility.UrlEncode(paypalEmailAddress, encoding));
    if (!string.IsNullOrEmpty(paypalCharset)) { url.AppendFormat("&charset={0}", paypalCharset); }
    if (amount != 0.00M) { url.AppendFormat("&amount={0:f2}", amount); }
    if (!string.IsNullOrEmpty(transactionCurrency)) { url.AppendFormat("&currency_code={0}", transactionCurrency); }
    if (!string.IsNullOrEmpty(siteTitle) url.AppendFormat("&item_name={0}", HttpUtility.UrlEncode(string.Format("{0} Invoice #{1}", siteTitle, merchantReference), encoding));
    if (!string.IsNullOrEmpty(merchantReference)) { url.AppendFormat("&invoice={0}", HttpUtility.UrlEncode(merchantReference, encoding)); }
    if (!string.IsNullOrEmpty(successPage)) { url.AppendFormat("&return={0}", HttpUtility.UrlEncode(successPage, encoding)); }
    if (!string.IsNullOrEmpty(successPage)) { url.AppendFormat("&rm={0}", HttpUtility.UrlEncode(paypalReturnMethod, encoding)); }
    if (!string.IsNullOrEmpty(paypalNotifyUrl)) { url.AppendFormat("&notify_url={0}", HttpUtility.UrlEncode(paypalNotifyUrl, encoding)); }

    return url.ToString();

Pay attention to line 9 above, that’s your winner. Plus all the UTF-8 UrlEncode() stuff.

Really tell the IPN gateway that you want to use UTF-8

This is the bit that got me. I was specifying the character set I wanted PayPal to use in my request, but when I went to verify the transaction with PayPal the non-ASCII character always got mangled, and the verification failed. Both request and response forms were set to UTF-8 encoding, and everything I looked at said I was doing the right thing.

Except, I wasn’t doing the right thing that mattered most. You have to explicitly set UTF-8 encoding in your PayPal profile, while you’re logged into the PayPal site.

No, as far as I can tell, you can’t set this programmatically – you have to actually log into their management interface and tell them no, I’m serious, I really really want to use UTF-8. Here’s how (because it’s buried in the user interface):

  • Log into PayPal
  • Click the ‘Profile’ link in the menu bar under ‘My Account’
  • Click the ‘Language Encoding’ link under the ‘Selling Preferences’ column
  • Click ‘More Options’
  • Set ‘Encoding’ to ‘UTF-8’
  • Click ‘Save’


Handle the UTF-8 payment notification verification

Now PayPal is actually sending through UTF-8 responses, and your verification won’t get mangled. Well, it won’t get mangled as long as you use the raw form response.

I’ve had mixed success doing this other ways (Form.ToString() or pulling out the individual query parameters and rebuilding the query string), but your mileage may vary. BinaryRead() seems to give me the most consistent results.

public bool VerifyIpn(HttpContext context)
		Encoding encoding = Encoding.UTF8;          

		HttpWebRequest req = (HttpWebRequest)WebRequest.Create(paypalAccessUrl);

        req.Method = "POST";
        req.ContentType = "application/x-www-form-urlencoded";
        byte[] param = context.Request.BinaryRead(HttpContext.Current.Request.ContentLength);
        string strRequest = encoding.GetString(param);
        strRequest += "&cmd=_notify-validate";
        req.ContentLength = strRequest.Length;

        //Send the request to PayPal and get the response
        StreamWriter streamOut = new StreamWriter(req.GetRequestStream());

        StreamReader streamIn = new StreamReader(req.GetResponse().GetResponseStream());
        string strResponse = streamIn.ReadToEnd();

        if (strResponse == "VERIFIED")
            //PayPal says we got paid, so let's post-process this order
            //Do something else, like display "Pending payment" to the user
    catch (Exception error)
        log.Fatal(error.Message, error);
        return null;

Line 12 is where the magic UTF-8 decoding happens.

Basic IPN plumbing

I’ve skipped a bunch of the basic IPN handling and setup steps, because I’m assuming you’ve already got that working and you’ve made it to this page because someone from Germany tried to order from your store. There are plenty of examples out there about how to write your own IPN handler, so I won’t repeat those.

Hopefully this will save someone else hours of debugging. And swearing.



This blog entry was posted @ Friday, April 9, 2010 10:06 PM | Feedback (0)

GST and the SFF

Chances are that National’s new GST plans are keeping some small business owners up at night and making IT consultants rub their hands together with glee.

Most people will realise that a change in GST (or any  other tax rate) will mean a change to any computer system that involves money. Fortunately the developers of most modern systems are usually smart enough to store things like tax percentages in the application’s configuration – well, most of them are. Murphy’s law states that as soon as anyone assumes that “The tax rate won’t ever change!” then odds are it will – tomorrow - twice.

However sometimes being able to change that figure alone isn’t enough, and that’s when the fun begins. For example, what about reporting – everyone loves reporting! If you’ve got a single setting for a certain tax rate in your system, then what happens when you’re reporting on a mix of past and present transactions? If the application calculates tax values on each transaction based on that one configuration value then the reporting will be incorrect, as it doesn’t take the variation in tax rate into consideration. Obvious, right? You’d be surprised – you really really would.

There’s many different ways to address this problem, however whatever approach is used one thing holds true – it’s always going to be cheaper if the application is designed to handle this scenario when it is built, rather than needing to have the fix retroactively applied. Hindsight is a wonderful thing, but foresight can save you a lot of money.

Last year when we were working on the ecommerce features of the Site Foundation Framework, we made some decisions around what data we’d store in the database. At the time, storing some of that information felt a little redundant – however we took the cautious route, and added a few figures which some systems may have been tempted to leave out for reasons of efficiency / disk space (the latter making very little sense with the cost of storage today).

In some ways this goes a little against the YAGNI principle however it was pretty easy to justify for the ecommerce information because when it comes to money, more information is always better. More money is always better too, but that’s another story entirely. For reasons of compliance, and of course graphing and reporting – more figures means more shiny graphs and reports, and as we know, people like their graphs and charts.

The result is that we’re currently in a position where we’re not going to have to modify any code when the tax changes kick in. We (or our customers themselves) simply update a configuration value using the Administration UI, and they’re good to go – simple.



This blog entry was posted @ Friday, April 2, 2010 11:39 AM | Feedback (1)