Skip to main content

Confused by online payment processing? You're not alone.

In the old days during "polite" conversation, it was considered rude to talk about sex, politics, religion and money. You might think we're done with taboos, we're not (and I'll leave Steven Pinker to make the general argument about that, as he does so well in The Better Angels of Our Nature).

The taboo I'm wrestling with is about money - not how much you make, but about online payment processing, how it works, and what it costs. In this case, I think the taboo exists mainly because of the stakes at hand (i.e. lots of money) and the fact that most of those who are involved don't get much out of explaining how it really works - i.e. the more nuanced communications are overwhelmed by sales-driven messaging, and the nuanced stuff is either proprietary secrets or likely to get slapped down by the sales department.

In other words, if you want to really understand about online payment processing because you want to decide between one system and another, you're in for a rough ride.

Several years ago I wrote a couple of payment processors for CiviCRM, and more recently I've been working on a new version of one of them. At the same time, two clients have recently been trying to understand their existing payment processor services in order to integrate those processes into their CiviCRM installations. So this is my "Payment Processor primer for CiviCRM administrators" blog post.

What You Need To Know

Here's a simplified but useful model of what's happening. A typical online payment has three phases, and each phase may be the responsibility of a different (or the same) service provider. I'm talking about a typical real-time transaction via credit card - other flavours will introduce new complications.

Phase 1: The Form

The web form is the public interface where the visitor inputs things like a name and credit card number. Sometimes, it's a two part form. Depending on the transaction, you'll want this form customized so that your visitor doesn't get confused and leave. The "depending" bit is really about your visitor's relationship to you. If they already know and love you, it probably matters less. If they're new and not yet convinced they want to give you money, it's big.

CiviCRM can support the form, but also supports payment processors that insist on doing the form themselves (e.g. paypal standard). The big advantage to CiviCRM doing the form is customization and not-alarming-or-confusing-the-visitor (e.g. the paypal form allows credit cards, but many people get to that form and bail because they think they need to sign up for a paypal account). The big disadvantage is that you need to worry about your server and something called PCI compliance, which is another topic.

Phase 2: The Transaction Processing or The Payment Gateway

This phase starts after the visitor clicks the submit or confirm button and may happen entirely in the background, or may involve the visitor in supplementary forms and clicking. This phase is the responsibility of a "Payment Gateway", a service that you have to buy unless you're a large corporation that builds their own. This payment gateway service has business contracts and software relationships with the phase 3 section. The key services they provide are to abstract away the individual details of the different card company interfaces and to take responsibility for financial compliance stuff (e.g. they need to keep those credit card numbers very, very safe ...).

CiviCRM does not try and do this, but provides interfaces to many payment gateways and in theory allows you to write an interface to any payment gateway that publishes some kind of "API" or programmer interface. It can be confusing because many payment gateways also try to be in the business of providing Phase 1 services (e.g. 'hosted forms') and it may not be obvious if there is such a thing as an API, and sometimes they call it something else.

An excellent, more technical description is here on wikipedia:

Phase 3: Transaction Completion

This is the murkiest phase, where the payment gateway, the institution behind the card (issuing bank or card association) and the "merchant account" all exchange information and some kind of electronic trail gets laid that eventually results in money being transferred from the card holder to the "merchant account". The "merchant account" is a special kind of bank account that is enabled for credit card payments. What makes it special is that the credit card companies have a noose around it's neck - i.e. they take a chunk of money before it gets to the account, and reserve the right to take the money back if there's a problem. The "merchant account" might be directly owned by you the site owner, or it might be owned by someone else who then passes the money to you.

It's not unreasonable to confuse this phase with phase 2 since they happen together, and since phase 2 without phase 3 is kind of pointless, but it's important to separate it out in terms of costs and responsibilities. Phase 2 is really a technical and business relationship service that is handling the details of the transaction (kind of like an electronic teller, or maybe the hired gun in a drug deal). Phase 3 is where the money ends up and is accounted for (the backroom settling of accounts ...).

It's also important to separate them out because you can have phase 3 stuff going on without phase 2. For example: a 'recurring transaction' where a donor says they'll give you money every month. Once the initial deal is sealed, the subsequent transactions don't need to go through phase 1 or 2 (but might anyway).

What You Actually Get and How Much it Costs

So the challenge of comparing various payment processor "solutions" is to figure out the apples and oranges. With CiviCRM you have to buy services from at least one company in order to get online payments from credit cards, but any company you find may offer a mix of services covering these three phases. Paypal standard will only give you a soup-to-nuts end-to-end solution. A merchant account will only get you the last phase and you still need a payment gateway service. If you don't have an ssl certificate for your site, you will need phase 1 services, but if you do have ssl, you probably don't want phase 1 services. Most payment gateway services will offer to bundle in a merchant account and/or a hosted payment form service. And each of these offerings will be better than the competition for reasons x, y and z. And each one will use a different vocabulary to describe what they are giving you.

So, here's what you should expect and look at:

1. Phase 1 services. You probably shouldn't pay anything for this unless they are providing really good customization - it should be a one-time and change fee for the customization of the form. Getting CiviCRM to host this form is usually a better idea unless you're doing cheap hosting and/or can't get an ssl certificate.

2. Phase 2 services. These will typically cost a monthly or yearly fixed fee, sometimes a set up fee, and probably always a per-transaction fee. There's no particular reason that there should be a % fee, since the costs of providing these service are basically per transaction + a setup and account maintenance, unless the company is trying to do some kind of gambling, which is stupid.

3. Phase 3 services. The merchant account part of the service is really about paying off the card mafioso, plus an extra handling charge to the bank. Each major card has it's own rates that they charge the bank, but MasterCard and Visa are similar and Amex costs about an extra 1%. If you're a small-medium organization, then you'll probably pay a pretty standard amount, but if you're a large organization, you can usually negotiate a better rate, which is just going to reduce the extra amount that the bank charges you. It'll never go below the industrial rate, which is complicated (i.e. it fluctuates and depends on lots of things), but I'd hazard lives at around 1.5% (why not? for a start, consider all those points reward systems out there and put that together with a certainty that card companies aren't losing money ...).

One thing that this model helps you do is to compare the bundled services, which will typically be the monthly or yearly + per transaction costs of the payment gateway plus the % costs of the merchant account. You can sometimes see how they gamble on the % costs and give you a single 'blended rate', and sometimes gamble on the size by shifting costs back and forth from per transaction to % rates.

Popular posts from this blog

Me and varnish win against a DDOS attack.

This past month one of my servers experienced her first DDOS - a distributed denial of service attack. A denial of service attack (or DOS) just means an attempt to shut down an internet-based service by overwhelming it with requests. A simple DOS attack is usually relatively easy to deal with using the standard linux firewall called iptables.  The way iptables works is by filtering the traffic based on the incoming request source (i.e., the IP of the attacking machine). The attacking machine's IP can be added into your custom ip tables 'blacklist' to block all traffic from it, and it's quite scalable so the only thing that can be overwhelmed is your actual internet connection, which is hard to do.

The reason a distributed DOS is harder is because the attack is distributed from multiple machines. I first noticed an increase in my traffic about a day after it had started - it wasn't slowing down my machine, but it did show up as a spike in traffic. I quickly saw that…

CiviCRM's invoice_id field and why you should love the hash

I've been banging my head against a distracted cabal of developers who seem to think that a particular CiviCRM core design, which I'm invested in via my contributed code, is bad, and that it's okay to break it.

This post is my attempt to explain why it was a good idea in the first place.

The design in question is the use of a hash function to populate a field called 'invoice_id' in CiviCRM's contribution table. The complaint was that this string is illegible to humans, and not necessary. So a few years ago some code was added to core, that ignores the current value of invoice_id and will overwrite it, when a human-readable invoice is generated.

The complaint about human-readability of course is valid, and the label on the field is misleading, but the solution is terrible for several reasons I've already written about.

In this post, I'd like to explain why the use of the hash value in the invoice_id field is actually a brilliant idea and should be embrac…

Upgrading to Drupal 8 with Varnish, Time to Upgrade your Mental Model as well

I've been using Varnish with my Drupal sites for quite a long while (as a replacement to the Drupal anonymous page cache). I've just started using Drupal 8 and naturally want to use Varnish for those sites as well. If you've been using Varnish with Drupal in the past, you've already wrapped your head around the complexities of front-end anonymous page caching, presumably, and you know that the varnish module was responsible for translating/passing the Drupal page cache-clear requests to Varnish - explicitly from the performance page, but also as a side effect of editing nodes, etc.

But if you've been paying attention to Drupal 8, you'll know that it's much smarter about cache clearing. Rather than relying on explicit calls to clear specific or all cached pages, it uses 'cache tags' which require another layer of abstraction in your brain to understand.

Specifically, the previous mechanism in Drupal 7 and earlier was by design 'conservative' …