How to choose the right PCI DSS QSA

A few weeks ago (1st July 2010), I was a speaker and an expert panellist at PCI London. One particular subject which I spoke about generated a lot of interest from the mainly merchant delegates, and the QSA representatives, it was my views on PCI Qualified Security Assessors (QSAs). Specifically how merchants should go about selecting a quality QSA to help become and maintain their PCI DSS compliance. In my experience in working within the payment security field and with PCI compliance for many years, I find there are still far too many dodgy QSA individuals and QSA service providing companies out there, misadvising their clients with bad advice and providing merchants with what I personally call placebo PCI DSS compliance assurance.

Low Budget means PCI Fail
A QSA company should never be selected solely based on cost, as you tend to pay for what you get in the QSA provision world. Low budget tends to underpin a half hearted approach to PCI compliance, usually such an approach is just an attempt to tick the old compliance box; this is a sure way to PCI DSS failure. Typically in the QSA industry, I find the cheaper you pay the worst quality QSA you are likely to receive, although there are sometimes the odd exception, so I would recommend avoiding scrapping the bottom of the barrel if you are serious about PCI DSS compliance.

QSA status means little
An individual holding QSA status does not make them some sort of PCI god, the truth is, it is not too difficult to become QSA qualified, until recently the QSA exam was an “open book” exam. I find that individual QSAs are certainly not all cut from the same cloth, and there plenty of variation in their ability and dept of knowledge of the PCI standard, so do not blindly trust an individual knows all PCI DSS because he or she is QSA qualified.

The best QSA qualified individuals tend to be employed by the best QSA companies, typically companies which specialise in providing information security consultancy service, where QSA work is a core function of the business. This is as opposed to companies and security vendors which have "bolted on" a QSA business function just to get in on the PCI gravy train.

Selecting a QSA Partnership
Your QSA should be a partnership as opposed to a client auditor relationship. Certainly during the actual PCI DSS assessment process the QSA will be in an auditor mode, however prior and post your PCI DSS assessment your QSA needs to be acting as your PCI consultant, and not just for a month or two leading up to the assessment either, but 365 days a year, you should be able call on your QSA at anytime and obtain clear PCI advice on demand.

QSA Company Vetting Questions
The key question to ask a QSA company is not how many individual QSAs they have on their roster, but how many PCI DSS assessment, aka Reports of Compliance (RoCs), they have completed and submitted during the last 12 months. If the answer is over 20 then you know you are dealing with an organisation which has a specialism in performing top level PCI DSS assessments, anything less than 10, then to be brutally honest, you are likely to be dealing with an amateur QSA organisation, which may not be suitable if you are responsible for PCI compliance at a large and complex environment for instance.

There are many QSA companies out there which only ever submit a handful of RoCs each year. Interestingly along side the expected small time QSAs companies, there are several big name security vendors which fall into the category of performing less than 5 RoCs a year. So it is always well worth asking your QSA organisation, the question “how many RoCs have you submitted in the last 12 months?” so you are clear on the QSA organisation’s true QSA validity and general level of PCI expertise and experience.

Failing QSAs / QSAs in Remediation
The next part in vetting a QSA organisation before signing up, is to check whether they are in remediation on the PCI Security Standard Council's website – see direct link below, where the QSAs in remediation are marked on their listing.

https://www.pcisecuritystandards.org/pdfs/pci_qsa_list.pdf

This is a list of QSA companies which have at least one individual QSA that has failed to perform an adequate PCI DSS assessment in the view of the PCI Security Standard Council (PCI SSC). The PCI SSC oversees the standard requirements, all QSAs and performs quality control against QSAs, specifically by checking QSA reports to ensure QSAs have done their job properly.

Where the PCI SSC find an individual QSA has done a poor job in assessing a company, the PCI SSC put that individual's QSAs entire company into remediation, as it is part of responsibility of the QSA organisation to ensure their QSAs are doing their job correctly. Remediation is not the end of the world for a QSA organisation, it just means they have to ensure they resolve the problems with the individual QSA, however if they don’t, the QSA company can be delisted as being a QSA provider. The idea here is for PCI DSS QSA assessment quality control, this process is all about weeding out the bad QSAs and poor QSA companies.

Vetting the Individual QSA
After selecting a QSA company, do not yet sign up to anything before assessing your assigned individual QSA. Ensure you meet that individual before signing any agreements, and carry out your own assessment on the QSA individual, either directly or covertly by asking questions. I recommend the following areas to assess your QSA against.

1. EXPERIENCE
How many past assessments have they done?
The best QSAs tend to have many years of industry and on the job experience in payment card security, and so should be able to reel off a list of previous clients they have assisted.

How complicated were their past assessment environments?
If you have complicated multi-site environment to validate against PCI DSS, it is no good having a QSA who has never assessed anything larger than a corner shop.

How long have they been with their QSA organisation?
The relationship with the individual QSA can be vital, as this is the person who you will have to explain your environment and payment processing operation to. Your environment and road to PCI DSS compliance is nearly always going to be unique to your business, if an individual QSA has a history of moving around between QSA organisations, which many do, then it is likely that you will be assigned a new QSA, and will have to start all over again in explaining your environment, payment operations and your approach to meeting PCI DSS compliance.Be aware that some individual QSAs may have different views on the way you should be meeting your PCI compliance in specific areas, such as with virtualisation, and so a new QSA may well disagree with your previous QSAs agreed approach, even if they work for the same QSA company. There is nothing worst than having a QSA leave their parent company a couple of days before your on site assessment; weeks of your preparation work can go down the pan, so it’s certainly worth asking this question, and being aware of the risk it poses.

2. KNOWLEDGE
Most merchants tend to be in a position where the QSA will know more about PCI DSS than they will ever do, however you can still research the latest hot and contentious topics in the PCI DSS industry, trust me there are always areas within the PCI standard which are in contention and in debate. Then ask your QSA for his or her view. If the QSA cannot provide a consist response and explain clearly the issues to you, then this is a tell tail sign of a lack of individual knowledge and confidence around the standard. QSAs which aren’t very confident about every aspect about the PCI standard tend not to have the experienced background, which in turn builds their knowledge around on the standard.

Consistency
If a QSA changes his or her views, provides inconsistence advice, this is a definite red flag, and tends to mean the QSA is not knowledgeable and experienced. I have heard of situations where companies have spent £100,000s on IT systems following the advice of an individual QSA, only for that QSA change their advice (and their mind) down the line, resulting in their PCI budget being completely misdirected.

A Good Information Security Professional does not equate to a good Payment Card Security Professional

QSAs from a general information or IT security background do not always make for a good QSA. Sure much of the PCI standard is industry information security best practices, but it is equally important that your QSA understands how your card payment systems work, and how cardholder data flows through your organisation's IT systems, from your cash offices, to PDQs, to Call Centres to your payment processor or acquiring bank, a regular industry information security professional knows very little about such areas, unless they have been specially involved in it.

3. ATTITUDE
Is your QSA helpful or acting like a typical auditor all the time?

There are QSAs out there who tend to be from strong auditor backgrounds, and as such may often be very poor in providing clear advice and guidance on how to address any issues they uncover or highlight to you. For instance they tend to say “No you can’t do that find another way”, whereas a quality QSA will say “No you can’t do that, but here's how you can do it”, then provide you with the right solution for your environment in a high degree of detail.

If your QSA is worth his or her salt, they will go out of their way to obtain a thorough understanding of your environment from day one, so they are in a strong position to provide you with the correct advice you need.

4. TRUST AND CONFIDENCE
If a QSA is ever not completely honest with you, you must drop them like a stone. For instance I have heard of several QSAs which deliberately avoided telling their clients that their organisation has gone into remediation, as I said remediation is not the end of the world for a QSA. A decent QSA would tell you they have gone into remediation, why it happened and what they are doing to correct their issues, as oppose to taking a stance of deliberately not informing you.

Ethical
I once came across one individual QSA, who within minutes of first meeting him told me about specific PCI compliance issues with his existing and past clients, and then went on to give me details about a cardholder compromise that another client had suffered, which to date has never been publicly disclosed. Is this the sort of guy you want to trust as your QSA?

There needs to be a high level of trust between you and your appointed QSA, as after all you will be providing a high degree of sensitive information about your company’s IT systems and security. You certainly need to ensure your QSA company’s has it’s own security all in order to protect any information you share. Ensure you have a non-disclosure and data retention contract in place, and make sure your QSA is fully aware that you don’t want them to share your organisations information with any other third parties.

Another side of ethics I have encountered are folk calling themselves a QSA when they aren’t employed by a QSA organisation. You cannot be a QSA unless you work for a QSA organisation, there is no such thing as an independent QSA, anyone pretending to be QSA should be shown the door. You can check an individual’s QSA status by checking on the PCI SSC website, you must always make sure the QSA is currently qualified and is in good standing, see the PCI SSC link below.

http://www.pcisecuritystandards.org/qsa_lookup/index.html

Finally remember your QSA should be a 365 day a year “partner”, not a once a year auditor, PCI DSS compliance itself is not a once a year event but a 365 day a year continued state.