John Ragsdale of the Service & Support Professionals Association (and eStara Thought Leader series alum) offers a few tips on what questions to ask vendors during your research.  Here are some suggestions he makes for managing the reference process and maximizing the effectiveness of each reference:

• Insist on references for any modules or functions that are critical for a successful implementation.
• Ask about “out-of-box” functionality vs. site customization.
• Request references in your vertical market.
• Similarly, request references using your technology platform of choice. 
• Go on-site. While some phone references are fine, go to a few customer sites and talk to the people actually using the applications, as well as the front-line managers for the service teams.
• Ask the tough questions. What worked and what didn’t? What would you do differently next time? What was more difficult than promised?
• Ask about “shelf-ware.” Did they implement all modules purchased?
• Don’t forget the “R” in CRM. These vendors are selling relationship software; they should be the relationship experts. What sort of relationships do they have with the customers?
• Conduct all or part of the interview without the vendor present.
• Finally, ask what the customer is getting in return for the reference.