Making the Case: The Success Case Method in Training Evaluation

This month, we explored various aspects of learning and development, from creating a learning strategy and cultivating a learning culture to selecting tools and technology. However, our discussion would not be complete without an exploration of training evaluation. Whether your organization is evaluating a single training event or a comprehensive training program, the importance of training evaluation remains constant. Conducting an evaluation provides insight into a training’s effectiveness and value. These insights are then used as the foundation for informed decision-making about the future of training, such as decisions around instructional design modifications. Without such information, organizations may be tempted to guess or make gut decisions, leading to undesired outcomes or wasted resources.  

While research in recent years indicates an increased focus on measuring the impact of training, many organizations continue to struggle to effectively do so (Training Magazine, 2017). For example, in practice, many organizations focus solely on initial reactions or immediate learning, rather than impacts which occur on the job or to the business at large. Recently, the Association for Talent Development (ATD) found only 35% of sampled professionals reported their organization evaluates the business impact of training programs (ATD, 2016). There are likely many reasons for this occurrence, from reduced financial resources, limited time, lack of interest, or all of the above. Yet, it is clear organizations can and should do more to evaluate their training.

However, training evaluation is not a one size fits all activity. So, in a world of Kirkpatrick’s Four Levels of Evaluation and Phillip’s ROI Methodology, which may be often cited as time consuming and challenging to implement well, what other evaluative methods exist? Enter the Success Case Method (SCM) (Brinkerhoff, 2003). While the aforementioned methods rely heavily on quantitative data and metrics, the SCM shifts the focus to qualitative data, and in particular, the art of storytelling. As the founder of SCM Robert Brinkerhoff states, “the Success Case Method is a carefully blended balance of the ancient art of storytelling with more modern methods and principles of rigorous evaluative inquiry and research” (Brinkerhoff, 2003).

Storytelling is a concept we are all likely familiar with from childhood. However, its reach extends beyond childhood bedtime stories. Research in the field of neurobiology has also demonstrated its relevance in business settings – information presented in a story format has been shown to result in greater understanding of key points, as well as improved recall later on (Zak, 2014). Following these findings, then might training evaluation have a greater impact if it were presented as more than the sum of its quantitative data? Inherent to training are an organization’s employees, each with stories of their own. The SCM aims to bring their stories to light in order to inform decision-making about how to enhance training outcomes. By linking evaluative results to these employees, this method can help uncover a deeper understanding of the external variables and challenges impacting training effectiveness (e.g., work environment, job satisfaction). Let’s explore more about what makes the Success Case Method a good option to add to your arsenal of training evaluation tools.

Although the SCM is rooted in storytelling, you’d be mistaken to think it does not provide meaningful results or is not rigorous. To the contrary, this method can still link to business results and on-the-job behavior. Using the SCM can help organizations answer questions such as, “What results are being achieved?”, “What is the value of the results?”, and “How can the training be improved?” (Brinkerhoff, 2003). To answer these questions, the SCM seeks out stories from two polar types of employees: employees who have experienced success as a result of training and those who have not. This method encourages organizations not to focus solely on the positive, negative, or average experiences. Nothing is ever likely to result in 100% success or failure; exploring only one side or the median can leave room for misrepresentation. By screening and selecting for these extreme groups and then exploring each more deeply through inquiry and research (e.g., through interviews, focus groups), organizations can understand specifically what each group is doing differently and identify steps to make the training more impactful. For example, a better understanding of what has contributed to success can create a platform to disseminate best practices for the application of training content on the job. Speaking to rigor, not any story can be used in this method; each story must be backed up by verifiable and documented facts (e.g., sales data). Let’s see how this might work in your organization.

Let’s imagine your organization has recently implemented a new time and attendance system. To coincide with the new system, a training course has been developed to teach employees how to properly use the system, with the ultimate goal of having timely and accurate timesheet submissions. Using the SCM, you might begin by reviewing system records or existing post-training surveys to identify cases of success and nonsuccess in reaching this goal. In this example, success cases might include employees that have submitted recent timesheets on time and with no errors. On the other hand, cases of nonsuccess may include employees that have continuously submitted late and inaccurate timesheets. To better understand why and how this is happening, you conduct interviews with these two groups. In doing so, you might find that those who have been successful created a tip sheet based on the training content and have worked with their managers when questions arose, while unsuccessful employees have done neither. Using the interview results as support, you might recommend implementing additional post-training reinforcements, such as a toolkit which recaps key content from the training or encourage managers to speak with employees the new system.

You may be thinking gathering stories from multiple groups of employees seems like a lot of work. However, by focusing only on a few, select stories, organizations can avoid compiling thousands of data points or analyzing long-term metrics, which can take years to materialize. By coupling the SCM with your organization’s existing, more comprehensive evaluation methods (e.g., Kirkpatrick), some of the necessary data points to execute the SCM may already exist. For example, Kirkpatrick-based Level 1 or Level 2 surveys can provide data needed to screen and identify successes and nonsuccesses. Another use of the SCM is evaluating changes among training or training programs; for instance, during training pilots or after major revamps of training content. In fact, change is where the value of the SCM really begins, extending beyond just training evaluation. The SCM is also an excellent option for evaluating the effectiveness change management initiative, including the implementation of new tools, organizational structures, or processes. So, next time you’re thinking of evaluation, consider incorporating storytelling using the Success Case Method. 

To close, remember that not all training evaluation methods will fit the needs of your organization. For instance, the SCM won’t provide as comprehensive or robust of results as Kirkpatrick or ROI. Therefore, when considering which evaluation method to implement, it is best to determine if it aligns with your desired outcomes, organizational environment, and expected resources. But, no matter which method is used for your training evaluation, FMP has the skill and experience to guide you at each stage. Reach out to FMP’s Learning and Development Center of Excellence (COE) to learn more!   

About the Author: From needs analyses to evaluation, Christine enjoys supporting her clients to think more strategically and data-minded about their learning and development programs. But, when she isn’t in the office, you can find her dominating FMP’s fantasy football league (sorry, not sorry!), reading up on the latest Royal family gossip, and attending barre class.

Sources:

Association for Talent Development (ATD). (2016). Evaluating Learning: Getting to Measurements that Matter.

Brinkerhoff, R. O. (2003). The Success Case Method: Find Out Quickly What’s Working and What’s Not.San Francisco: Berrett-Koehler Publishers, Inc.

Training Magazine. (2017). 2017 Training Industry Report .

Zak, P. J. (2014, 28 October).Why Your Brain Loves Good Storytelling. Retrieved from Harvard Business Review: https://hbr.org/2014/10/why-your-brain-loves-good-storytelling