About PG&E

Pacific Gas and Electric Company is one of the largest combined natural gas and electric energy companies in the United States. Based in San Francisco, PG&E has approximately 23,000 employees who carry out the utility’s primary business — the transmission and delivery of energy. The company provides natural gas and electric service to 16 million people throughout a 70,000-square-mile service area in northern and central California.

Establishing Best Practice: Portfolio Approach to Measuring Training Effectiveness

PG&E Academy has spent years investing in and improving its training programs to ensure employee competence across all critical roles in the company. Executive leadership increasingly requires measurable data that demonstrates training’s value and contribution to attaining business goals. To position themselves as critical business partners, PG&E Academy has instituted a portfolio approach to measuring training effectiveness.

This portfolio uses a phased approach that includes three categories of measurement:

1. Performance outcomes (L3 and L4)
2. Compliance (required)
3. Expectations (L1 and L2)

All Academy training programs include Expectation measurement. High-risk, high-consequence programs are subsequently evaluated for performance outcomes based on the following criteria:

• Is there a clear business reason for the training (business metric alignment)?
• Have associated performance metrics and targets been identified (what good looks like)?
• Is the sponsor committed to measuring training effectiveness and participating in ongoing data gathering?
• Is baseline data available for future comparison?

The Academy uses traditional L3 assessments administered through focus groups, surveys to employees and supervisors (30/60/90 days), and field observations. This L3 data, combined with L4 data from the QA/QM team is consolidated into the Academy’s performance outcome initiatives and used to demonstrate how training programs are contributing to meeting the business goals and objectives of the company.

Currently, 22% of the Academy’s portfolio is evaluated for performance outcomes (L3 and L4). The longer-term goal is to focus exclusively on training initiatives that can be aligned to business metrics, with the idea that those that cannot may not be necessary and/or needed.

Predictive L3

Because L3 assessments can be difficult and expensive, the Academy uses predictive L3’s as another method of measurement. Four questions are added to L1 surveys that help predict knowledge transfer from training to the field. While not perfect, student responses to these questions are a proven indicator of whether the training received will be applied back on the job. The following questions are rated on a scale of one to five:

• This training increases the overall safety of my work.
• I will be able to use what I learned in this training on the job.
• I understand the business purpose for attending this training.
• How much of this training do you plan to use on the job?

The Academy’s target for predictive L3 average ratings across courses is 4.52. According to Metrics that Matter, with the addition of a 60-day post training survey, the predictive L3 ratings are within 5% of observed L3’s across all industries.

Scorecard Example

The Service Mechanic scorecard shown below highlights how training contributed to increased capacity and quality of work. This level of transparency and detail has enabled the Academy to demonstrate the value of training and build enhanced credibility as a strategic business partner. The result is an increase in executive support of training programs and ongoing collaboration between the lines of business and the Academy.

Keys to Success

• Credibility of the training organization. Rarely can an organization point to training as the sole reason for impacted business metric(s). Operations needs to see training as a partner that is working in collaboration to achieve or exceed business goals.
• Consistent support from operations. Line of business support is critical to identify and track the metrics enabling the evaluation of training effectiveness.
• A consultative approach. The training team must be able to influence the business on the importance of measuring training effectiveness. Very often, training is thrown out as the solution to a business problem, when a better solution may lie elsewhere.
• By having a partnership with the business that fosters open dialogue and trust, the business can think about what outcomes they are seeking, then determine if training is in fact the correct solution and prove its value that after the fact.
• A venue to share results. The results and what they mean for the business must be communicated across the organization.

This demonstrates value to the impacted audience, and positions training to become a valuable resource for other groups.