GitHub case study: Enhancing customer support with AI

AI accelerates more than just developer productivity

Since the general availability of our AI assistant in GitHub's support portal in February 2024, 60% of the cases presented to it were resolved, with the majority being solved in under 7 minutes. This investment is part of GitHub's commitment to leveraging AI across the Software Development Lifecycle - including crucial moments when engineers need support! Let's take a look at how engineers at GitHub used AI within our support business to better serve our customers.

How GitHub improved customer support with AI

Primary goal

• Improve GitHub's customer support experience by leveraging automation powered by AI.

Support Portal Reach

• GitHub serves a global community of over 100 million developers. GitHub's support portal assists an average of 178,000 customers each month with product and support-related questions.

Key objectives

• Enhance responsiveness.

• Increase customer satisfaction.

• Allow the support team to focus on more complex and long standing issues.

Specific focus

Streamline the resolution of level 1 support queries by making known solutions easily discoverable, providing customers with a more direct and efficient route to resolution.

What was done

To achieve these objectives, we developed and implemented an AI-powered conversational assistant. This assistant is designed to autonomously address routine inquiries, providing immediate assistance and surfacing known solutions more effectively for our customers. The model is grounded to GitHub's extensive product documentation, helping ensure that the AI provides accurate and relevant guidance based on our existing knowledge bases. Our development and deployment process was underpinned by Microsoft Responsible AI principles

By integrating AI into this process, we aimed to reduce the number of steps and overall process friction that customers often face, while simultaneously allowing our support team to apply their expertise to more complex cases that require human ingenuity and deeper investigation.

The AI-powered conversational assistant was seamlessly embedded into the support portal as a fully optional feature where customers could choose to interact with the assistant if they desired. Those who opted to engage with the AI-assistant received immediate assistance. Additionally, customers were encouraged to provide feedback on their experience, helping us continuously improve the tool and better serve their needs.

Impact of the AI-assistant

Since the General Availability (GA) release in February 2024, 60% of the cases presented to it were resolved, with the majority being solved in under 7 minutes. This is equivalent to over 20,000 customers having their problems solved through Generative AI.

By resolving these queries, the AI assistant has allowed our support team to focus on more complex customer questions, where their expertise can have the most impact. We have seen customer trust and satisfaction in the system grow through increased usage since general availability.

Future implementations

GitHub support's AI assistant has been deployed into Copilot chat on GitHub.com. This capability helps GitHub users access product & support related information whilst remaining within their workflow.

We are currently developing further AI capabilities tailored specifically for our internal support team, aiming to further enhance their ability to deliver exceptional customer service. This tool will provide intelligent suggestions, assist with routine task automation, and offer quick access to relevant information, enabling our support staff to manage their workflow more effectively and address complex inquiries with greater ease. We are also planning to expand the knowledge base for the AI-assistant and refine its response capabilities to better handle a wider range of inquiries.

Insights Gained

The power of tailored AI experiences

Throughout the development and deployment of the AI assistant, one significant lesson learned was the importance of tailoring experiences. Our engineers discovered that by optimizing the AI assistant based on customer queries---through tailored responses and context-specific guidance---we could significantly enhance its relevance and effectiveness across a broad range of use cases. For instance, developers and admins often require different types of support, and we've built the system to adapt accordingly. By taking into account the size and complexity of the user, we enable the AI assistant to deliver personalized assistance that addresses the specific needs of each user.

Expanding innovation beyond productivity

Another key takeaway is the potential for AI to drive innovation beyond just enhancing productivity. We encourage engineers to experiment with AI, pushing the boundaries of what's possible. Consider new ways in which AI could be leveraged to achieve novel customer and business outcomes, such as creating more personalized customer experiences. By thinking creatively about the possibilities, engineers can support their organizations to unlock new value and opportunities that go beyond the traditional scope of support automation. This proactive approach to exploring the capabilities of AI not only enhances customer satisfaction but also positions your organization to lead in their respective fields, using AI as a powerful tool for innovation.

How to enhance your organization with AI

Bring a customer-centric approach to engineering with AI:

1. Identify a customer pain point that may benefit from the power of AI.

2. Define the benefits you want to deliver to your customer and any disbenefits you want to avoid and evaluate different AI solutions to address the customer pain point. Ensure you plan for tracking customer feedback and whether the solution is achieving the desired benefits.

3. Customize and implement your AI solution. Based on your evaluation, consider an initial high-level architecture proposal. It can be useful to undertake a proof of concept (POC) to test your proposal for the likelihood of meeting the target benefits. If your POC is promising, move into building for production ensuring ongoing testing as you build and that you meet any of your internal development processes and requirements. Once you have finished development and completed any release activities required by the team/organization, deploy to a small number of customers for initial feedback.

4. Review and iterate. Regularly collect feedback from your customers and internal stakeholders and scale your deployment of the solution as you grow confidence. Use feedback to refine the solution continuously, enhancing its effectiveness and ensuring it continues to deliver value.

Looking forward to the possibilities ahead for GitHub Support

As we continue to refine and expand the usage of AI in GitHub Support, we are excited about the broader possibilities it presents for enhancing various aspects of our services. The success of AI in handling level 1 support queries has demonstrated the potential of automation to not only improve responsiveness and customer satisfaction but also to empower our support team to tackle more complex challenges.

Looking ahead, we see significant opportunities to extend the capabilities of AI in the realm of customer success:

• Product usage guidance. AI could be adapted to assist users in navigating and utilizing GitHub's features more effectively. By offering real-time guidance and tips based on user behavior and context, AI could enhance the overall user experience, helping customers unlock the full potential of our platform.
• Support data analytics. We envision AI playing a crucial role in support data analytics, where it can help identify common issues, trends, and patterns. This data-driven approach can provide valuable insights to both our support and product development teams, leading to proactive improvements in our products and services that directly benefit our customers. We recently published a blog post on how we harness AI to transform customer feedback into action.
• Training and onboarding. Beyond supporting the support team, AI can be instrumental in the training and onboarding of new team members across the organization. By providing instant access to a wealth of knowledge, best practices, and contextual guidance, AI can help new employees quickly become productive, accelerating their integration into the team.

In conclusion, AI gives engineers an unprecedented opportunity to enhance both customer and business outcomes in an organization. GitHub will continue to explore these possibilities and collaborate closely with developers, and customers to ensure that AI continues to evolve and meet the ever-changing needs. If you haven't yet, try GitHub's Support AI-assistant by visiting the GitHub Support Portal and entering your query in the search bar. We value your feedback and would love for you to be part of this exciting journey as we continue to improve how we serve our community. 

Will you join the call to use AI to accelerate your organization's success? The journey ahead is full of potential, and we are excited to realize these opportunities together.