Ending the Rising Threat of Zero-Day Exploits

Zero-Zero-Day Exploits

On January 4, 2023, CircleCI, a prominent CI/CD service, disclosed a significant security incident, revealing the growing challenges posed by zero-day vulnerabilities. This incident serves as a stark reminder of the need for robust security measures in software development environments.

The incident began on December 29, 2022, when CircleCI was alerted to suspicious GitHub OAuth activity by one of its customers. This triggered an extensive review by CircleCI’s security team in collaboration with GitHub. By December 30, 2022, it was determined that the customer's GitHub OAuth token had been compromised by an unauthorized third party. Although the issue was quickly resolved for that customer, CircleCI proactively decided to rotate all GitHub OAuth tokens for their customers, a process that took time despite efforts to increase API rate limits.

By January 4, 2023, CircleCI's internal investigation had uncovered the full extent of the intrusion. An unauthorized third party had leveraged malware deployed on a CircleCI engineer’s laptop to steal a valid, 2FA-backed SSO session. This malware, which was not detected by the antivirus software in use, enabled the attacker to impersonate the targeted employee and escalate access to a subset of CircleCI’s production systems. The malware was first deployed on December 16, 2022, and the unauthorized access continued until data exfiltration on December 22, 2022. The attacker extracted encryption keys from a running process, potentially accessing encrypted data.

In response to the incident, CircleCI took several immediate actions to contain and mitigate the breach. On January 4, they shut down all access for the compromised employee and limited production access to a small group for operational issues. They rotated all potentially exposed production hosts and API tokens and worked with partners like GitHub, Atlassian, and AWS to notify and assist affected customers.

CircleCI also engaged third-party cybersecurity specialists to validate their findings and enhance their investigation. They implemented additional security measures, including monitoring and alerting for specific behaviors identified in the attack, to prevent future incidents.

CircleCI has taken extensive steps to ensure that such an attack cannot recur. They added detection and blocking mechanisms through their MDM and A/V solutions for the specific behaviors exhibited by the malware. Access to production environments has been restricted to a limited number of employees, with added step-up authentication steps and controls. Continuous monitoring and alerting for the specific behavior patterns observed in this incident have been put in place across multiple triggers and third-party vendors.

The company has also committed to ongoing security enhancements, including periodic automatic OAuth token rotations and a shift from OAuth to GitHub apps for more granular permissions. They plan to perform a comprehensive analysis of all tooling configurations, supported by third-party reviews, and to expand alerting, reduce session trust, and add additional authentication factors.

CircleCI maintained active communication with its customers throughout the incident. Upon completing the rotation of all production hosts, they issued disclosure emails, posted a security notification on their blog, and notified customers via social media and support forums. Customers were advised to rotate their secrets, including OAuth tokens, Project API Tokens, and SSH keys, and to investigate any suspicious activity in their systems. CircleCI also introduced new tools to support customers in securing their environments, such as a script for secret finding and changes to the CircleCI API to help customers verify the successful rotation of variables. Audit logs were made accessible to all customers to aid in reviewing platform activity.

This incident highlighted the need for continuous evolution in security practices. CircleCI recognized that while they had tools in place to deter and detect attacks, there is always room for improvement. They plan to optimize the configuration of existing tools to create additional layers of defense and make it easier for customers to adopt advanced security features.

The CircleCI breach underscores the critical importance of robust security measures in protecting software development environments from sophisticated cyber threats. Solutions like CodeLock, with advanced malware detection, continuous biometric authentication, and real-time monitoring, can play a vital role in preventing such incidents. By integrating proactive security measures, organizations can better protect their systems from the ever-evolving landscape of cyber threats.