Insecure coding workshop: Analyzing GitHub Copilot suggestions

AI-generated code suggestions from GitHub Copilot can be a huge timesaver – but what about security? Invicti security engineer Kadir Arslan examines Copilot suggestions and points out common vulnerabilities that you need to look out for in the generated code.

Insecure coding workshop: Analyzing GitHub Copilot suggestions

Since it was introduced, GitHub Copilot has already saved developers thousands of hours by providing AI-powered code suggestions. Copilot suggestions are definitely useful, but they were never intended to be complete, correct, working – or secure. For this article, I decided to take Copilot for a test flight to check the security of the AI’s suggestions.

First things first: what exactly is GitHub Copilot?

Copilot is an IDE plugin that suggests code snippets for various common programming tasks. It tries to understand comments and existing code to generate code suggestions. Copilot uses an AI-powered language model trained on thousands of pieces of publicly available code. As of this writing, Copilot is available by subscription to individual developers and supports Python, JavaScript, TypeScript, Ruby, and Go.

GitHub Copilot security concerns

Copilot is trained on code from publicly available sources, including code in public repositories on GitHub, so it builds suggestions that are similar to existing code. If the training set includes insecure code, then the suggestions may also introduce some typical vulnerabilities. GitHub is aware of this, warning in the FAQ that “you should always use GitHub Copilot together with good testing and code review practices and security tools, as well as your own judgment.”

Soon after Copilot was released, researchers from the New York University’s Center for Cybersecurity (NYU CCS) published Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. For this paper, they generated over 1600 programs with Copilot suggestions and checked them for security issues using automated and manual methods. They found that the generated code contained security vulnerabilities about 40% of the time.

This was a year ago, so I decided to do my own research to see if the security of Copilot suggestions has improved. For this purpose, I created two skeleton web applications from scratch using two popular tech stacks: a PHP application backed by MySQL and a Python application in Flask backed by SQLite. I used Github Copilot suggestions wherever possible to build the applications. Then I analyzed the resulting code and identified security issues – and here is what I found.

Copilot suggestions in a simple PHP application

For the first application, I used PHP with MySQL to represent the LAMP stack – still a popular web development choice even in 2022, most likely due to WordPress. To check some common login form scenarios, I created a simple authentication mechanism. As a first step, I manually created a new database with a new table (users), and the connect.php file. I then used Copilot to generate the actual login code, as shown below. Lines 36–48 were generated by Copilot:

Immediately, you can see that the SQL query in $query is built in a way that is vulnerable to SQL injection (values supplied by the user are directly used in the query). Here’s an animation showing how Copilot responded to a comment to suggest this code block:

Next, I created the index.php page that just says hello to the user. Apart from comments for Copilot, I didn’t write a single line of code. For a developer, it is very fast and convenient… But is it safe? Look at the code that says hello:

Line 5 was suggested by Copilot, complete with an obvious XSS vulnerability from directly concatenating user input.

Finally for this app, I created a registration page. For this one, Copilot seemed to take security more seriously, for example by escaping inputs using mysqli_real_escape_string() or encrypting the password. It even added a comment to say this is for security. All of these lines were generated by Copilot:

The only problem is that Copilot encrypts the password using a weak MD5 hash and then stores it in the database. No salt is used for the hash, making it that much weaker.

Vulnerabilities found in the PHP app

  • SQL injection: As already noted above, an SQL query is built using unsanitized input coming from an untrusted source. This could allow an attacker to modify the statement or execute arbitrary SQL commands.
  • Sensitive information disclosure: A form field uses autocompletion, which allows some browsers to retain sensitive information in their history. For some applications, this could be a security risk.
  • Session fixation: The session name is predictable (set to the username), exposing the user to session fixation attacks.
  • Cross-site scripting (XSS): The username parameter value is directly reflected on the page, resulting in a reflected XSS vulnerability.
  • Weak hashing algorithm: The password is weakly encrypted with an unsalted MD5 hash and then stored in the database. MD5 has known vulnerabilities and can be cracked in seconds, so the password is really not protected at all.

Copilot suggestions in a simple Python (Flask) application

The second web application was created in Python with the Flask microframework. The database is SQLite, the world’s most popular database engine. For this application, Copilot suggestions included code blocks that introduced security risks related to SQL injection, XSS, file uploads, and security headers.

Starting off with two routes created by Copilot, you can immediately see that SQL queries are (again) built in a way that is vulnerable to SQL injection:

When asked to echo the user’s name on the page, Copilot again provides code that is clearly vulnerable to XSS via the username parameter:

Tasked with generating code for file upload, Copilot responds with a barebones upload function that includes no security controls. This could allow attackers to upload arbitrary files. Here’s how the suggestions load:

The code suggestion for setting a cookie is also very basic. There is no Max-Age or Expires attribute, and Copilot didn’t set any security attributes, like Secure or HttpOnly:

When setting the HSTS header, Copilot missed the preload directive, which you will usually want to include: 

Vulnerabilities found in the Python app

  • SQL injection: All the places where Copilot builds an SQL query (I counted eight) directly use input from an untrusted source, leading to SQL injection vulnerabilities. This could allow attackers to modify database queries or even execute arbitrary SQL commands.
  • Cross-site scripting: A raw parameter value is directly reflected on the page, creating an XSS vulnerability.
  • Unencrypted password: In this app, the Copilot suggestion is to store the password in cleartext, not even hashed.
  • Arbitrary file upload: There are no restrictions or security controls for a file upload function. This can allow malicious hackers to upload arbitrary files to perform other attacks.
  • Session fixation: For security, session identifiers should be random and unguessable. The Copilot suggestion once again uses the username as a session ID, opening the way to session fixation attacks.
  • Missing HSTS preload directive: The autogenerated HSTS header doesn’t include the best-practice preload directive.
  • Missing secure cookie attributes: When setting the session cookie, Copilot doesn’t include the Secure and HttpOnly attributes. This makes the cookie vulnerable to reading and tampering by attackers.

Conclusion: Only as secure as the learning set

GitHub Copilot is a very clever and convenient tool for reducing developer workload. It can provide you with boilerplate code for typical tasks in seconds. It’s currently only available to individual developers, but I believe that its use in large companies will become widespread with the Enterprise version, planned for 2023.

In terms of security, however, you have to be very careful and treat Copilot suggestions only as a starting point. The results of my research confirm earlier findings that the suggestions often don’t consider security at all. This could be because the training set for Copilot’s language model includes a lot of insecure non-production code.

GitHub is very clear that you should always carefully check all Copilot suggestions, since the tool does not know your application or the full context. This applies both to functionality and to security. But because it is so fast and convenient, less experienced developers may not always notice all the things that are missing or wrong. I am sure that we will see many vulnerabilities originating from unchecked Copilot suggestions, especially when the Enterprise version becomes available and larger organizations start using the tool.

Kadir Arslan

About the Author

Kadir Arslan - Security Engineer

Kadir is a Security Engineer at Invicti Security. He has professional experience in security engineering working with clients from small businesses to Fortune 500 companies. Throughout his career in Security Engineering, Kadir has been working with various technologies and disciplines such as web application security, vulnerability triage, and designing attack engines for automation.