Overview of AI Coding Tools Performance

Aug 31, 2024

Lecture Notes on AI Coding Tools

Introduction to Tab Nine

  • Similar to GitHub Copilot
  • Autocompletes code in IDEs (e.g., VS Code)
  • Includes AI chat feature

Example: Factorial Function

  • Autocompletes the factorial function without manual input.
  • Time complexity question:
    • Correctly stated as O(n) but incorrectly concluded as O(n^2).

Code Testing

  • Modified code to test error handling for negative inputs.
  • Tab Nine identified the base case condition as incorrect.
  • Provided partial credit for recognizing the base case issue.

Explanation and Comments

  • Correctly explained the factorial function, but struggle with comments.
  • Generated generic comments, lacking detail on inputs/outputs.

Test Case Generation

  • Generated appropriate test cases, e.g., factorial(5) returns 120.
  • Slight repetition in generated cases.

Merge Sort Function

  • Autocompleted but needed to manually prompt for the merge function.
  • Test case generation was limited.

AI Coding Tools Tier List

  • Tab Nine placed in the "decent" category.

ChatGPT

  • Well-known coding assistant.
  • Provides good extra context and conversation capability.
  • Context switching with code editor can be cumbersome.
  • Potential for incorrect information without citation.
  • Placed in the "actually useful" category.

GitHub Copilot

  • Offers inline AI chat and auto-completion.
  • Correctly identified improvements for handling negative numbers in code.
  • Generated exhaustive test cases using the Jest framework.
  • Placed in the "10x developer" category as the top tool.

Bard

  • Similar to ChatGPT, but cites sources.
  • Slightly less accurate than ChatGPT in tests.
  • Placed in the "actually useful" category.

Amazon CodeWhisper

  • Lacks a chat feature, similar to Copilot.
  • Generated correct outputs but struggled with comments and test cases.
  • Placed in the "acceptable" category.

Sourcegraph Cody

  • Claims to understand entire code repositories.
  • Performance in finding code logic was lacking.
  • Auto-completion capable but slow.
  • Placed in the "acceptable" category.

Codium AI

  • Focuses on testing and improving code rather than general completion.
  • Excellent at generating test suites and code suggestions.
  • Placed in the "10x developer" category as a top tool.

Conclusion

  • Suggested tools for better coding practice: ChatGPT, Codium AI, GitHub Copilot.
  • Recommended further learning through additional resources.