Multiple researchers using the same tools to find the same bugs are creating ‘unnecessary pain and pointless work’
You must log in or # to comment.
I’ve found that the LLMs tend to over classify and nitpick a fair bit, often missing broader context that accounts for the flaw being tolerated or undiscovered.
They’re not wrong, but have no context for triage and so give far too many results. It forces you to consider an LLM subscription yourself just to keep up with the other LLM users which is starting to feel like some form of zero sum red queen’s race.
The tsunami of reports won’t be receding for a while yet, and we can only hope the teams on the receiving end don’t drown in it.
Reminder: if you can put code in a chatbot and get it to find bugs, the devs can do it too. As such, even if your “LLM bug finding trip” works, it’s still useless, and a waste of everyone else’s time.
Is it really a waste of time if the computer can find the bugs, and the dev can focus on fixing them instead of pulling double duty?
It is because everyone else’s computer is finding that same bug, and they’re all reporting it separately, and reduplicating efforts associated with trialling the bugs. It’s clearly more costly than having the dev themself fire a spare machine and have their computer alone find bugs.
The human will still need to check and understand the bug.
Particularly if a given bug is reported dozens of times in slight variations, someone will have to check for each of these reports whether it’s a bug already reported earlier. If they’re all checked by the same person, that person may quickly recognise “Okay, yeah, that’s the same thing I logged earlier” but if it’s multiple people, there’s just so much extra overhead associated with keeping track of what’s new and what’s “30th reported of same bug”.
The AI generating the report probably doesn’t check if the issue hss been found before. If the people subvmitting it also don’t, you end up with a load jof chaff by lazy people thinking they’re helpful when really they’re obstructing the efforts.
Welcome to AI
It must suck because all chatbot output I seen speaks like a corporate email, just can’t get to the point.
one of the big reasons i don’t understand how people can use anything other than basic search assist stuff, that’s at least directly rephrasing stuff like wikipedia and thus unlikely to ramble.
Seriously, I can’t stand talking to Chat GPT
You don’t say
I’ll say it till I turn blue in the face. Unless you’ve been verified as a trusted source, there should be a small donation required for submitting any type of “help” to a project. (pull requests, bug bounties, etc.) Especially since it always requires humans take time out of their day to review the issues and code changes.
If you love the project, donate to it. If you’re a trusted source or can’t afford even a small donation, get verified.
that entirely depends on the size of the project though, i’m not gonna demand donations for people to contribute to my fucking modpack with 1000 downloads…
I mean, thats exactly my point. If the donation money wouldn’t pull you out of a hole that has you feeling like you’re stuck in an inescapable nightmare, this option isn’t for you.
Yeah, there’s a ton of spam now. My view is that devs should use LLMs themselves to scan for issues, and then see if there’s anything to fix. But when it comes to accepting reports or patches, you kind of have to be selective. A lot of the time stuff LLMs will flag can be either hallucinated, or not really an issue. A lot of the reports come from automated systems that don’t really do any due diligence to figure out if something is an actual issue that needs addressing. So, I can definitely understand why projects might want to stop accepting random bug reports or code submissions going forward.
Fully automated systems that file issues sound like a nightmare. I hope it’s easy to ban those as they appear.
Yeah, honestly that’s the dumbest thing anybody could think of. It’s just a pure waste of resources that wastes people’s time. Even if these systems find genuine issues, the sheer volume of spam ensures nobody is going to actually look at them.
I figured you’d probably be sympathetic.
The next few years are going to be interesting because we’re moving into uncharted territory in a lot of ways. There’s a ton of hype around LLMs, and tons of people abusing this tech in every which way, and then there are useful nuggets where people figure out how to apply it effectively. Eventually we’ll need to figure out how to suppress the noise and how to start using these things in productive ways.
deleted by creator
