ChatGPT v4 aces the bar, SATs and can identify exploits in ETH contracts

GPT-4 completed many of the tests within the top 10% of the cohort, while the original version of ChatGPT often finished up in the bottom 10%.

GPT-4, the latest version of the artificial intelligence chatbot ChatGPT, can pass high school tests and law school exams with scores ranking in the 90th percentile and has new processing capabilities that were not possible with the prior version.

The figures from GPT-4’s test scores were shared on March 14 by creator OpenAI, revealing it can also convert image, audio and video inputs to text in addition to handling “much more nuanced instructions” more creatively and reliably.

“It passes a simulated bar exam with a score around the top 10% of test takers,” OpenAI added. “In contrast, GPT-3.5’s score was around the bottom 10%.”

The figures show that GPT-4 achieved a score of 163 in the 88th percentile on the LSAT exam — the test college students need to pass in the United States to be admitted into law school.

*Exam results of GPT-4 and GPT-3.5 on a range of recent U.S. exams. Source:* *OpenAI*

GPT4’s score would put it in a good position to be admitted into a top 20 law school and is only a few marks short of the reported scores needed for acceptance to prestigious schools such as Harvard, Stanford, Princeton or Yale.

The prior version of ChatGPT only scored 149 on the LSAT, putting it in the bottom 40%.

GPT-4 also scored 298 out of 400 in the Uniform Bar Exam — a test undertaken by recently graduated law students permitting them to practice as a lawyer in any U.S. jurisdiction.

*UBE scores needed to be admitted to practice law in each U.S. jurisdiction. Source:* *National Conference of Bar Examiners*

The old version of ChatGPT struggled in this test, finishing in the bottom 10% with a score of 213 out of 400.

As for the SAT Evidence-Based Reading & Writing and SAT Math exams taken by U.S. high school students to measure their college readiness, GPT-4 scored in the 93rd and 89th percentile, respectively.

GPT-4 excelled in the “hard” sciences too, posting well above average percentile scores in AP Biology (85-100%), Chemistry (71-88%) and Physics 2 (66-84%).

However its AP Calculus score was fairly average, ranking in the 43rd to 59th percentile.

Another area where GPT-4 was lacking was in English literature exams, posting scores in the 8th to 44th percentile across two separate tests.

OpenAI said GPT-4 and GPT-3.5 took these tests from the 2022-2023 practice exams, and that “no specific training” was taken by the language processing tools:

“We did no specific training for these exams. A minority of the problems in the exams were seen by the model during training, but we believe the results to be representative.”

The results prompted fear in the Twitter community too.

Nick Almond, the founder of FactoryDAO, told his 14,300 Twitter followers on March 14 that GPT4 is going to “scare people” and it will “collapse” the global education system.

Assessment theory was a big chunk of my life for several years. I was banging on about this day coming many years ago. I literally sounded like the resident crank at the time.

But… really this means that anything but invigilated assessment is over from this point on.
— drnick ️² (@DrNickA) March 14, 2023

Former Coinbase director Conor Grogan said he inserted a live Ethereum smart contract into GPT-4, and the chatbot instantly pointed to several “security vulnerabilities” and outlined how the code mighbe exploited:

I dumped a live Ethereum contract into GPT-4.

In an instant, it highlighted a number of security vulnerabilities and pointed out surface areas where the contract could be exploited. It then verified a specific way I could exploit the contract pic.twitter.com/its5puakUW
— Conor (@jconorgrogan) March 14, 2023

Earlier smart contract audits on ChatGPT found that its first version was also capable at spotting out code bugs to a reasonable degree as well.

Rowan Cheung, the founder of the AI newsletter The Rundown, shared a video of GPT transcribing a hand-drawn fake website on a piece of paper into code.

I just watched GPT-4 turn a hand-drawn sketch into a functional website.

This is insane. pic.twitter.com/P5nSjrk7Wn
— Rowan Cheung (@rowancheung) March 14, 2023

Many NFT projects lack adequate smart contract testing, says nameless founder

Coin Telegraph

Last year, the Akutars NFT collection sold out 15,000 tokens, but a major big saw $33 million worth of Ether (ETH) generated from sales locked into an inaccessible smart contract.

Jimmy McNelis, the founder of Web3 tech firm nameless, says there are too many NFT projects rushing to market without proper smart contract testing — potentially leading to millions lost.

Speaking with Cointelegraph, McNelis suggested that a lot of NFT projects often rush to market without fully simulating how its smart contracts will work, even skipping extensive audits in some cases.

McNelis said an example of this was observed during the sale of the Akutars NFT collection in February 2021 — featuring 15,000 tokens that went up for sale on Winklevoss-owned NFT marketplace Nifty Gateway.

McNelis said while the NFT drop sold out, a major bug saw $33 million worth of Ether (ETH) generated from the sale locked up in a smart contract that the devs have no access to, explaining:

“That was the sort of thing that they could have tested more completely in a private test environment and run the tests against those sales and edge cases, that they may or may not have taken the time to do or thought to do on a public testnet.”

McNelis emphasized the importance of getting the test phase right, given that smart contract bugs can’t be patched post-launch:

“The testing phase of a project is extremely critical because it's going to determine really the success of your drop or launch as far as the technical and marketplace solutions go.”

McNelis explained that while projects can use public test nets to conduct trials for networks like Ethereum, many don’t as it could open the door for copycat scam projects. He also says that some don't want to test in public environments of the lack of confidentiality.

“The other thing is there's a lot of brands that may be wanting to explore the Web3 space but aren't ready to announce publicly that they're doing so.”

Nameless was founded by McNelis in mid-2021, and the project has so far received backing from popular entrepreneur and NFT proponent Gary Vaynerchuck among others.

It is gearing up for a new product launch later this month with an NFT software called StealthTest, which provides private testnets for devs to trial smart contracts for Ethereum, IPFS, and Arweave.

Commenting on the NFT market, McNelis expects big-name companies to continue to pile into the space with their own tokenized products, and for organic retail interest to continue to increase.

He did note that in terms of investments, it's still too early for the big financial firms to want to speculate on NFT themselves.

“I think institutions are still going to be primarily focused on producing things like that. But some of the braver ones may speculate into some NFTs, but I don't think that NFTs are mature enough yet and the markets are mature enough yet to make safe long-term investments,” he said.

Smart contract bugs