Phase Relationships in the AI Regulation Process
Introduction
With the advent of general AI and the resulting increase in AI regulation, a pivotal question arises: How much regulation is necessary, and at what point should it be implemented, to guide optimal policy decisions? Should we allow technology to advance unchecked, risking potential negative consequences, or do we risk hindering innovation with stringent regulations that might not even fully mitigate the risks? Furthermore, how should we analyze these questions, both qualitatively and quantitatively?
Fortunately, wise individuals have previously pondered a similar problem, providing valuable insights applicable to our current AI regulatory challenges. James Gosling, the founder and lead designer behind the Java programming language, wrote a darkly humorous article in 1990 expressing a concern about the premature standardization of technology. His article, “Phase Relationships in the Standardization Process”, begins with the following preamble:
This is a moderately sarcastic note on the phases that the standardization process goes through, and the relationship between the level of technical and political interest in a topic. It is purely a personal view.
He could hardly have imagined how his work would be so relevant to our contemporary debate, 30 years later. In this hopefully no less sarcastic article, I intend to take a step further. I will apply the concept of phase relationships to AI regulation, employing quantitative analysis. Wouldn’t everyone be impressed and convinced if I wrap my arguments in charts and numbers and make it sound like a statistical proof?
If the reader lacks the time to review the original article, allow me to provide a summary. James Gosling plotted two curves on a single chart: one representing the interest of politicians in standardizing a specific technology, and the other the efforts invested in developing this technology. By integrating these two functions, the transformed chart illustrated the accumulated 'calcified' standards versus the technology’s accrued maturity. The ideal scenario occurs when a technology matures before significant standardization efforts are undertaken, thereby benefiting humanity and ensuring the technology reaches its full potential before 'calcification' sets in. Conversely, when standards are imposed prematurely, before the technology has fully matured, it hampers progress and ultimately proves detrimental to humanity. Reproduced with permission from the original article, these are typical scenarios:
In my analysis, I propose a hypothesis: the same methodology can generate insights into the relationship between AI development and the efforts to regulate AI. By examining actual data, we can draw meaningful conclusions from this analysis.
Data Collection
Quantitative analysis always begins with data collection. In data science, we often resort to using proxy metrics that sufficiently correlate with the phenomena we want to analyze when perfect data isn't available. As a proxy for the AI development efforts, I've chosen the number of monthly paper submissions to the open-access repository of scientific papers, Arxiv.org, labeled cs.ai. To assess the political momentum for AI regulation, I've averaged Google Trends data for specific terms: 'AI regulation' and 'Responsible AI,' measured globally on a monthly basis over the past decade.
Google Trends offers a web interface for manually downloading trend data in a comma-separated values (CSV) file. While Arxiv.org also allows for the manual download of certain submission statistics, cs.ai monthly statistics over the last decade cannot be easily accessed. Fortunately, Arxiv.org provides a user-friendly API. I opted to test whether the AI subject of my analysis could assist in downloading the AI data, and asked ChatGPT-4 to generate the Python code for downloading the data from Arxiv.org.
At the beginning, I assumed that the weekly data would be useful, so I gave this prompt to chatGPT:
Write me a code in python that calculates the number of articles submitted to arxiv.org? with cs.AI category, per week, for the last 5 years, using arxiv API. A week should be marked by its beginning date on Sunday.
The first result was missing the implementation of the article count, and I asked to write it in a follow up prompts by providing more information about the data format in the XML that Arxiv API returns:
Modify this code under the assumption that the API returns a valid XML document, where each article submission starts with <entry> and ends with </entry>, so create the correct count calculation based on this information.
I added pagination, improved the resulting file format, reduced the size of the downloaded XML, and switched to monthly counts with the following prompts. I examined and tested the results of every prompt before proceeding to the next prompt to make sure no errors keep accumulating:
Implement pagination to make sure more than 5000 results can also be captured correctly
Change the output to a comma-delimited format, with the first column to be the date (start_of_the_week) and submission count, and a comma in between
How to hide abstracts in the end results, to make the script faster, since I dont need abstracts anyway
领英推荐
Change this code to downloading number of article submissions per month, for the last 10 years
I noticed that the range ended with the first day of the next month. The API treats the range as “inclusive”, leading to double-counting the first day of each month. I had to fix this bug myself, because chatGPT seemed to have difficulties responding to my prompts. The code can be downloaded from my GitHub here, along with the datasets.
The process of writing this code with chatGPT highlighted an important issue that should serve as a cautionary note to anyone who thinks chatGPT can create perfect code, it cannot, or it will soon take over humans. But it can help an experienced developer to reduce googling, creating scaffolding of the application, similar to how Copilot works in GitHub.
Analysis of the data
I proceeded with the data analysis of three time series, two from Google Trends, and one from Arxiv, thanks to my AI sidekick. First, I prorated the November article submissions, and limited the dataset to the past 10 years, starting from 11/2013. Then I integrated the time series by adding each month’s data to a cumulative total. Unlike James Gosling’s original phase relationships, which span from zero to full technological maturity, our dataset is inherently incomplete as the time machine has not been invented, yet. Additionally, our data differs in other crucial aspects:
The chart surprisingly displays the best case scenario as described in the original phase relationships framework, with a cross-road: whether we will calcify the underdeveloped technology or allow it to mature without excessive regulatory burden. The data strongly suggests that we are approaching an era where regulation will outstrip technology progress, potentially leading to adverse effects.
Conclusion
The reality that this approach, comparing regulatory activity with technological progress, was initially proposed over 30 years ago — a time when AI was largely confined to the realm of science fiction — yet still holds relevance in today's discussions on AI regulation, is both 'tragically hilarious and yet painfully accurate,' in the words of James Gosling himself. What can we conclude from our analysis?
-???? It is easier to regulate than to innovate. Our chart is clear: the knowledge growth, measured by human contribution to science, cannot scale as well as the attention of politicians when they discover a new focus of interest, a depressing observation.
-???? Regulatory “calcification” is real. Once ‘calcification’ sets in, companies are likely to shy away from deploying solutions that deviate from well-regulated and compliant AI systems. AI regulation will undoubtedly increase the costs of developing, deploying, and maintaining AI systems, favoring larger companies with ample resources.
-???? Caution is warranted. The pessimist in me fears that regulating such complex systems as AI might not yield improved outcomes. Cybersecurity serves as a good example, where regulation has not evidently reduced the likelihood of cybersecurity incidents and their negative impacts on businesses and the data privacy of individuals.
-???? Board-driven oversight is a double-edged sword. Recent events at OpenAI demonstrate that boards can play both a positive role in oversight and, often, a role that impedes innovation, even to the point of being destructive. The human mind is most creative when it is not subjected to political maneuvering, and many brilliant and creative scientists do not believe in authority, regardless of its legitimacy. Managing innovation and talent is challenging, yet possible.
-???? Optimism wins, after all. The optimist in me hopes that AI regulation will raise awareness about best practices, thereby reducing risks associated with building and operating AI systems, while we continue to reap their benefits.
I would like to conclude this article by illustrating how AI (DALL-E) visualizes the negative effects of regulatory calcification: a vibrant ecosystem around a coral reef versus a gloomy place where technological progress does not yield societal benefits due to overregulation. We in the AI community, from scientists to engineers to lawyers, should strive to drive AGI development and regulation in a way that not only brings the benefits of AI to humanity in a transformative and responsible manner but also continues to foster innovation.
References