Interesting Content in AI, Software, Business, and Tech- 12/13/2023
Devansh Devansh
Chocolate Milk Cult Leader| Machine Learning Engineer| Writer | AI Researcher| | Computational Math, Data Science, Software Engineering, Computer Science
A lot of people reach out to me for reading recommendations. I figured I’d start sharing whatever AI Papers/Publications, interesting books, videos, etc I came across each week. Some will be technical, others not really. I will add whatever content I found really informative (and I remembered throughout the week). These won’t always be the most recent publications- just the ones I’m paying attention to this week. Without further ado, here are interesting readings/viewings for 12/13/2023. If you missed last week’s readings, you can find it here.
Reminder- We started an AI Made Simple Subreddit. Come join us over here- https://www.reddit.com/r/AIMadeSimple/. If you’d like to stay on top of community events and updates, join the discord for our cult here: https://discord.com/invite/EgrVtXSjYf.
Community Spotlight- Dylan Reid(Moskowitz)
Dylan Reid(Moskowitz) is working to bring the developments in AI Healthcare out to everyone. He puts a lot of effort into his AI healthcare reports, which are a great way to keep track of what's going on in DC. It's not super useful for engineers (unless you want to work in the space), but if you're a product person, entrepreneur, or have other interests in monitoring the field; Dylan is a worthwhile connection.
If you're doing interesting work and would like to be featured in the spotlight section, just drop your introduction in the comments/by reaching out to me. There are no rules- you could talk about a paper you've written, an interesting project you've worked on, some personal challenge you're working on, ask me to promote your company/product, or anything else you consider important. The goal is to get to know you better, and possibly connect you with interesting people in our chocolate milk cult. No costs/obligations are attached.
Highly Recommended
These are pieces that I feel are particularly well done. If you don't have much time, make sure you at least catch these works.
A great post on how ChatGPT works, by the exceptional Cameron R. Wolfe, Ph.D. All of his work is high-quality but this one made me laugh because he starts the post with "Looking for something to talk to your family about while you’re home for the holidays?". Made me imagine him lurking in the shadows, waiting for someone to bring up ChatGPT. Real talk though, he has great insights into AI, and I can't recommend his work enough.
"TL;DR: We can explain ChatGPT pretty easily by focusing on three core ideas.
1. Transformer architecture: the neural network architecture used by LLMs.
2. Language model pretraining: the (initial) training process used by LLMs.
3. The alignment process: how we teach LLMs to behave to our liking.
Although AI researchers might know these techniques well, it is important that we know how to explain them in simple terms as well! AI is no longer just a research topic, but rather a topic of public interest. "
I'm not American, but this series was still perspective-shifting. In his lecture series "Myths of the American Mind", philosopher Wes Cecil challenges many fundamental tenets of society (our ideas on smartness, money ...). The whole thing is worth a watch, but I particularly like this on scientism- where we try to make completely irrelevant things seem scientific to justify them.
"This lecture, presented by Wesley Cecil PhD. at Peninsula College, explores the way science has shaped our thinking and led to the peculiar and peculiarly misleading take on the world called "scientism". "
At the risk of sounding like a broken record, Ksenia Se is very good at producing instant classics. Given how we've established the importance of high-quality data for models, this becomes a must-read.
"In this Token, we will discuss the data requirements for a foundation model (FM), the effect of bias in datasets, and ways to mitigate it. We'll also explore how data is gathered to train FMs, introduce a few data efficient training techniques and touch upon the ethics of data. Foundation models are hungry models, and as they grow larger (and hungrier), we expect more discussions on data sourcing next year. Let’s catch up on what’s been happening with data for FMs so far!"
I've been pushed into researching a rabbit hole on gradients. A deep investigation will come soon, but meanwhile, check this out. Thank you to Ravindranath Nemani for his great recommendations on these topics.
"A counter-proposal, implicitly made by proponents of natural gradient, is that instead of defining our safety window in terms of distance in parameter space, we should define it in terms of distance in distribution space. So, instead of “I’ll follow my current gradient, subject to keeping the parameter vector within epsilon distance of the current vector,” you’d instead say “I’ll follow my current gradient, subject to keeping the distribution my model is predicting within epsilon distance of the distribution it was previously predicting”. The notion here is that distances between distributions are invariant to any scaling or shifting or general re-parameterizing. For example, the same Gaussian can be parameterized using either a variance parameter or a scale parameter (1/variance); if you looked in parameter space, two distributions would be different distances apart based on whether they were parameterized using variance or scale. But if you defined a distance in raw probabilities space, it would be consistent."
Speaking of someone with great recommendations, I have a love-hate relationship with Manny Ko 's paper suggestions. Every single recommendation of his has a lot of math that I've never heard of, and learning all of that makes me cry. And every single suggestion is too good to ignore. This one was no exception. I would have double the confidence and eight of the knowledge I have currently if not for Manny's suggestions, and I'm excited to share what I learn with y'all.
"This paper introduces an extension of the backpropagation algorithm that enables us to have layers with constrained weights in a deep network. In particular, we make use of the Riemannian geometry and optimization techniques on matrix manifolds to step outside of normal practice in training deep networks, equipping the network with structures such as orthogonality or positive definiteness. Based on our development, we make another contribution by introducing the Stiefel layer, a layer with orthogonal weights. Among various applications, Stiefel layers can be used to design orthogonal filter banks, perform dimensionality reduction and feature extraction. We demonstrate the benefits of having orthogonality in deep networks through a broad set of experiments, ranging from unsupervised feature learning to fine-grained image classification."
How do you stay cutting edge w/o being seduced by technobabble and investing in tech for the sake of it? Adam Haney has some very valuable advice.
In the dynamic realm of technology, change is the only constant. As the VP of Engineering at Invisible Technologies, I’ve learned that our success hinges not only on our ability to embrace new technologies but also on our capacity to do so strategically.
Engineers thrive on innovation, but the rapid evolution of tools and frameworks can sometimes lead to chaos. It’s a delicate balance between harnessing the power of the latest advancements and preventing them from becoming a Pandora’s box of unforeseen challenges.
领英推荐
Shopify has a great engineering blog. They managed to scale to 30 TB of traffic every minute w/ near perfect uptime. This explains how they held their payment processing together through all that. This also inspired me to write about idempotency here.
I"t’s hard to learn something when you don’t know what you don’t know. As I learned things over the years—sometimes the hard way—I eventually found myself passing on these lessons to others. I distilled these topics into a presentation I gave to my team and boiled that down into this blog post. So, without further ado, here are my top?10 tips and tricks for building resilient payment systems."
AI Content
"On December 6th, 2023, the House Oversight and Accountability Subcommittee on Cybersecurity, Information Technology, and Government Innovation held a hearing called “White House Policy on AI”. The topics that were discussed ranged from AI implementation to American innovation."
This is the most sane take on the whole faked demo fiasco. Marketing tends to overpromise, and Google doing this isn't that bad imo. Gen AI in general has been making exaggerations for a while, so I pretty much expect it at this moment. The bigger problem is the lack of information shared in technical documents, which makes assessing any possible claims/results much harder. A massive shoutout to Yannic Kilcher for keeping it real with his vids, and not turning into an attention-hungry jellyfish who just jumps on trends w/ needless dramabait.
Marina T Alamanou, PhD is a great resource to follow for biomedical AI. This was a great look into the process of biomedical development, and how AI can change the process.
"What is happening right now, between those two, is going to change forever the way we do research, by rewriting our SOPs (standard operating procedures) in the lab, by adding a brand new chapter in the “Materials and Methods” section, by creating a “before AI data” and “after AI data” PubMed section, by giving a new meaning to the statistical analysis while doing experiments, eventually tearing down the wall of scientific separatism."
Economics/Business Content
How Money Works is one of the best channels to understand the business side of corporate culture. Yes I recommend this channel a lot. There's a reason for it.
"There are nine points six MILLION [9,600,000} job openings in America right now and businesses are desperate to fill roles that have now been empty for months. BUT at the same time you are reading endless articles about people who have applied to thousands of jobs and haven’t heard anything back from the companies that were supposed to be desperate for workers. So if there are so many job opportunities, then why can’t anybody get a job? According to the U.S. Bureau of Labor Statistics there are currently nine point six million [9,600,000] job openings in America and only six point five million [6,500,000] unemployed people. So why not just take the openings that desperately need workers, and give them to the people that desperately need jobs?Well, it’s not that simple and there are two reasons why companies are struggling to fill roles even when so man people are looking for a job, and two equally important reasons why people can’t find a job even when so many companies are looking for workers."
A good look at an economy doing things differently.
"Most countries really want to bring in tourists for the big infusion of money into the local economy. But Bhutan makes it incredibly expensive to visit, and bans all visitors from climbing the parts of the Himalayan mountains that are within their borders. Why? Do they just want to protect their culture from tourists (with the hotels, photo spots and trouble that comes with them)?"
I regularly share mini-updates on what I read on the Microblogging sites X(https://twitter.com/Machine01776819), Threads(https://www.threads.net/@iseethings404), and TikTok(https://www.tiktok.com/@devansh_ai_made_simple)- so follow me there if you're interested in keeping up with my learnings.
Reach out to me
Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.
Check out my other articles on Medium. : https://rb.gy/zn1aiu
My YouTube: https://rb.gy/88iwdd
Reach out to me on LinkedIn. Let’s connect: https://rb.gy/m5ok2y
My Instagram: https://rb.gy/gmvuy9
My Twitter: https://twitter.com/Machine01776819
Government Affairs|Specialized in AI Healthcare|Health Policy and Tech
10 个月Devansh Devansh thank you for the mention!
Devansh Devansh many thanks for the mention
ML @ Netflix
10 个月You're not just imaging it. I am indeed lurking in the shadows, waiting to bring up ChatGPT at my family holiday gatherings. ??
Life Science Consultant ????????????????
10 个月Many thanks Devansh Devansh ??
Founder at Turing Post
10 个月Thank you, Devansh! Hugely appreciate the shoutout ??