Mike Stackhouse的动态

?? datasetjson 0.3.0 is now on CRAN! ?? This update brings major improvements, including Dataset JSON v1.1.0 support and 70% faster write times. For details, check out the post below from Atorus CIO, Mike Stackhouse. #RStats #ClinicalData #DatasetJSON

CSV Loader in RAG We load CSV data using langchain <------ Code ------> !pip install langchain from langchain_community.document_loaders.csv_loader import CSVLoader loader = CSVLoader(file_path = 'Your file path') data = loader.load() data /* customizing the csv parsing and loading *\ loader = CSVLoader (file_path='Your file path', csv_args={ ? ? 'delimiter': '/', ? ? 'quotechar': '"' }) data = loader.load() data /* specify a column to identify the document source *\ loader = CSVLoader(file_path='Your file path', source_column='MSZoning') data = loader.load() data check colab notebook link ---> https://lnkd.in/dmyf-hcs Muhammad Irfan, Dr. Sheraz Naseer - (PhD Artificial Intelligence, Data Science), Muhammad Haris Tariq, Xeven Solutions #AI #RAG #ML #DL #NL

???? ??????? ??????? ?????????????????? ???????? ?????????????????? ???? ???????????? i am excited to share that i have completed ???????? ???????????????????????? ???????? ???????????? ???????????? ??/???? on DataCamp. In this Course chapter 1: i have Learn how to inspect DataFrames and perform fundamental manipulations, including sorting rows, subsetting, and adding new columns. chapter 2: calculate summary statistics on DataFrame columns, and master grouped summary statistics and pivot tables. chapter 3: how indexes can be combined with slicing for powerful DataFrame subsetting. chapter 4: Learn to visualize the contents of your DataFrames, handle missing data values, and import data from and export data to CSV files.

The quality and usefulness of open source projects lately is impressive. Using NLSQLTableQueryEngine from the llamaindex project reduces RAG to a few lines of code and works amazingly well for structured data. Using VectorStore index does the same for unstructured data. Both allow the engineer to focus on the problem at hand rather than the minutia. https://lnkd.in/eRKj-YvQ

?????? ???? ????????????? Tanzim and I were working on a script to convert a large number of CSV files to raster data. It is a pretty simple task. However, if your dataset consists of thousand of CSV files, you can't do that manually. It will take weeks. And if you plan to do that often, you need an automated way. ???????????????? 1 ??We used Pandas to read the CSV files. ??Next, we used Shapely to convert the coordinates stored in the CSV files into point features. ??Using GeoPandas, we created a GeoDataFrame of the points and saved them as shapefiles on the hard disk. ??Then, we used ArcPy to convert the points into raster data. ??The final raster datasets were not merged. ????????????: It worked perfectly. However, it was taking a huge amount of time to export the points (from GeoDataframe) as shapefile because each csv file had nearly 1.2 ?????????????? ??????????????????????. ???????????????? 2 ??We used Pandas to read the CSV files and added headers. ??Instead of converting the coordinates into points, we used ArcPy to virtually load the coordinates into the map canvas. ??We then used ArcPy again to convert the virtually loaded points into raster data. ??Finally, we used GDAL to merge all the rasters into a single raster file. ????????????:?The script was able to make a single raster by combining 784 csv files in about 3 hours. ???????? ???? ???????????? 950 ?????????????? ????????????. ???????????????????? ??Approach 1 was output bound. It would take 2-3 days for the script to generate desired output. For each csv file, it was writing a large amount of data to the hard disk. Therefore, it was taking a lot of time. ??Approach 2 is ?????? ?????????????? ?????? ???????? ???? ???????? ???????? at the time of processing the data. It was loading the coordinates in the system memory, generating raster, and writing the final raster to hard disk. ?????? ????????????????: ???? ????????????????????, ???????????????? ????????????????, ?????????? ?????????????? ???????????????????????? ???????? ???? ?????? ???????? ????????. The attached video is showing a demo with just two files. #gis #gisprogramming #arcpy #geopandas #shapely #gdal #automation

PremSQL End-to-End Local-First Text-to-SQL Pipelines PremSQL is an open-source library designed to help developers create secure, fully local Text-to-SQL solutions using small language models. It provides all the essential tools to build and deploy end-to-end Text-to-SQL pipelines with customizable components, making it ideal for secure, autonomous AI-powered data analysis. https://lnkd.in/gRBSMzei #TextToSQL #LanguageModels #LocalFirst #AIpowered #DataAnalysis #OpenSource #NaturalLanguageProcessing #MachineLearning #DataScience

Is there a limit on JSON data returned from an API performing ML analysis on large CSV files with 440 genes per sample? ? #JSON #API #DataAnalysis #MachineLearning Hey everyone! ?? I've been working on an API that processes large CSV files with a ton of data points - specifically 440 genes per sample. After running some machine learning algorithms and data analysis, each sample could potentially return up to 460 data points in JSON format. ?? So, here's the burning question: How much JSON is too much to return fro... Source: https://lnkd.in/gZJ-65it #mymetric360

Great post by Vishal Waghmode on how to make data quality rules reusable across multiple Delta Live Tables pipelines! This makes the life of data engineers a lot easier in managing these data quality rules and applying them across pipelines and delivering quality data products that business can actually trust. Add Lakehouse Monitoring on top of this to monitor and create alerts to create a robust framework with data governance and quality top of mind with Databricks! #databricks #dataquality #dataengineering

We had a fun side quest processing Medicare Part D information for multiple consumers - a case study on leveraging DuckDB to produce enriched parquet files for data structure and compression. With a few light mods it could also be a solid use case for polyglot notebooks. And it's also a great way to make "flat files" available within editors as a SQL-like data source - really handy! #dotnet #fsharp #dataengineering #machinelearning #AI https://lnkd.in/gBJbjHus

Last week, our engineering team built a custom dbt alerting tool that enriches alerts with actionable context. But how does it work, and how could you build something similar? Here are 4 of the most interesting challenges we tackled, along with how we solved them: ??. ???????????????????? ??????-?????????? ?????????? ?????????? ?????? ?????? ?????? ?????????????? Solution: Use a specifically named exposure that stores custom configuration data. This supports a wide range of dbt setups, avoids users having to update the ???????? property of each model, and does not require access to underlying git repos. ??. ???????????????? ???????????? ?????????? ?????????????????? ?????????? The most expressive yet intuitive filters for us are database path matching, tag-based matching, and owner-based matching. These cover the three (mostly) orthogonal directions of data structure, domain knowledge, and responsibility. ??. ???????????????? ??????????-???????? ?????????????????????????? Upon each test failure, we evaluate rules against both the test itself and the actual parent models that the tests are applied to, by traversing the many-to-one relationship on each failure. ??. ???????????? ???????? ???????????????? ???????? ???? ???????????? We created a macro to download failing rows captured by dbt via ??????????_???????????????? to a Snowflake stage and attach a link directly to dbt run results. You can learn more about the thought process and details here at the link in the comments. But of course you can also use this tool for free too – our setup is not unique, so we figured that if it’s useful for us, then it’ll hopefully be useful for other folks. Would love to hear your feedback, especially if you’ve customized dbt alerts yourself! #dataengineering #dbt #analytics