EXECUTIVE SUITE – STRATEGY DISCUSSION POST-LUNCH

Today, the team will be exploring ScrapeGraphAI, a powerful Python LLM based web scraping package.

I'm not sure if web scraping is really necessary for our business. Can't we just gather data manually?
Also, I've had trouble scraping data from them in the past.

ScrapeGraphAI is an innovative Python library that revolutionizes web scraping by integrating Large Language Models (LLMs) and modular graph-based pipelines. This powerful tool simplifies the extraction of structured data from websites and local documents, allowing users to effortlessly gather information with just a simple prompt. By leveraging the capabilities of LLMs, ScrapeGraphAI can understand complex data structures and extract relevant information without the need for manual parsing or intricate rule-based systems.

The library offers various specialized graph classes, such as SmartScraperGraph for single-page scraping, SearchGraph for multi-page extraction from search results, and ScriptCreatorGraph for generating custom scraping scripts. ScrapeGraphAI supports multiple LLM providers, including OpenAI, Groq, and Azure, as well as local models through Ollama, providing flexibility in choosing the most suitable AI backend for your scraping needs.

Web Scraping using LLMs

 Hey, welcome, and in this video we're going to have a look at a Python package called Scrape AI. Scrape AI is a great tool that allows you to automatically extract structured data directly from images, web pages. So the first thing that we're going to do is install a different couple of packages. So Scrape AI, we're also going to look at some lang chain stuff, and then we're going to run nestasyncio because we're currently working in a Jupyter notebook.

You will need to enter your OpenAI API key to run this code. And we're just going to run that apply function for any asynchronous jobs that also will run. Now the basic usage of this kind of tool is something called the smart scraper graph. And what you can do from this is you can specifically provide your API key.

You provide a large language model that you'd like to use. And then you have a prompt so get me all of the services on this page with their descriptions And you want to put a source here in the smart scraper graph There's a couple of different inputs that you can have for the attributes The ones that we are focusing on at the moment is the prompt and the source of where that graph will look And also notice you've got this configuration, which is your graph configuration So that can change the configuration of this now You'll see that when we run this and i'll just scroll back up so you can have a look at the results  Let's have a look at this.

If I print this result, you'll see we've got back the services of this webpage, it's got the name, description and if we go and have a look at this webpage, you can see it is actually accurately being able to extract like these various descriptions and names of the services that I offer on my website.

So that's really great.  Now, the one thing that can be an improvement upon this, which is basically the same as before, is we can hook this up with LangChainCore, and we can use a Pydantic model. Now, if you don't know what a Pydantic model is, it's basically a  model that you can use to add on types,  and this allows you to serialize and deserialize data with some degree of data validation so you can test to make sure that The string that you're that you're actually going to deserialize into an object has these relevant fields.

So in my case, I wanted to know that there's a name, which is a type of string, and also a description, which has a type of string. And I've created another model here, and this is typical where you'll have a plural model. So you've got services, and we've got the key of that, and it's a list of these service schema models.

And so all we have to do is just provide on the prompt here what the schema is that you'll let back. And so let's give this a run and see what happens. So when this run comes back, you will still get a string, but it will be in the format. So if we actually go and have a look at this result, you can see you've got actually it should be a Python dictionary, yes, you've got a Python dictionary but you can see that when we try and extract that. 

In this scenario it has actually been able to extract that so we actually have a look we can do that model for example dot services And you can see we've got all these objects there So I can for example go into the first one and I could get the name of that so this is a great way for you to easily be able to run a specific graph  on a source of document and that will allow you to basically extract that schema directly out from that web page.

The other thing that can be good is you can also use this same Python package with local models. So if you have, for example, Olama you can see this is using Olama Mistral and here's the base URL and you can also use some open source embeddings too. The other thing that I think that's quite interesting is you can ask it to search on the web so you can say list the traditional recipes.

So the Key point here is that we don't have a source. And if there isn't a source here, then this search graph now is actually slightly different to what we have before. So if we scroll up, we were using a smart scraper graph and here we're actually using a search graph and this will be finding specific results.

From from Google. And you can combine this to actually also have a proxy so that you can do this at scale too. Now, one thing that's also quite interesting is there's a different type of graph called an Omni Scraper Graph. And this basically allows you to extract stuff from web pages.

So just like the first one we were showing you.  But the interesting thing is you can also analyze the images with this one.  So you can say, here's the LLM. We're going to use GPT 4. 0 with this API key and we want to analyze up to five images. You've got the prompt here. So describe the main projects and the associated image.

And you've got the source here as well of what we're analyzing with that graph configuration coming in as the config key. And then we just do the omni scrapers or whatever your graph is called. And then we just do the dot run and then we'll get an output as well here. And you can see when we get this result back, if we actually have a look at this we have a bunch of projects, and we've got the name, the description, and we've got the image.

So it's actually able to get these images here and it's also been able to analyze these images as well. So that, that's really cool. I think that's really interesting that you can analyze these images. Now the other use case that you might want to be using this for is to actually generate scripts.

And I find this quite an interesting use case and it's great for the fact that, websites do change, but they don't change all the time. So if we can generate Python scripts that allow us to scrape a website, then that means that you could reuse that same code until that code no longer works and is broken, and then you could regenerate a script.

Which would then fix the problem of your original pipeline. This really is going to add on another layer of flexibility for your data pipelines and your web scraping pipelines. So you can have a flow that looks like this. Okay, so if we're looking at this flow, the first bit would be we would get from the database the existing web script.

We try and run that existing web scraping script. If it fails, what we're going to do is generate a new script. And then after we've generated that new script, we'll then save it the results of to the database And we'd also save the new script back and overwrite the existing web scraping script But if it succeeds, then all we have to do is save the results to that database, right?

So you can imagine that we don't always have to use the large language model to extract Every single time we're almost trying an existing web scraping script first And if that fails, then we can generate a new web scraping script and save those results to the database. So there's a lot of cost saving efforts that can be gone into setting up a pipeline that looks like this. 

So how it works is you basically have this script creator graph, and you have whatever library you want to use, so in this scenario we've decided to choose Beautiful Soup. And you create the script creator graph, and for example we've got the prompt here, so list me all the projects with their description, you've got the source, And you've got the graph config now when we run this What you'll see is we're also getting some execution info.

So this returns some, Cost savings about the time now. This is the script that it decided to use. So python you got beautiful soup You've got the main here That's being called. So this is the script And then you've got the amount of time it costs, and the various costs for that so I think this is a really interesting thing to use and I can see this being very useful for large web scraping projects where you have lots of different customizable sites and you have to be able to avoid doing that by hand.

Now it's also possible to customize your graphs behavior. There's lots of different things you can do. So should you open a web browser when fetching HTML specific websites and view or react or spell any single page application might require you to run with a headless browser. You've also got, debugging information and you've also got a web proxy that you can use specifically for for getting around rate limiting.

And also you can customize things like the LLM. It's also possible to customize the embedding models as well. So you can run this as a completely open source piece of software. So hopefully you enjoyed this introduction into Scraper Graph AI. 

What is ScrapeGraphAI and what is its purpose?

How does ScrapeGraphAI handle web scraping with images?

What is the purpose of the SmartScraperGraph class in ScrapeGraphAI?

How can you customize the behavior of ScrapeGraphAI graphs?

How can ScrapeGraphAI be used with local models?

Master the art of intelligent web scraping using Large Language Models (LLMs) and ScrapeGraphAI. Learn to extract structured data from websites using natural language prompts, work with graph-based pipelines, and handle multi-modal content. 

Vexpower Learning Certificate of Completion

Congratulations Juveria Dalvi 💪