1.2 Import Data from Website

The knowledge base supports crawling content from public web pages using third-party tools such as Jina Reader and Firecrawl, parsing it into Markdown content, and importing it into the knowledge base.

​Firecrawl and Jina Reader are both open-source web parsing tools that can convert web pages into clean Markdown format text that is easy for LLMs to recognize, while providing easy-to-use API services.

The following sections will introduce the usage methods for Firecrawl and Jina Reader respectively.

Firecrawl

1. Configure Firecrawl API credentials

Click on the avatar in the upper right corner, then go to the DataSource page, and click the Configure button next to Firecrawl.

Configuring Firecrawl credentials

Log in to the Firecrawl website to complete registration, get your API Key, and then enter and save it in Dify.

Get the API Key and save it in Dify

2. Scrape target webpage

On the knowledge base creation page, select Sync from website, choose Firecrawl as the provider, and enter the target URL to be crawled.

The configuration options include: Whether to crawl sub-pages, Page crawling limit, Page scraping max depth, Excluded paths, Include only paths, and Content extraction scope. After completing the configuration, click Run to preview the parsed pages.

Execute scraping

3. Review import results

After importing the parsed text from the webpage, it is stored in the knowledge base documents. View the import results and click Add URL to continue importing new web pages.

---

Jina Reader

1. Configuring Jina Reader Credentials

Click on the avatar in the upper right corner, then go to the DataSource page, and click the Configure button next to Jina Reader.

Configuring Jina Reader

Log in to the Jina Reader website, complete registration, obtain the API Key, then fill it in and save.

2. Using Jina Reader to Crawl Web Content

On the knowledge base creation page, select Sync from website, choose Jina Reader as the provider, and enter the target URL to be crawled.

Web crawling configuration

Configuration options include: whether to crawl subpages, maximum number of pages to crawl, and whether to use sitemap for crawling. After completing the configuration, click the Run button to preview the page links to be crawled.

Executing the crawl process

After importing the parsed text from web pages into the knowledge base, you can review the imported results in the documents section. To add more web pages, click the Add URL button on the right to continue importing new pages.

Importing parsed web text into the knowledge base

After crawling is complete, the content from the web pages will be incorporated into the knowledge base.