EasySpider is a completely free and open source visual crawler software. This software allows everyone to use a graphical interface, design and execute crawler tasks visually without code.

You only need to select the content you want to crawl on the webpage and operate according to the prompt box to complete the crawler design and execution. At the same time, the software can also be executed directly in the command line by passing parameters, so that it can be easily embedded into other systems.

New features in V0.3.0, including downloading pictures, screenshots of elements,Execute arbitrary JS instructions and system commands,Conditional judgment through JS code, OCR recognition and other functions, all the functions you want, and these functionsIt’s completely free! ! !

The following is a sample interface:

Download EasySpider

Go to the Releases Page:https://github.com/NaiboWang/EasySpider/releases Download the latest version.

video tutorial

1. Introduction to EasySpider – China Seismic Network Acquisition Case:https://www.bilibili.com/video/BV1Fk4y1L7xX/

2. How to visually crawl websites that require login to crawl without code – Zhihu website case:https://www.bilibili.com/video/BV1HV4y1r7v8

3.[Important]Use the JS command return value in the loop item for custom condition judgment:https://www.bilibili.com/video/BV1mu411x7Nn/

4. Flowchart execution logic analysis – 58 city housing listings collection case:https://www.bilibili.com/video/BV1YL411z7uW

5. MacOS system design and implementation of eBay website crawler task tutorial:https://www.bilibili.com/video/BV1WL411h71r

6. How to execute self-written JS code and system code (custom operation):https://www.bilibili.com/video/BV1qs4y1z7Hc/

7. How to customize the loop and judgment conditions – the first bullet:https://www.bilibili.com/video/BV1Ys4y1z777/

8. How to take screenshots of elements and webpages and command line execution guides:https://www.bilibili.com/video/BV1dV4y1z764/

9. OCR recognition element content function:https://www.bilibili.com/video/BV1xz4y1b72D/

10. How to crawl the website that needs to enter the verification code:https://www.bilibili.com/video/BV18c411K7FH

11. How to switch IP pool and use tunnel IP – Open the details page to collect cases:https://www.bilibili.com/video/BV1KT411t79n

document

Please click here to enterTutorial Documentationif you have English, you can temporarily translate it, or see the author’sMaster’s thesis(mainly see Chapters 3 and 5).

Documentation can be found from GitHub Wiki.

Why use EasySpider

Compared with other visual crawler software, EasySpider has the following advantages:

1. The code is open source, so secondary development is possible.

2. It is completely free, unlike the “free” software such as Octopus, EasySpider is a software that requires no login, unlimited multi-opening, and unlimited machine deployment, and does not need to pay a penny to the author himself. (Of course, EasySpider is protected by patents, so if you want to use it commercially, please contact Tiandao Patent Office of Zhejiang University). In contrast, other free software has many restrictions, you can see their price details page for details.

3. Safe, all information is completely stored locally on the user, including tasks and collected data, so there is no need to worry about data leakage.

4. Cross-platform: supports Windows, Linux and MacOS at the same time.

5. The speed is fast. Usually, a crawler task can be designed and completed in only 2-5 minutes, and the collection speed is also fast, which usually depends on the specific machine environment.

6. It is more flexible and saves more browser configuration information. The most important thing is that it can be expanded and various plug-ins can be installed freely, such as verification code recognition plug-ins.

7. It can be executed directly through the command line, and can be deployed infinitely in any desired machine.

8. You can execute custom instructions in the task process, including JavaScript instructions and system-level instructions, which is something that all current visual crawler software cannot or is unwilling to do.

9. The new functions of V0.3.0 version, including element screenshots, execution of arbitrary JS instructions and system commands, OCR recognition, etc., have all the functions you want, and these functionsIt’s completely free! ! !

From a demand-oriented point of view, crawlers are a basic need. We often need to crawl some online information. For example, for scientific researchers, crawling Wikipedia corpus for training is what students who do NLP often do; doing social networking Students who analyze often need to crawl information from Twitter and Weibo; students who work on recommendation systems will crawl information from shopping websites and so on. There are a lot of reptiles in the market, so I won’t go into details here. With EasySpider, no matter whether you know how to write crawlers before, you can now write code without bothering.