Open Source Crawler in JAVA

-
In case you have the need to crawl the web to get information. I would suggest few:

  1. Smart and Simple Web Crawler - https://crawler.dev.java.net/
  2. Websphnix - http://www.cs.cmu.edu/~rcm/websphinx/
  3. Archive-Crawler - http://archive-crawler.sourceforge.net
Of these i would recommend https://crawler.dev.java.net/
Why? because I have used it.

This is a quite simple crawler which serves the purpose. It loads the page and parses the page.
Throws appropriate events while crawling. (i.e supports event model to serve the requirement).

It has HTML and HTTP parsing capabilities.

Comments

Post a Comment

Popular posts from this blog

Hibernate: a different object with the same identifier value was already associated with the session

-
People working with Hibernate might be familiar with the below error. org.springframework.orm.hibernate3.HibernateSystemException: a different object with the same identifier value was already associated with the session: nested exception is org.hibernate.NonUniqueObjectException: a different object with the same identifier value was already associated with the session This exception is generated when we try to load an instance object in session which has been already loaded by hibernate. Easier said than done. Hibernate considers two persistent entities identical if the two rows of the table can be said identical. Above error could be generated in many ways and there may be various solutions on net like using session merge etc to get away with the problem. I will discuss here the reason for me getting this problem and the way I resolved it. Below is the code which generated the exception for me. Following is the method which caused the exception @Transactional(propagation=Propagation.
Read more

BeanDefinitionStoreException: Failed to parse configuration class: Could not find class [javax.jms.ConnectionFactory]

-
If you are creating a Spring Boot application and see the following error stack . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.2.0.RELEASE) 2020-06-27 19:30:32.786 INFO 35224 --- [ main] SchoolApplication : Starting SchoolApplication on LTUS76179 with PID 35224 (C:\personal\eclipse-workspace\apievolution\target\classes started by 381548 in C:\personal\eclipse-workspace\apievolution) 2020-06-27 19:30:32.796 INFO 35224 --- [ main] SchoolApplication : No active profile set, falling back to default profiles: default 2020-06-27 19:30:32.964 WARN 35224 --- [ main] ionWarningsApplicationContextInitializer : ** WARNING ** : Your ApplicationContext is unli
Read more