Screen Scraping using Java

1. Here we have to use the basic package

com.gargoylesoftware.htmlunit;

2. How to get a page using com.gargoylesoftware.htmlunit.WebClient

WebClient webClient = new WebClient();
webClient.setThrowExceptionOnFailingStatusCode(false);
webClient.setThrowExceptionOnScriptError(false);
webClient.setJavaScriptEnabled(false);
webClient.setTimeout(60000); // 1 minute
URL url = new URL("http://www.iberia.com/?language=en");
// Now you have the url into homepage
HtmlPage homepage = (HtmlPage)webClient.getPage(url);

3. taking a form from a page
// single form

HtmlForm frm1 = (HtmlForm) homepage .getFormByName("airPaxFormSimpl");

// multiple form

List listFormss = homepage.getForms();
HtmlForm frm1 = (HtmlForm)listFormss.get(5);


4. How to set or get value from a text box

HtmlTextInput p0name = (HtmlTextInput) frm1.getInputByName("passengers[0].name");
p0name.setValueAttribute("testA");
HtmlTextInput p0surname = (HtmlTextInput) frm1.getInputByName("passengers[0].surname1");
p0surname.setValueAttribute("sur");
HtmlTextInput p0surname2 = (HtmlTextInput) frm1.getInputByName("passengers[0].surname2");
p0surname2.setValueAttribute("sur");


5. How to set or get value from a check box


HtmlCheckBoxInput returnDirection = (HtmlCheckBoxInput) frm1.getInputByName("vehicle");
returnDirection.setChecked(true);

6.How to set or get value from a dropdown


HtmlSelect departCounty = (HtmlSelect)frm1.getSelectByName("model");
HtmlOption optDepartCountry = departCounty.getOptionByValue("volvo");
optDepartCountry.setSelected(true);


7.How to set or get value from a radiobutton

List list = frm1.getRadioButtonsByName("sex");
for ( int i = 0; i < list.size(); i++){
HtmlRadioButtonInput rbFlex = (HtmlRadioButtonInput) list.get(i);
if (rbFlex.getValueAttribute().equals(("male")) ){
rbFlex.setChecked(true);
break;
}
}
8.How to append a new component into existing form

HtmlHiddenInput sessID = (HtmlHiddenInput)frm1.getInputByName("BV_SessionID");
HtmlHiddenInput engnID = (HtmlHiddenInput)frm1.getInputByName("BV_EngineID");

System.out.println(sessID.getValueAttribute() + " " + engnID.getValueAttribute());

frm1.appendDomChild(sessID);
frm1.appendDomChild(engnID);

9.How to add a new component into existing form

Map hmap1 = new HashMap();
hmap1.put("id", "__EVENTTARGET");
hmap1.put("name", "__EVENTTARGET");
HtmlHiddenInput hid1 = new HtmlHiddenInput(pagePlan,hmap1);
frmPlan.appendDomChild(hid1);

Maphmap2 = new HashMap();
hmap2.put("id", "__EVENTARGUMENT");
hmap2.put("name", "__EVENTARGUMENT");
HtmlHiddenInput hid2 = new HtmlHiddenInput(pagePlan,hmap2);
frmPlan.appendDomChild(hid2);


10.How to remove component from form

HtmlElement elem = frmPlan.getHtmlElementById("SearchBy");
elem.remove();
frmPlan.getHtmlElementById("SearchBy").remove();