• http://jaunt-api.com/index.htm
• Nutch - requires hadoop
• A simpler and popular alternative is webmagic
• Crawler4J

Related

• python Scrapy
• javascript PhantomJS/CasperJS

Log4j async logger

    <?xml version="1.0" encoding="UTF-8"?>
        <configuration status="warn" name="MyApp" packages="">
        <appenders>
            <File name="MyFile" fileName="logs/app.log">
                <PatternLayout>
                    <pattern>%d %p %c{1.} [%t] %m%n</pattern>
                </PatternLayout>
            </File>
            <Async name="Async">
                <appender-ref ref="MyFile"/>
            </Async>
        </appenders>
        <loggers>
            <root level="error">
        <appender-ref ref="Async"/>
    </root>
 </loggers>
</configuration>

Logging Key Value pairs

 Map<String,String> myMap = getMyMap(); Logger.debug(new MapMessage(myMap));

The log4j2 pattern matcher of the Pattern Layout can be used to log the map or specific keys

• Full Map logging pattern

"%-5p [%t]: %m %map%n"

• Specific key

"%-5p [%t]: %m %mdc{userID}%n"

https://github.com/jhalterman/failsafe

import org.apache.commons.cli.BasicParser
import org.apache.commons.cli.CommandLine
import org.apache.commons.cli.CommandLineParser
import org.apache.commons.cli.DefaultParser
import org.apache.commons.cli.HelpFormatter
import org.apache.commons.cli.Options
 
/**
 * Created by rdono on 02/05/2017.
 */
class zipi {
 
     static void main (String [] args ) {
         // setup some Options object
         Options options = new Options();
         options.addOption("t", false, "display current time");
 
         // then parse the received requirements 
         CommandLineParser parser = new DefaultParser();
         CommandLine cmd = parser.parse(options, args);
 
         if (cmd.getOptions().size() == 0) {
             HelpFormatter formatter = new HelpFormatter();
             println(formatter.printHelp( "Bulk tool injector", options));
             return
         }
     }
 
}

Interactive text input can be got with http://text-io.beryx.org

• http://unirest.io - Lightweight HTTP Request Client Libraries in multiple languages
• https://square.github.io/retrofit/ A type-safe HTTP client for Android and Java

To prevent Unirest logging the following configuration around cookies stops it

RequestConfig globalConfig = RequestConfig.custom()
.setCookieSpec(CookieSpecs.IGNORE_COOKIES).build();
 
HttpClient httpclient = HttpClients.custom().setDefaultRequestConfig(globalConfig).build();
Unirest.setHttpClient(httpclient);

Sites with invalid SSL certificates (often in development environments), we have to setup unirest with a HttpClient that does not freak out:

  private static HttpClient unsafeHttpClient;
 
    static {
        try {
            SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(null, new TrustSelfSignedStrategy() {
                public boolean isTrusted(X509Certificate[] chain, String authType) throws CertificateException {
                    return true;
                }
            }).build();
 
            unsafeHttpClient = HttpClients.custom().setSSLContext(sslContext)
                    .setSSLHostnameVerifier(new NoopHostnameVerifier()).build();
 
        } catch (KeyManagementException | NoSuchAlgorithmException | KeyStoreException e) {
            e.printStackTrace();
        }
    }
 
    public static HttpClient getClient() {
        return unsafeHttpClient;
    }
 
 
    @Test
    public void testSitemapUp() throws Exception {
        Unirest.setHttpClient(unsafeHttpClient);
 
        HttpResponse<String> response = Unirest.get("https://wtr-wec-t5-test.waitrose.com/sitemap/sitemapIndex.xml")
                .asString();
        assertEquals(200, response.getStatus());
     }

• Clamshell-Cli is a framework for building console-based command-line applications in Java.