Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
home [2019/09/13 03:03]
root [BigData File Formats]
home [2020/02/21 05:54] (current)
root [Big Data]
Line 14: Line 14:
 * [[audio]] * [[audio]]
 * [[Java Build Tools]] Jenkins, Teamcity ​ * [[Java Build Tools]] Jenkins, Teamcity ​
-** [[Dependency Injection]] ​inlcuding ​SpringBoot ​+** [[Dependency Injection]] ​including ​SpringBoot ​
 * [[Graalvm]] * [[Graalvm]]
  
Line 25: Line 25:
 ** [[http://​www.pippo.ro/​ | pippo]] ​ ** [[http://​www.pippo.ro/​ | pippo]] ​
 ** [[vertx]] ** [[vertx]]
-*** [[apiman]] + 
-*** Service discovery [[consul]] +Graal Vm Compatible Microservices:​ 
-Misc +[[https://​codurance.es/​2018/​12/​28/​native-microservices-with-sparkjava-and-graal/​ | Spark Java]] 
 +[[https://​micronaut.io/​ | Micronaut]] - A modern, JVM-based, full-stack framework for building modular, easily testable microservice applications. 
 +* [[https://​quarkus.io/​ | Quarkus]] A Kubernetes Native Java stack tailored for OpenJDK HotSpot and GraalVM, crafted from the best of breed Java libraries and standards. 
 + 
 +Api Gateways  
 +* [[apiman]] 
 + 
 +Cloud Comms Mgt  
 +* Service discovery [[consul]] ​- also security authentication when used with [[vault]] or [[forgerock]] 
 + 
 + 
 + 
 + 
 +== Gradle ==  
 +* [[Gradle]] Notes 
 +** [[Gradle Docker]] 
 +** [[Gradle Openshift]] 
 +** [[Gradle Docker and TestContainers]] 
 +** [[Gradle yml templates]] 
 +** [[Gradle Plugins]] 
 + 
 +== Misc ==  ​
 * [[Dropwizard Metrics]] that has become a popular extension in other frameworks ​ * [[Dropwizard Metrics]] that has become a popular extension in other frameworks ​
 * [[Kotlin]], [[Kotlin native]] and  [[Kotlin Javascript]] * [[Kotlin]], [[Kotlin native]] and  [[Kotlin Javascript]]
Line 36: Line 57:
 ** [[Grails]] Notes and [[Gorm]] notes ** [[Grails]] Notes and [[Gorm]] notes
 ** [[Spock]] Notes  ** [[Spock]] Notes 
-** [[Gradle]] Notes 
 * [[Javascript]] , [[Javascript Toolkits]], [[AngularJs]] ..note you can now load dependencies with http://​www.webjars.org/​ * [[Javascript]] , [[Javascript Toolkits]], [[AngularJs]] ..note you can now load dependencies with http://​www.webjars.org/​
 ** [[Vue.js]] and mobile natvie version [[https://​www.nativescript.org/​ | nativeScript]] and more generic [[https://​gonative.io/​examples | gonative]] ** [[Vue.js]] and mobile natvie version [[https://​www.nativescript.org/​ | nativeScript]] and more generic [[https://​gonative.io/​examples | gonative]]
Line 71: Line 91:
 * [[Go Language Notes]] * [[Go Language Notes]]
 * [[Go Applications]] * [[Go Applications]]
 +* [[https://​github.com/​google/​ko | Google Ko]] Building Go and deploying on Kubernates
 == Python ==  == Python == 
  
Line 82: Line 102:
 ** [[Python Async]] ** [[Python Async]]
 ** [[Python Pandas]] ** [[Python Pandas]]
-** [[Python Machine Learning]]+** [[Python Machine Learning]] ​scaling machine learning https://​dask.org/​
 ** [[Postgres Machine Learning]] ** [[Postgres Machine Learning]]
 ** [[Postgres graph and madlib]] ** [[Postgres graph and madlib]]
Line 94: Line 114:
 * [[TensorFlow Notes]] 2.0 * [[TensorFlow Notes]] 2.0
 * [[PyTorch Notes]] 1.0 * [[PyTorch Notes]] 1.0
 +* Kubeflow - Kubeflow is an open, community driven project to make it easy to deploy and manage an ML stack on Kubernetes
  
 == Api Mgt ==  == Api Mgt == 
 * [[Api Gateways]] * [[Api Gateways]]
 +* [[Cloud Native]] Ambassador, Tyko, Kong
  
 +=== Api/Service Mesh === 
 +- [[Service Mesh]] Linkderd ​
  
 == Testing == == Testing ==
Line 136: Line 159:
 == Build Tools ==  == Build Tools == 
 * [[Maven]] * [[Maven]]
 +* [[Gradle]]
  
 == Test Tools ==  == Test Tools == 
Line 155: Line 179:
 * [[https://​github.com/​OpenGamma/​ElSql | ElSql]] Java fluent sql interface. ​ * [[https://​github.com/​OpenGamma/​ElSql | ElSql]] Java fluent sql interface. ​
 * [[http://​www.jooq.org/​doc/​3.4/​manual-single-page/​ | Jooq]] another fluent interface ​ * [[http://​www.jooq.org/​doc/​3.4/​manual-single-page/​ | Jooq]] another fluent interface ​
 +* Also worth a look [[https://​ebean.io/​]] that also provides an elastic, postgresgis interface
 * [[Voltdb Notes]] * [[Voltdb Notes]]
 * [[http://​www.jdbi.org/​]] sql binding without the orm overhead ..used by DropWizard ​ * [[http://​www.jdbi.org/​]] sql binding without the orm overhead ..used by DropWizard ​
Line 218: Line 243:
 * [[parquet]],​ Avro * [[parquet]],​ Avro
  
-==== Json Store ===+==  ​Elastic ​== 
 * [[Elastic Search]] ​ * [[Elastic Search]] ​
 ** [[Kibana]] ** [[Kibana]]
-** [[Timelion]] ​+** Writing [[Kibana plugins]] 
 +** [[Timelion]] 
 +** [[Pipelines]] 
 * [[Elastic Engineer 1]] * [[Elastic Engineer 1]]
 * [[Elastic APM]] * [[Elastic APM]]
 +* [[Elastic filebeats]]
 +* [[Elastic metric beats]]
 +* [[Elastic Uptime Monitor]]
 +* [[Elastic Infastructure]]
 +* [[Elastic Lens]]
 +* [[Elastic Logstash]]
  
 EFK (ElasticSearch,​ Fluentd, Kibana) EFK (ElasticSearch,​ Fluentd, Kibana)
 * [[Fluentd]] * [[Fluentd]]
  
 +== Other Visualisations == 
 +* https://​github.com/​finos/​perspective - very fast rendering based on webassembley
 +* https://​github.com/​uwdata/​falcon - Similar in memory very fast visualisation
 +
 +==== Json Store ===
 Others: Others:
 * Solr Notes, [[Embedding Solr]] ​ * Solr Notes, [[Embedding Solr]] ​
 * [[couchbase]] * [[couchbase]]
 * Java json path query https://​github.com/​jayway/​JsonPath library * Java json path query https://​github.com/​jayway/​JsonPath library
 +
  
 === DB Migrations === === DB Migrations ===
Line 277: Line 316:
  
  
-== MetOps ==+== MetOps/​DataOps ​== 
 + 
 +* [[https://​github.com/​chen1649chenli/​dataOpsResource | Awesome DataOps]] list 
  
 Metics and Goal setting ​ Metics and Goal setting ​
Line 304: Line 345:
 Alerting: Alerting:
 * https://​alerta.io/​ * https://​alerta.io/​
 +
 +Tracing ​
 +* https://​www.jaegertracing.io/​ End to end distributed tracing (which works with zipkin)
 +* https://​zipkin.io/​ Zipkin is a distributed tracing system. It helps gather timing data needed to troubleshoot latency problems in service architectures. Features include both the collection and lookup of this data.
  
 Flask based dashboard ​ Flask based dashboard ​
Line 349: Line 394:
  
 * [[EMC'​s Options]] * [[EMC'​s Options]]
 +
 +Although Apache Spark and Apache Ignite both carries the power of in-memory computing, Still they have some conceptual differences:​
 +
 +Spark doesn’t store data, actually it loads data from other storages for processing purpose, usually disk-based. As soon as the processing is finished it discards the data. On the other hand, Ignite provides a distributed in-memory key-value store (distributed cache or data grid) with SQL querying capabilities and ACID transactions.
 +
 +Spark is used for non-transactional,​ read-only data (RDDs don’t support in-place mutation), while Ignite supports both fully ACID compliant transactions (OLTP) and non-transactional (OLAP) payloads.
 +
  
 Data Analytics ​ Data Analytics ​
Line 374: Line 426:
  
 == Stream Processsing ==  == Stream Processsing == 
-* [[kafka]] and [[python kafka]]+* [[kafka]] and [[python kafka]] ​ and [[https://​github.com/​yahoo/​CMAK | kafka manager]] 
 +** http://​curator.apache.org/​ Zookeeper manager -  a distributed coordination service. It includes a highlevel API framework and utilities to make using Apache ZooKeeper
 *  http://​samza.incubator.apache.org/​ *  http://​samza.incubator.apache.org/​
 * It’s worth noting that Cassandra or HBase can be used a resequencing buffer, this technique is described in http://​www.slideshare.net/​dave_revell/​nearrealtime-analytics-with-kafka-and-hbase * It’s worth noting that Cassandra or HBase can be used a resequencing buffer, this technique is described in http://​www.slideshare.net/​dave_revell/​nearrealtime-analytics-with-kafka-and-hbase
Line 405: Line 458:
 * [[mule_databases]] * [[mule_databases]]
 * [[mule devkit]] and cloud connectors * [[mule devkit]] and cloud connectors
-* [[https://​malta.ricston.com/​confluence/​display/​CLOUD/​Building+Cloud+Connectors+with+Devkit | DevKit]] ​Ricston Blog+* [[https://​malta.ricston.com/​confluence/​display/​CLOUD/​Building+Cloud+Connectors+with+Devkit | DevKit]] ​
 * [[Mule Migration]] * [[Mule Migration]]
 * [[Mule Messaging]] specifically [[RabbitMq]],​ ActiveMq, ZeroMq, mqtt * [[Mule Messaging]] specifically [[RabbitMq]],​ ActiveMq, ZeroMq, mqtt
Line 416: Line 469:
 * [[Mule Logging]] * [[Mule Logging]]
 * [[Mule Munit Testing]] * [[Mule Munit Testing]]
 +* [[Mule command line]] and automated deployments
  
 Mule Api Integration:​ Mule Api Integration:​
Line 427: Line 481:
 * [[Talend Notes]] * [[Talend Notes]]
  
-Others+StreamSets  
 +* [[StreamSets]] 
 +* [[StreamSets Python]] 
 +* [[Scaling Streamsets on Kubernates]] 
 + 
 +== Data Ops == 
 * [[Gobblin]] * [[Gobblin]]
 * [[Apache Airflow]] ​ * [[Apache Airflow]] ​
 +* [[https://​github.com/​argoproj/​argo | Argo]] a kubernates/​workflow that claims to rival airflow (could be commercialised?​)  
 +* [[Apache Beam]] batch etl tool  and [[https://​cloud.google.com/​dataflow/​]] 
 +* [[Hazelcast]] and hazelcast Jet <== potential commercial tie up 
 == Camel ==  == Camel == 
 * [[Camel Notes]] * [[Camel Notes]]
Line 445: Line 506:
 * [[Dropwizard]] * [[Dropwizard]]
 * [[springBoot]] * [[springBoot]]
 +* [[quarkus]] with graalvm
  
 +== Alibaba == 
 +* [[https://​github.com/​alibaba/​arthas | Arthas]] Alibaba Java Diagnostic Tool Arthas
 +* [[https://​github.com/​alibaba/​Sentinel | Sentinel]] lightweight powerful flow control component enabling reliability and monitoring for microservices. entinel also provides real-time monitoring ability. You can see the runtime information of a single machine in real-time, and the aggregated runtime info of a cluster with less than 500 nodes.
 +* [[https://​github.com/​alibaba/​nacos | Nacos]] is an easy-to-use platform designed for dynamic service discovery and configuration and service management.
 +* an [[https://​github.com/​ellerbrock/​awesome-alibaba-cloud | awesome list]] for alibaba'​s cloud 
 == Links ==  == Links == 
 * [[srm links]] * [[srm links]]
-* [[Ricston links]] 
 * [[javascript framework links]] * [[javascript framework links]]
  
Line 470: Line 536:
 * [[http://​www.lavishbootstrap.com/​ Lavish]] extract the colors from a photo and provide a bootstrap css template * [[http://​www.lavishbootstrap.com/​ Lavish]] extract the colors from a photo and provide a bootstrap css template
  
-== Configuration Mgt ==+== Agile ==  
 +* [[Agile Development]] 
 +== DevOps ​==
 * [[jenkins]] * [[jenkins]]
 +** [[Jenkins groovy dsl]]
 +** [[Openshift Jenkins]]
 +** [[Openshift Jenkins Terraform]]
 +** [[Kubernates Jenkins]]
 +
 * [[Puppet, Chef, SaltStack, and Ansible]] and glu * [[Puppet, Chef, SaltStack, and Ansible]] and glu
 ** [[Puppet Notes]] ** [[Puppet Notes]]
 * [[Terraform notes]] * [[Terraform notes]]
 +** [[Jenkins - Terraform]]
 * [[openStack]] * [[openStack]]
 * Monitoring [[http://​stackhunter.com/​ | stackhunter]] * Monitoring [[http://​stackhunter.com/​ | stackhunter]]
-* [[openShift]]+* [[Docker]] 
 +** local [[docker registry]] 
 + 
 +=== Elastic Search ===  
 +* [[Elastic Search]]  
 +** [[Kibana]] 
 +** [[Timelion]]  
 +* [[Elastic Engineer 1]] 
 +* [[Elastic APM]] 
 +* [[Elastic filebeats]] 
 +* [[Elastic metric beats]] 
 +* [[Elastic Uptime Monitor]] 
 + 
 +=== Ansible ===  
 +* [[Ansible]] basics  
 +** [[Ansible Openshift]] 
 +** [[Ansible Gcloud]] 
 +** [[Ansible Operators]] 
 +== OpenShift ==  
 +* [[openShift]] ​overview and [[OC command]]  
 +**[[Openshift Template]] 
 +**[[Openshift Routes]] 
 +**[[Openshift Certificates]] 
 +**[[Openshift Clusters]] 
 +**[[Openshift Deployments]] templates, config, rollback etc.  
 +**[[Openshift Monitoring]] - including metrics 
 +**[[Openshift roles and security]] settings 
 +** [[Build and import and image]] 
 + 
 +* Openshift development [[minishift]] Openshift 4 [[Code Ready Containers CRC]] 
 +** [[Accessing Openshift]] cluster externally, including Kafka 
 + 
 +Monitoring and Management  
 +* [[Openshift Jenkins]] 
 +* [[Openshift Jenkins Terraform]] 
 +* [[Openshift Security]] and secrets with Vault  
 + 
 +Enterprise Deployments 
 +* [[Dual Site Deployemnts]] deploying-openshift-applications-multiple-datacenters/​ 
 + 
 +On other platforms  
 +*[[Openshift on GCP]] or [[Openshift on Azure]] and [[Openshift on AWS]] 
 + 
 +*[[Openshift 4]] 
 +* https://​docs.okd.io/ ​ OKD is a distribution of Kubernetes optimized for continuous application development and multi-tenant deployment. OKD also serves as the upstream code base upon which Red Hat OpenShift Online and Red Hat OpenShift Container Platform are built. 
 + 
 * [[https://​kapitan.dev/​]] a json/yml file property manager to (intally ) drive devops ​ * [[https://​kapitan.dev/​]] a json/yml file property manager to (intally ) drive devops ​
 +
 +Application Development ​
 * consul from Hashicorp - Consul is a distributed,​ highly-available,​ and multi-datacenter aware tool for service discovery, configuration,​ and orchestration * consul from Hashicorp - Consul is a distributed,​ highly-available,​ and multi-datacenter aware tool for service discovery, configuration,​ and orchestration
 +* [[Redhat ServieMesh]] Istio on OpenShift
  
 See also  See also 
Line 487: Line 610:
 ** Powerful configuration mechanisms (interface binding, multi-source support with fallback strategy, merging, ...) ** Powerful configuration mechanisms (interface binding, multi-source support with fallback strategy, merging, ...)
 ** Distributed-environment friendly ( caching, support for multiple environments [test, preprod, prod], ...) ** Distributed-environment friendly ( caching, support for multiple environments [test, preprod, prod], ...)
 +
 +== Cloud Storage == 
 +* [[Cloudian Hyperstore]]
  
 === DevSecOps ===  === DevSecOps === 
-* Hashicorp Vault and the nice vuejs front end https://​vault-ui.io/​ [[vault notes]]+* Hashicorp Vault and the nice vuejs front end https://​vault-ui.io/​ [[vault notes]] .. and  
 +[[https://​dzone.com/​articles/​ansible-vault | Ansible Vault]] 
 +* Similarly [[forgerock]] 
 * [[https://​www.pingidentity.com/​content/​ping/​en.html | Ping]] auth and identity mgt  ​ * [[https://​www.pingidentity.com/​content/​ping/​en.html | Ping]] auth and identity mgt  ​
 * [[Consul]] - Consul is a distributed service mesh to connect, secure, and configure services across any runtime platform and public or private cloud * [[Consul]] - Consul is a distributed service mesh to connect, secure, and configure services across any runtime platform and public or private cloud
 +* [[Envoy]](https://​www.envoyproxy.io/​) Open Source Edge and Service proxy, designed for cloud native applications ​
 * [[Packer]] - a tool for creating identical machine images for multiple platforms from a single source configuration * [[Packer]] - a tool for creating identical machine images for multiple platforms from a single source configuration
-* [[https://​www.sumologic.com/​ | SumoLogic]] tag line of "The world’s leading platform for DevSecOps"​ <== also has partnership ​potenial ​+* [[https://​www.sumologic.com/​ | SumoLogic]] tag line of "The world’s leading platform for DevSecOps"​ <== also has partnership ​potential  
 +* Manage Secrets even in Git [[https://​github.com/​bitnami-labs/​sealed-secrets | Sealed Secrets]] Encrypt your Secret into a SealedSecret,​ which is safe to store - even to a public repository. The SealedSecret can be decrypted only by the controller running in the target cluster and nobody else (not even the original author) is able to obtain the original Secret from the SealedSecret. 
 + 
 +* [[credentials management]] 
 + 
 +Configuration Management:​ 
 +* [[https://​github.com/​ManageIQ/​manageiq | ManageIq]]
  
 +Ci/CD
 +* [[devSec Gradle Plugins]]
 == Video ==  == Video == 
  
Line 515: Line 652:
 * [[http://​themeforest.net/​item/​material-resume-cv-vcard/​13703959?​s_rank=1 | resume template * [[http://​themeforest.net/​item/​material-resume-cv-vcard/​13703959?​s_rank=1 | resume template
 ]] ]]
-* [[http://​themeforest.net/​item/​businessbox-corporate-business-template/​5180049?​WT.ac=search_thumb&​WT.oss_phrase=corporate&​WT.oss_rank=9&​WT.z_author=trendyWebStar | potential Ricston]]+
  
  
Line 575: Line 712:
 * [[App Id]] * [[App Id]]
 * [[serverless]] framework ​ * [[serverless]] framework ​
 +
 +=== Kubernates ===
 * [[kubernates]] * [[kubernates]]
 +** [[Getting Started for Development]] - with miniKube
 +** [[Getting Started with Helm]]
 ** [[Kubernates GCP]] ** [[Kubernates GCP]]
 ** [[Kubernates AWS]] ** [[Kubernates AWS]]
 ** [[Kubernates Azure]] ** [[Kubernates Azure]]
 +** [[Kubernates Ingress Controllers]] including traefik
 +** [[Kubernates Security]]
 +** [[Cloud native storage]] Rook
 +** [[Kubernates Monitoring]] including readinessProbes etc. 
 +** [[Gradle Kubernates]] ​
 +** [[Kubernates Daemonsets]]
 +** [[Kubernates Rasberry Pi]]
 +* [[Openshift]]
  
 == Google Cloud (Gcloud) == == Google Cloud (Gcloud) ==
Line 610: Line 759:
 * [[Financial Apps, Api'​s]] * [[Financial Apps, Api'​s]]
 * [[Jira Api]] * [[Jira Api]]
-* Ricston [[https://​docs.google.com/​a/​ricston.com/​spreadsheet/​ccc?​key=0AjynU6mIbKbadF9VTnpNdHZvOWU4QkdUalZwUnFXa3c&​usp=drive_web#​gid=9 | work resources]] +
-Screen share +
 * http://​www.ScreenLeap.com ​ * http://​www.ScreenLeap.com ​
 * [[nlp]] * [[nlp]]
Line 678: Line 826:
 * [[https://​github.com/​discourse/​discourse | discorse]] open source chat tool  * [[https://​github.com/​discourse/​discourse | discorse]] open source chat tool 
  
 +== French == 
 +
 +* [[French Lyrics]]
 +== Korean == 
 +* [[korean lyrics]]
 == Rasberry Pi and Ubuntu Core ==  == Rasberry Pi and Ubuntu Core == 
 * [[Step 1 setup wireless networking]] * [[Step 1 setup wireless networking]]
-* [[Step 2 add ssh server and connect]]+* [[Step 2 add ssh server and connect]]A
  
 +== Alchemy == 
 +* [[Alchemical Tarot]]
 
home.1568368993.txt.gz · Last modified: 2019/09/13 03:03 by root
 
RSS - 200 © CrosswireDigitialMedia Ltd