The research community around the world increasingly use the “cloud” and “containers” to support reproducible data science research activities.

• Researchers need “ data science workflows ” when working with data and publishing results
• Peer-review of research findings -> results needs to be reproducible
• Workflows increasingly involve open data but also reproducible analyses (code)
• Convergence of HPC and interative web applications - running together on containers.

Creating

• How can I as a researcher create reproducible open research results? Tools and support?
• What do I need to know - skills in “data carpentry” etc?

Sharing

• What does it mean in practice if the research funds requires me to follow FAIR principles?
• How do I deal with “personal data”?
• How do I share my analysis? Is it reproducible?

What are best practices and recommended workflows?

Several such “science clouds” based on open source platforms were represented at the FOSDEM track for HPC, Big Data and Data Science.

Examples: Compute Canada and Austrian BioCenter in Vienna

Trends:

• Converging HPC computing and interactive web applications through containers
• JupyterHub + RStudio as web-based analytics front-ends and Apache Spark as data backend
• Open source with deployments “On-premise” and/or “in cloud”

Compute Canada provides HPC infrastructures and support to every academic research institution in Canada.

Researchers are provided with a complete HPC cluster software environment including a Slurm scheduler (jobs), a Globus Endpoint (file sharing), JupyterHub, LDAP, DNS, and over 3000 research software packages.

https://fosdem.org/2020/schedule/event/magic_castle/

Compute Canada staff has been using this software to deploy ephemeral clusters for training purposes every other week for the past two years.

• Félix-Antoine Fortin at Université Laval, Canada Digital Research Institute
• 5 datacenters, usage free for researchers
• 150 workshops per year, ephemeral accounts approx 3 days
• Access for researchers through https / ssh / globus (GridFTP)

• Audience could log in at “superman.calculquebec.cloud”

• DNS names automated incl SSL/TLS termination
• Demo: Google Talk -> Dialogflow -> Flask -> MagicCastle -> OpenStack
• On GitHub: https://github.com/magic_castle

• Frontend: JupyterHub, GPU support on Compute Nodes

Exploratory data analysis has increased the demand for interactive tools. In the same way, workshops and other teaching events often benefit from immediate and on-demand access to preconfigured, interactive environments.

On-premise container orchestration is often preferable because it enables deploying interactive tools on existing compute infrastructure that provides access to both software packages and the data to be analysed.

The deployment on HPC batch systems specifically brings challenges on how to handle authentication, user identities, and job submissions.

• Colin Sanze, SWC-Abery, cos@aber.ac.uk
• https://github.com/SCW-Aberystwyth/Introduction-to-HPC-with-RaspberryPi
• https://github.com/colinsaze/pi_cluster

Concerns for researchers:

• The availability of a real HPC system and the effect running training courses
• New users fear to use a large and expensive HPC system for the first time
• Abstract HPC systems in data centers -> difficult to understand what you are using

Within BI, existing commercial solutions are QlikTech, TIBCO Spotfire, Tableau, Cognos, SAS mm. Downsides for academic domain:

• Vendor lock-in and license managment requirements and costs.
• Support for modern agile workflows not built-in , such as using GitHub Flow and reproducible containers (Docker Hub)
• Support for reproducibility is increasingly required in the frontlines of academic research, utilizing data science approaches using with R and Python for ML / AI.
• Weak support for Big Data / Fast Data such as data analysis powered by Apache Spark
• Sharing results and code openly

Shareable web-enabled “interactive” notebooks with data + code + visuals and support for job launchers, runs locally or in the cloud by way of containers.

• RStudio
• JupyterHub
• Apache Zeppelin (demo: https://kth-library.github.io/operations/docker-compose.yml)

    docker run -p 8080:8080 -e ZEPPELIN_ADDR=0.0.0.0 --rm \
      --name zeppelin apache/zeppelin:0.8.2

• https://towardsdatascience.com/building-a-graph-data-pipeline-with-zeppelin-spark-and-neo4j-8b6b83f4fb70

• Notebooks are not perfect; Summary of a paper about “What is wrong with notebooks?”
• http://openondemand.org/ (GitHub: https://github.com/OSC/Open-OnDemand) is a NSF-funded open-source HPC portal used by many US universities, exposes for example Jupyter w auth to researchers
• Presto is a high performance, distributed SQL query engine for big data used by 1000s of users at Facebook for interactive querying of data - runs locally, too

Open source-based platform for reproducible research including web-friendly data analysis:

• Front-end for open data science KONTARION

• Can do ML och AI, scales locally or in the cloud using Docker Swarm / Kubernetes, container-based, supports ML/AI workflows using R, Python (Jupyter) etc… GPU-scaling and slurm jobs)

• Add-on: Backend for big data using Apache Spark + Minio + Select S3

A “data science software stack” extending https://www.rocker-project.org/ w domain-specific functionality.

Web-friendly data analytics environment providing RStudio Open Source Edition (AGPL v3) and Jupyter.

Containerized Open Data Science Analytics front-end with sparklyr connection to Big Data backend.

• interactive web-based analytics (runtimes for Shiny och Dash)
• running “jobs”/tasks
• deploying and using APIs for open data
• markdown-based authoring of content

Data Lake w/ Minio + Apache Spark + S3 Select + RStudio Web Edition / JupyterHub

Minio blog article about S3 Select

Takeaways from presentations at FOSDEM20

• Ditch “meta store” and heavy Hadoop Hive setup due to complexity evident in Expedia presentation
• Instead use dockerized Minio + Apache Spark + S3Select due to faster, simpler, “hybrid-cloud” on-prem
• Spark on Docker versus bare metal: https://www.slideshare.net/SparkSummit/lessons-learned-from-running-spark-on-docker-63071214

https://therinspark.com/

A large Data Science community with millions of users from academic sector (and corporate, too), check out some links:

• https://spark.rstudio.com/mlib/ (Spark Machine Learning lib!)
• https://blogs.rstudio.com/tensorflow/
• https://tensorflow.rstudio.com/
• https://www.r-bloggers.com/
• https://plot.ly/dash/
• https://shiny.rstudio.com/