Providing Technology Training and Mentoring For Modern Technology Adoption
This tutorial is adapted from Web Age course Practical Machine Learning with Apache Spark. 8.1 Types of Machine Learning There are three main types of machine learning (ML), unsupervised learning, supervised learning, and reinforcement learning. We will be learning only about the unsupervised and supervised learning types. 8.2 Supervised vs Unsupervised Machine Learning In essence, unsupervised learning (UL) attempts to extract
This tutorial is adapted from Web Age course Practical Machine Learning with Apache Spark. 6.1 Python Dev Tools and REPLs In addition to the standard Python REPL, Python development is supported through these tools and systems: IPython, Jupyter with Python kernel (runtime), Visual Studio Code’s Python plug-in, PySpark (integrated with Python REPL). 6.2 IPython IPython (Interactive Python) is a command shell
7.1 What is Data Visualization? The common wisdom states that seeing is believing and a picture is worth a thousand words. Data visualization techniques help users understand the data, underlying trends and patterns by displaying it in a variety of graphical forms (heatmaps, scatter plots, charts, etc.) Data visualization is also a great vehicle for communicating analysis results
1.1 The Traditional Client–Server Processing Pattern It is good for small-to-medium data set sizes. Fetching 1TB worth of data might take longer than 1 hour. 1.2 Enter Distributed Computing We need a new approach — distributed computing where processing is done on clusters of computers. But there are challenges like Process (Software) / Hardware failures,
1.1 Data is King Data is king and it outlives applications. Applications outlive integrations. Organizations striving to become data-driven need to institute efficient, intelligent, and robust ways for data processing. Data engineering addresses many of the aspects of this process. 1.2 Translating Data into Operational and Business Insights Success of organizations is predicated on their ability to convert raw
1.1 What is Spark Shell? The Spark Shell offers interactive command-line environments for Scala and Python users. SparkR Shell has only been thoroughly tested to work with Spark standalone so far and not all Hadoop distros available, and therefore is not covered here. The Spark Shell is often referred to as REPL (Read/Eval/Print Loop).The Spark Shell session
1.1 What is Apache Spark? Apache Spark (Spark) is a general-purpose processing system for large- scale data. Spark is effective for data processing of up to 100s of terabytes on a cluster of machines. It runs on both Unix-like systems and Windows. Main players behind Spark are Apache Software, Databricks (founded by the creators of Spark), UC
This tutorial is adapted from Web Age course AWS Data Engineering Training. Multipart Upload Overview The Multipart upload API enables you to upload large objects in parts. Multipart uploading is a three-step process: ◊ You initiate the upload ◊ you upload the object parts ◊ after you have uploaded all the parts, you complete the multipart
This tutorial is adapted from Web Age course Applied Data Science with Python. This tutorial provides quick overview of Python modules and high-power features, NumPy library, pandas library, SciPy library, scikit-learn library, Jupyter notebooks and Anaconda distribution. 1.1 Importing Modules Command Comments from lib import x from lib import x as y Importing a single object
This tutorial is adapted from Web Age course Applied Data Science with Python. This tutorial aims at helping you refresh your knowledge of Python and show how Python integrates with NumPy and pandas libraries. Part 1 – Set up the Environment 1. Download and install the latest Google Chrome browser from: https://www.google.com/intl/en/chrome/browser 2. Install Anaconda