Programming Archives - Liwaiwai

10 Best Python Machine Learning Tutorials

Python is a high-level programming language that is widely used for Machine Learning (ML) applications. It is known for its readability, versatility and ease of use, making it an ideal choice for developers, data scientists, and machine learning engineers alike. The Python ecosystem has a large number of libraries and tools that support machine learning, such as NumPy, Pandas, Matplotlib, TensorFlow, and scikit-learn. These libraries provide powerful algorithms and tools that enable developers to perform complex data analysis, build predictive models and perform data visualization. Python is also popular in machine learning projects due to its robust and active development…

How To Run A Large Scale ML Workflow On Dataflow ML For Autonomous Driving

Developing autonomous driving technology is a battle with data, both from a volume and data format point of view. Sources include point cloud 3D data obtained from LIDAR, video data obtained from multiple cameras, GPS position information, millimeter-wave radar, steering and various sensor information. Even in a busy city, less than 1% of the raw data contains useful information, so substantial effort needs to be put into filtering, interpreting, and combining the information to be used in a coherent manner. One way to avoid all this data cleansing effort is to run simulations in 3D virtual space, which can generate this…

Machine Learning Experiments In Gaming And Why It Matters

Introduction Machine learning (ML) is essential to video game development. Predicting specific in-game actions and identifying and reaching your most valuable players helps to drive better outcomes. To this end, we need to keep track of all experiments that are happening behind the scenes. Google Cloud recently announced the general availability of a new feature called Vertex AI Experiments that can now help gaming companies do just that – keep track of their ML experiments to uncover insights and best practices across their ML Engineering and Data Science teams. In this blog post, we will focus on a popular video game dataset…

Building A Machine Learning Platform With Kubeflow And Ray On Google Kubernetes Engine

Increasingly more enterprises adopt Machine Learning (ML) capabilities to enhance their services, products, and operations. As their ML capabilities mature, they build centralized ML Platforms to serve many teams and users across their organization. Machine learning is inherently an experimental process requiring repeated iterations. An ML Platform standardizes the model development and deployment workflow to offer greater consistency for the repeated process. This facilitates productivity and reduces time from prototype to production. When first trying ML in the cloud, many practitioners will start with fully managed ML platforms like Google Cloud’s Vertex AI. Fully-managed platforms abstract out many complexities to…

Programming

Scaling Heterogeneous Graph Sampling For GNNs With Google Cloud Dataflow

This blog presents an open-source solution to heterogeneous graph sub-sampling at scale using Google Cloud Dataflow (Dataflow). Dataflow is Google’s publicly available, fully managed environment for running large scale Apache Beam compute pipelines. Dataflow provides monitoring and observability out of the box and is routinely used to scale production systems to easily handle extreme datasets. This article will present the problem of graph sub-sampling as a pre-processing step for training a Graph Neural Network (GNN) using Tensorflow-GNN (TF-GNN), Google’s open-source GNN library. The following sections will motivate the problem, present an overview of the necessary tools including Docker, Apache Beam, Google Cloud Dataflow, TF-GNN Unigraph format, TF-GNN…

Use R To Train And Deploy Machine Learning Models On Vertex AI

R is one of the most widely used programming languages for statistical computing and machine learning. Many data scientists love it, especially for the rich world of packages from tidyverse, an opinionated collection of R packages for data science. Besides the tidyverse, there are over 18,000 open-source packages on CRAN, the package repository for R. RStudio, available as desktop version or on the Google Cloud Marketplace, is a popular Integrated Development Environment (IDE) used by data professionals for visualization and machine learning model development. Once a model has been built successfully, a recurring question among data scientists is: “How do I deploy models written in…

Introducing Model Co-Hosting To Enable Resource Sharing Among Multiple Model Deployments On Vertex AI

When deploying models to the Vertex AI prediction service, each model is by default deployed to its own VM. To make hosting more cost effective, we’re excited to introduce model co-hosting in public preview, which allows you to host multiple models on the same VM, resulting in better utilization of memory and computational resources. The number of models you choose to deploy to the same VM will depend on model sizes and traffic patterns, but this feature is particularly useful for scenarios where you have many deployed models with sparse traffic. Understanding the Deployment Resource Pool Co-hosting model support introduces the concept of…

Doing Small Network Scientific Machine Learning In Julia 5x Faster Than PyTorch

Machine learning is a huge discipline, with applications ranging from natural language processing to solving partial differential equations. It is from this landscape that major frameworks such as PyTorch, TensorFlow, and Flux.jl arise and strive to be packages for “all of machine learning”. While some of these frameworks have the backing of large companies such as Facebook and Google, the Julia community has relied on the speed and productivity of the Julia programming language itself in order for its open source community to keep up with the pace of development. It is from this aspect which Flux.jl derives its “slimness”: while PyTorch…

Programming

A Programming Language For Hardware Accelerators

Moore’s Law needs a hug. The days of stuffing transistors on little silicon computer chips are numbered, and their life rafts — hardware accelerators — come with a price. When programming an accelerator — a process where applications offload certain tasks to system hardware especially to accelerate that task — you have to build a whole new software support. Hardware accelerators can run certain tasks orders of magnitude faster than CPUs, but they cannot be used out of the box. Software needs to efficiently use accelerators’ instructions to make it compatible with the entire application system. This translates to a lot of engineering…

Doing Small Network Scientific Machine Learning In Julia 5x Faster Than PyTorch

Machine learning is a huge discipline, with applications ranging from natural language processing to solving partial differential equations. It is from this landscape that major frameworks such as PyTorch, TensorFlow, and Flux.jl arise and strive to be packages for “all of machine learning”. While some of these frameworks have the backing of large companies such as Facebook and Google, the Julia community has relied on the speed and productivity of the Julia programming language itself in order for its open source community to keep up with the pace of development. It is from this aspect which Flux.jl derives its “slimness”: while PyTorch…

Input your search keywords and press Enter.