mattanalytix

GCP Batch Ingestion 1 - Local Development

Sat, 13 May 2023 10:03:58 +0700

An introduction to locally developing code for a batch ingestion process using python and the GCP API. Introducing the motivation behind this series of articles and the code concepts.

Productionising the Process

Mon, 08 May 2023 10:55:44 +0700

So your code is working fine locally but you don’t want to be stuck running manual scripts every day. In this article, I introduce automating the process with Flask, Docker and Cloud Run.

Google Drive Source Image

Mon, 27 Mar 2023 09:31:44 +0700

This is a quick tutorial which shows you how to use images from your google drive on a website.

First create a new folder on your Google Drive if you don’t have one already. Usually, when I create a website I like to have a folder insite my website directory called “Static” with all of the images for the site. I will make this folder public to avoid repitition.

Next, make sure that the folder is shared publicly. Right click on the folder and click “Share”. Then, share with “Anyone with the link” as shown below.

Finally, a better way to deploy DBT on Google Cloud!

Sat, 16 Apr 2022 09:09:54 +0100

For the last year or so I’ve been looking for a good way to productionise DBT pipelines on Google Cloud Platform but I’ve been frustrated by the solutions I’ve found on the web.

Local Environment Setup For DBT With GCP Using Conda

Sat, 16 Apr 2022 09:09:54 +0100

This article is also on medium.com

Setting up a local environment for DBT (BigQuery) using conda

Recently dbt has been increasing in popularity as a structured and robust framework for writing SQL logic. Although the dbt documentation is a great resource, it is not too specific on local configuration. So here is a quick article on setting up a local dbt-bigquery environment with anaconda that works on both Windows and a Mac …

Markdown Samples

Mon, 08 Jun 2020 08:06:25 +0600

This is a sample post intended to test the followings:

A different post author.
Table of contents.
Markdown content rendering.
Math rendering.
Emoji rendering.

Markdown Syntax Rendering

Headings

The following HTML <h1>—<h6> elements represent six levels of section headings. <h1> is the highest section level while <h6> is the lowest.

H1

H2

H3

H4

H5

H6

Paragraph

Xerum, quo qui aut unt expliquam qui dolut labo. Aque venitatiusda cum, voluptionse latur sitiae dolessi aut parist aut dollo enim qui voluptate ma dolestendit peritin re plis aut quas inctum laceat est volestemque commosa as cus endigna tectur, offic to cor sequas etum rerum idem sintibus eiur? Quianimin porecus evelectur, cum que nis nust voloribus ratem aut omnimi, sitatur? Quiatem. Nam, omnis sum am facea corem alique molestrunt et eos evelece arcillit ut aut eos eos nus, sin conecerem erum fuga. Ri oditatquam, ad quibus unda veliamenimin cusam et facea ipsamus es exerum sitate dolores editium rerore eost, temped molorro ratiae volorro te reribus dolorer sperchicium faceata tiustia prat.

Shortcodes Samples

Mon, 08 Jun 2020 08:06:25 +0600

This is a sample post intended to test the followings:

Default hero image.
Different shortcodes.

Alert

The following alerts are available in this theme.

This is sample alert with type="success".

This is sample alert with type="danger".

This is sample alert with type="warning".

This is sample alert with type="info".

This is sample alert with type="dark".

This is sample alert with type="primary".

This is sample alert with type="secondary".

Image

A sample image without any attribute.

A sample image with `height` and `width` attributes.

A center aligned image with `height` and `width` attributes.

A image with `float` attribute.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Cras egestas lectus sed leo ultricies ultricies. Praesent tellus risus, eleifend vel efficitur ac, venenatis sit amet sem. Ut ut egestas erat. Fusce ut leo turpis. Morbi consectetur sed lacus vitae vehicula. Cras gravida turpis id eleifend volutpat. Suspendisse nec ipsum eu erat finibus dictum. Morbi volutpat nulla purus, vel maximus ex molestie id. Nullam posuere est urna, at fringilla eros venenatis quis.

Introduction

Mon, 01 Jan 0001 00:00:00 +0000

Hello World

A sample go program is show here.

package main

import "fmt"

func main() {
 message := greetMe("world")
 fmt.Println(message)
}

func greetMe(name string) string {
 return "Hello, " + name + "!"
}

Run the program as below:

$ go run hello.go

Variables

Normal Declaration:

var msg string
msg = "Hello"

Shortcut:

msg := "Hello"

Constants

const Phi = 1.618

Introduction

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

SASS is a styling framework which sits on top of CSS, which allows you to use loops, variables …

Every css file is a valid scss file (i.e. if you have a bunch of css files already you can simply convert them to .scss files to get started).

Installation on Mac

SASS is written in Ruby, hence you need to download Ruby to use SASS. The two programmes required to use SASS are Ruby and Ruby Gems (a package manager for Ruby).

Basic Types

Mon, 01 Jan 0001 00:00:00 +0000

Strings

str := "Hello"

Multiline string

str := `Multiline
string`

Numbers

Typical types

num := 3 // int
num := 3. // float64
num := 3 + 4i // complex128
num := byte('a') // byte (alias for uint8)

Other Types

var u uint = 7 // uint (unsigned)
var p float32 = 22.7 // 32-bit float

Arrays

// var numbers [5]int
numbers := [...]int{0, 0, 0, 0, 0}

Pointers

func main () {
 b := *getPointer()
 fmt.Println("Value is", b)

func getPointer () (myPointer *int) {
 a := 234
 return &a

a := new(int)
*a = 234

Pointers point to a memory location of a variable. Go is fully garbage-collected.

Usage

Mon, 01 Jan 0001 00:00:00 +0000

Variables

Variables are intuitive as in any other programming language. The syntax for variables is like the below.

$my-color = "blue"

.my_class {
 color: $my-color;
}

Nesting

This is also intuitive but a very useful feature of scss. You can nest css classes. This is much more readable when you have several nested elements.

main article p {
 color: yellow
}

scss equivelant:

main {
 article {
 p {
 color: yellow;
 }
 }
}

Partials and Imports

You can create a partial in scss by naming your file with a leading _ (i.e. _header.scss). Then import the file with:

Flow Control

Mon, 01 Jan 0001 00:00:00 +0000

Condition

if day == "sunday" || day == "saturday" {
 rest()
} else if day == "monday" && isTired() {
 groan()
} else {
 work()
}

if _, err := doThing(); err != nil {
 fmt.Println("Uh oh")

Switch

switch day {
 case "sunday":
 // cases don't "fall through" by default!
 fallthrough

 case "saturday":
 rest()

 default:
 work()
}

Loop

for count := 0; count <= 10; count++ {
 fmt.Println("My counter is at", count)
}

entry := []string{"Jack","John","Jones"}
for i, val := range entry {
 fmt.Printf("At position %d, the character %s is present\n", i, val)

n := 0
x := 42
for n != x {
 n := guess()
}

File Manipulation

Mon, 01 Jan 0001 00:00:00 +0000

Condition

if day == "sunday" || day == "saturday" {
 rest()
} else if day == "monday" && isTired() {
 groan()
} else {
 work()
}

if _, err := doThing(); err != nil {
 fmt.Println("Uh oh")

Bash Variables

Mon, 01 Jan 0001 00:00:00 +0000

Variable

NAME="John"
echo $NAME
echo "$NAME"
echo "${NAME}

Condition

if [[ -z "$string" ]]; then
 echo "String is empty"
elif [[ -n "$string" ]]; then
 echo "String is not empty"
fi

DBT Basic

Mon, 01 Jan 0001 00:00:00 +0000

Useful Resources

Syntax overview - dbt’s note selection syntax
Optimizing VS Code for dbt on Mac - awesome vs code setup for dbt, article written specifically for Mac but same could be done for Windows
Pro tips for workflows - advanced dbt workflows
Youtube: Create Modular Date Models - presentation explaining how you can refactor dbt data models to make them more structured and performant
Project Checklist: DAG Auditing

Workflows

DBT saves artifacts when jobs are run which allows for creating powerful workflows. The states of commands are saved as artifacts. For example, the below code will reference the status of the data’s freshness and only build models that are “fresh”.

GCP Cloud SQL

Mon, 01 Jan 0001 00:00:00 +0000

Delete an Instance

By default Cloud SQL instances are created with deletion protection. To delete an instance you must first remove deletion protection using the gcloud cli. Run the following code in gcloud with your instance name to delete a Cloud SQL instance.

gcloud sql instances patch [INSTANCE_NAME] --no-deletion-protection
gcloud sql instances delete [INSTANCE_NAME]

mattanalytix

GCP Batch Ingestion 1 - Local Development

Productionising the Process

Google Drive Source Image

Finally, a better way to deploy DBT on Google Cloud!

Local Environment Setup For DBT With GCP Using Conda

Setting up a local environment for DBT (BigQuery) using conda

Markdown Samples

Markdown Syntax Rendering

Headings

H1

H2

H3

H4

H5

H6

Paragraph

Shortcodes Samples

Alert

Image

A sample image without any attribute.

A sample image with height and width attributes.

A center aligned image with height and width attributes.

A image with float attribute.

Introduction

Hello World

Variables

Constants

Introduction

Introduction

Installation on Mac

Basic Types

Strings

Numbers

Arrays

Pointers

Usage

Variables

Nesting

Partials and Imports

Flow Control

Condition

Switch

Loop

File Manipulation

Condition

Bash Variables

Variable

Condition

DBT Basic

Useful Resources

Workflows

GCP Cloud SQL

Delete an Instance

A sample image with `height` and `width` attributes.

A center aligned image with `height` and `width` attributes.

A image with `float` attribute.