Missing — "*In the beginning God said, the four-dimensional divergence of an antisymmetric, second rank tensor equals zero, and there was light, and it was good. And on the seventh day he rested***." --Michio Kaku**

Watch The Video Here: https://youtu.be/nLINKnoIl40

Open Code in Google Colab:

dl02

taudata Analytics

DL 02 - Pendahuluan PyTorch untuk Deep learning

https://taudataid.github.io/dl02/

Video Penjelasan:

Prasyarat Module ¶

Sebelum dilanjutkan, yakinkan sudah mempelajari dengan baik module:¶

Instalasi ¶

Sangat mudah, ikuti perintah di https://pytorch.org/get-started/locally/
Perhatikan harware yang dimiliki.
Saat ini AMD (Rocm) sudah disupport oleh PyTorch dan TensorFlow
Perhatikan jika anda install PyTorch dan Tensorflow pada environment yang sama.

Why Torch? ¶

https://discovery.hgdata.com/product/pytorch

In [2]:

# Load Module
import torch, numpy as np

In [3]:

# Kalau saya sukanya cek versi dan apakah PyTorch (atau tensorflow) sudah dapat mengakses GPU atau belum
print("Using Numpy and Pytorch versions = {}, {}".format(np.__version__, torch.__version__))
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print('Using device:', device, end=', ')
if device.type == 'cuda': #Additional Info when using cuda
    print(torch.cuda.get_device_name(0))
    print('Currently Memory Allocated, Cached =', round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB, ', round(torch.cuda.memory_reserved(0)/1024**3,1), 'GB')

Using Numpy and Pytorch versions = 1.21.5, 1.11.0+cu113
Using device: cuda, NVIDIA GeForce RTX 3060 Laptop GPU
Currently Memory Allocated, Cached = 0.0 GB,  0.0 GB

Tensor ¶

Tensors are a specialized data structure that are very similar to arrays and matrices. In PyTorch, we use tensors to encode the inputs and outputs of a model, as well as the model’s parameters.

Tensors are similar to NumPy’s ndarrays, except that tensors can run on GPUs or other specialized hardware to accelerate computing. If you’re familiar with ndarrays, you’ll be right at home with the Tensor API. If not, follow along in this quick API walkthrough.

In [4]:

# Tensor dari Python List

data = [[1, 2], [3, 4]]
x_data = torch.tensor(data)
x_data

Out[4]:

tensor([[1, 2],
        [3, 4]])

In [5]:

x_data.shape, x_data.type()

Out[5]:

(torch.Size([2, 2]), 'torch.LongTensor')

Penting untuk memahami Tipe Tensor¶

Beberapa permasalahan sensitive terhadap akurasi Floating Point
Menghindari overflow/underflow

In [6]:

torch.finfo(torch.float16)

Out[6]:

finfo(resolution=0.001, min=-65504, max=65504, eps=0.000976562, tiny=6.10352e-05, dtype=float16)

In [7]:

# Tensor dari Numpy
np_array = np.array(data)
x_np = torch.from_numpy(np_array)
x_np

Out[7]:

tensor([[1, 2],
        [3, 4]], dtype=torch.int32)

In [8]:

# Ukuran  & property (shape, datatype) sama, tapi element berbeda.
#  unless explicitly overridden.

x_ones = torch.ones_like(x_data) # retains the properties of x_data
print(f"Ones Tensor: \n {x_ones} \n")

x_rand = torch.rand_like(x_data, dtype=torch.float) # overrides the datatype of x_data
print(f"Random Tensor: \n {x_rand} \n")

Ones Tensor: 
 tensor([[1, 1],
        [1, 1]]) 

Random Tensor: 
 tensor([[0.6655, 0.1963],
        [0.0498, 0.6696]])

In [9]:

x_ones.shape, x_ones.type()

Out[9]:

(torch.Size([2, 2]), 'torch.LongTensor')

Warning on "Device" and "Dtype"¶

Jika memang ingin menjalankan DL kita di GPU, maka sangat penting secara eksplisit menetapkan Tipe dan Device
Terutama pada input variabel DL kita.
Data di device yang tepat DAN Model yang dijalankan di GPU adalah kunci sukses menjalankan DL-nya di GPU
Misal kelak kita akan menggunakan perintah berikut:
- model = torch.nn.Sequential(*modules).to(device)
- Begitu juga pada Grad Function di DL kita nanti.

Contoh¶

In [10]:

dtype = torch.float
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') 
# diatas sebenarnya ini sudah di define, diulang utk mempertegas

# instead of 
data = [[1, 2], [3, 4]]
x_data = torch.tensor(data)
print(x_data)

# Sebaiknya lakukan ini jika memang ingin menjalankan DL di GPU
x_data = torch.tensor(data, dtype=dtype, device=device)
x_data

tensor([[1, 2],
        [3, 4]])

Out[10]:

tensor([[1., 2.],
        [3., 4.]], device='cuda:0')

Warning 2¶

Jika DL sudah selesai, Tensor yang ada di GPU tidak bisa langsung di convert ke Numpy.
Harus di copy ke CPU terlebih dahulu

In [11]:

try:
    xx = x_data.detach().numpy()
    print(xx)
except Exception as err_:
    print(err_)

can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

What we should do instead:¶

In [12]:

try:
    xx = x_data.cpu().detach().numpy()
    print(xx)
except Exception as err_:
    print(err_)

[[1. 2.]
 [3. 4.]]

In [13]:

# dimensi Tensor adalah Tuple

shape = (2, 3)
rand_tensor = torch.rand(shape)
ones_tensor = torch.ones(shape)
zeros_tensor = torch.zeros(shape)

print(f"Random Tensor: \n {rand_tensor} \n")
print(f"Ones Tensor: \n {ones_tensor} \n")
print(f"Zeros Tensor: \n {zeros_tensor}")

Random Tensor: 
 tensor([[0.5431, 0.7966, 0.5175],
        [0.7834, 0.5468, 0.5039]]) 

Ones Tensor: 
 tensor([[1., 1., 1.],
        [1., 1., 1.]]) 

Zeros Tensor: 
 tensor([[0., 0., 0.],
        [0., 0., 0.]])

In [14]:

# Check Tensor ada dimana
tensor = torch.rand(3, 4)
print(f"Device tensor is stored on: {tensor.device}")

Device tensor is stored on: cpu

In [15]:

# Bisa pindah ke GPU ... perhatikan kita bisa punya >1 GPU, index GPU mulai dari 0
if torch.cuda.is_available():
    tensor = tensor.to('cuda')
    print(f"Device tensor is stored on: {tensor.device}")

Device tensor is stored on: cuda:0

In [16]:

# Slicing Tensor = Numpy
tensor = torch.ones(4, 4)
tensor[:,1] = 0.0
tensor[1,:] = -2.0
tensor[-1,-1] = 99.0
print(tensor)

tensor([[ 1.,  0.,  1.,  1.],
        [-2., -2., -2., -2.],
        [ 1.,  0.,  1.,  1.],
        [ 1.,  0.,  1., 99.]])

In [17]:

# Operasi Concatenasi (dan operasi lainnya) juga mirip dengan Numpy

t1 = torch.cat([tensor, tensor, tensor], dim=1)
print(t1)

tensor([[ 1.,  0.,  1.,  1.,  1.,  0.,  1.,  1.,  1.,  0.,  1.,  1.],
        [-2., -2., -2., -2., -2., -2., -2., -2., -2., -2., -2., -2.],
        [ 1.,  0.,  1.,  1.,  1.,  0.,  1.,  1.,  1.,  0.,  1.,  1.],
        [ 1.,  0.,  1., 99.,  1.,  0.,  1., 99.,  1.,  0.,  1., 99.]])

In [18]:

# Operasi Tensor = Numpy, berarti element Wise ... Hati-hati tidak seperti Matlab.
print(f"tensor.mul(tensor) \n {tensor.mul(tensor)} \n")
# Alternative syntax:
print(f"tensor * tensor \n {tensor * tensor}")

tensor.mul(tensor) 
 tensor([[1.0000e+00, 0.0000e+00, 1.0000e+00, 1.0000e+00],
        [4.0000e+00, 4.0000e+00, 4.0000e+00, 4.0000e+00],
        [1.0000e+00, 0.0000e+00, 1.0000e+00, 1.0000e+00],
        [1.0000e+00, 0.0000e+00, 1.0000e+00, 9.8010e+03]]) 

tensor * tensor 
 tensor([[1.0000e+00, 0.0000e+00, 1.0000e+00, 1.0000e+00],
        [4.0000e+00, 4.0000e+00, 4.0000e+00, 4.0000e+00],
        [1.0000e+00, 0.0000e+00, 1.0000e+00, 1.0000e+00],
        [1.0000e+00, 0.0000e+00, 1.0000e+00, 9.8010e+03]])

In [19]:

# Perkalian Matrix

print(f"tensor.matmul(tensor.T) \n {tensor.matmul(tensor.T)} \n")
# Alternative syntax:
print(f"tensor @ tensor.T \n {tensor @ tensor.T}")

tensor.matmul(tensor.T) 
 tensor([[ 3.0000e+00, -6.0000e+00,  3.0000e+00,  1.0100e+02],
        [-6.0000e+00,  1.6000e+01, -6.0000e+00, -2.0200e+02],
        [ 3.0000e+00, -6.0000e+00,  3.0000e+00,  1.0100e+02],
        [ 1.0100e+02, -2.0200e+02,  1.0100e+02,  9.8030e+03]]) 

tensor @ tensor.T 
 tensor([[ 3.0000e+00, -6.0000e+00,  3.0000e+00,  1.0100e+02],
        [-6.0000e+00,  1.6000e+01, -6.0000e+00, -2.0200e+02],
        [ 3.0000e+00, -6.0000e+00,  3.0000e+00,  1.0100e+02],
        [ 1.0100e+02, -2.0200e+02,  1.0100e+02,  9.8030e+03]])

In [20]:

# Tanda "_" artinya operasi "inplace" ... ini penting untuk diingat untuk menekan memory. Terutama di GPU

print(tensor, "\n")
tensor.add_(5)
print(tensor)

tensor([[ 1.,  0.,  1.,  1.],
        [-2., -2., -2., -2.],
        [ 1.,  0.,  1.,  1.],
        [ 1.,  0.,  1., 99.]]) 

tensor([[  6.,   5.,   6.,   6.],
        [  3.,   3.,   3.,   3.],
        [  6.,   5.,   6.,   6.],
        [  6.,   5.,   6., 104.]])

In [21]:

# Tentu saja Tensor dapat diubah kembali kke numpy
# Coba sampaikan di kolom komentar, kira-kira kapan kita butuh hal ini?
t = torch.ones(5)
print(f"t: {t}")
n = t.numpy()
print(f"n: {n}")

t: tensor([1., 1., 1., 1., 1.])
n: [1. 1. 1. 1. 1.]

In [22]:

# Tapi "reference to variable" seperti di adsp-01 tetap berlaku (pythonic)
# perubahan di t akan merubah n
t.add_(1)
print(f"t: {t}")
print(f"n: {n}")

t: tensor([2., 2., 2., 2., 2.])
n: [2. 2. 2. 2. 2.]

Tapi kalau Tensor ada di GPU jadi agak beda¶

In [23]:

t = torch.ones(5, dtype=dtype, device=device)
print(t)

try:
    print(t.numpy())
except Exception as err_:
    print(err_)

tensor([1., 1., 1., 1., 1.], device='cuda:0')
can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

What we need to do instead¶

Pindahkan ke CPU dulu

In [24]:

t = torch.ones(5, dtype=dtype, device=device)
print(t)

try:
    print(t.cpu().numpy())
except Exception as err_:
    print(err_)

tensor([1., 1., 1., 1., 1.], device='cuda:0')
[1. 1. 1. 1. 1.]

Hati-hati Tensor (GPU) hanya bisa diaplikasikan ke fungsi di Torch¶

Alternativenya juggle ke numpy CPU (seperti di cell sebelumnya), tapi ini biasanya mempengaruhi performa.
Sebaiknya gunakan fungsi Torch saja.
Fungsi Torch juga wajib menggunakan parameter input tensor (error jika input = array)

In [25]:

t = torch.ones(5, dtype=dtype, device=device)

try:
    print("Numpy (CPU) Function: ", np.sin(t))
except Exception as err_:
    print(err_)
    
try:
    print("Torch Function: ", torch.sin(t))
except Exception as err_:
    print(err_)

can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
Torch Function:  tensor([0.8415, 0.8415, 0.8415, 0.8415, 0.8415], device='cuda:0')

In [26]:

# Torch Function only works for tensors
t = np.ones(5)
try:
    print("Torch Function: ", torch.sin(t))
except Exception as err_:
    print(err_)

sin(): argument 'input' (position 1) must be Tensor, not numpy.ndarray

In [27]:

# perbandingan logic tensor dan Float masih bisa dilakukan
t = torch.ones(5, dtype=dtype, device=device)

print("elemen terakhir t", t[-1])

try:
    print(t[-1]>2.0)
except Exception as err_:
    print(err_)

elemen terakhir t tensor(1., device='cuda:0')
tensor(False, device='cuda:0')

In [28]:

# Tapi walau t[-1] adalah sebuah nilai, tapi ini tetap tensor sehingga ... 
# Expected karena numpy ndak bisa bekerja di GPU (tensor)
try:
    print(np.isnan(t[-1]))
except Exception as err_:
    print(err_)

can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

In [29]:

print(t[-1].cpu().numpy())

1.0

Perhatikan bahwa konversi diatas adalah Copy memory (GPU & CPU)¶

Jika mau memindahkan Tensor dan mengeluarkannya dari Computational Graph-nya PyTorch maka gunakan perintah Detach
Detach juga digunaan jika tensor kita punya attribut Grad, dengan kata lain kita tidak bisa convert ke numpy jika tensornya punya attribut "grad".
Jika konsep grad membingungkan jangan hawatir, module setelah ini akan membahas hal ini.

In [30]:

t = t.cpu().detach().numpy()
t

Out[30]:

array([1., 1., 1., 1., 1.], dtype=float32)

Flatten .. Bukan karena Bumi itu datar :)¶

In [35]:

t = torch.tensor([[[1, 2],
                    [3, 4]],
                   [[5, 6],
                    [7, 8]]])
print(t, end='\n\n')
print(t.flatten().numpy())

tensor([[[1, 2],
         [3, 4]],

        [[5, 6],
         [7, 8]]])

[1 2 3 4 5 6 7 8]

Catatan Penting terakhir¶

Beberapa model DL yang ada terkadang "menipu", karena sebenarnya modelnya sensitive terhadap initial weights namun tidak dilaporkan.
Sebaiknya minimal selalu Gunakan Seed di awal code DL kita:
Nilai seed bebas, biasanya by default adalah 0.

In [13]:

np.random.seed(0)
torch.manual_seed(0)

Out[13]:

<torch._C.Generator at 0x1d288b743b0>

Akhir Modul DL 02 - Pendahuluan PyTorch untuk Deep learning¶

Referensi: https://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html#sphx-glr-beginner-blitz-tensor-tutorial-py

DL-02: Pengenalan Tensor di Deep Learning

Watch The Video Here: https://youtu.be/nLINKnoIl40

Open Code in Google Colab:

taudata Analytics

DL 02 - Pendahuluan PyTorch untuk Deep learning

https://taudataid.github.io/dl02/

Video Penjelasan:

Prasyarat Module ¶

Sebelum dilanjutkan, yakinkan sudah mempelajari dengan baik module:¶

Instalasi ¶

Why Torch? ¶

Tensor ¶

Penting untuk memahami Tipe Tensor¶

Warning on "Device" and "Dtype"¶

Contoh¶

Warning 2¶

What we should do instead:¶

Tapi kalau Tensor ada di GPU jadi agak beda¶

What we need to do instead¶

Hati-hati Tensor (GPU) hanya bisa diaplikasikan ke fungsi di Torch¶

Perhatikan bahwa konversi diatas adalah Copy memory (GPU & CPU)¶

Flatten .. Bukan karena Bumi itu datar :)¶

Catatan Penting terakhir¶

Akhir Modul DL 02 - Pendahuluan PyTorch untuk Deep learning¶

Tidak ada komentar:

Posting Komentar

SEARCH

LATEST

FOLLOW ME

ABOUT

Visitors

Search

Translate~Terjemahkan

Pages

Follow Us

Popular

Archive

Postingan Populer

Latest courses

Comments

About

Top Links Menu

DL-02: Pengenalan Tensor di Deep Learning

Watch The Video Here: https://youtu.be/nLINKnoIl40Open Code in Google Colab:

taudata Analytics

DL 02 - Pendahuluan PyTorch untuk Deep learning

https://taudataid.github.io/dl02/

Video Penjelasan:

Prasyarat Module ¶

Sebelum dilanjutkan, yakinkan sudah mempelajari dengan baik module:¶

Instalasi ¶

Why Torch? ¶

Tensor ¶

Penting untuk memahami Tipe Tensor¶

Warning on "Device" and "Dtype"¶

Contoh¶

Warning 2¶

What we should do instead:¶

Tapi kalau Tensor ada di GPU jadi agak beda¶

What we need to do instead¶

Hati-hati Tensor (GPU) hanya bisa diaplikasikan ke fungsi di Torch¶

Perhatikan bahwa konversi diatas adalah Copy memory (GPU & CPU)¶

Flatten .. Bukan karena Bumi itu datar :)¶

Catatan Penting terakhir¶

Akhir Modul DL 02 - Pendahuluan PyTorch untuk Deep learning¶

Tidak ada komentar:

Posting Komentar

SEARCH

LATEST

FOLLOW ME

ABOUT

Visitors

Search

Translate~Terjemahkan

Pages

Follow Us

Popular

Archive

Postingan Populer

Latest courses

Comments

About

Watch The Video Here: https://youtu.be/nLINKnoIl40

Open Code in Google Colab: