Machine-Learning | Haihao Sun's Homepage

D2L - Chapter 3.4 Linear Regression Implementation From Scratch

Sun, 20 Apr 2025 00:00:00 +0000

3.4 Linear Regression Implementation From Scratch

We are now realy to implement the entire method from scratch, including

model
loss function
a minibatch stochastic gradient descent optimizer
training function that stitches all these pieces together

3.4.1 Defining the Model

Initialize weights and bias.

class LinearRegressionScratch(d2l.module) #@save
 num_inputs: int
 lr: float
 sigma: float = 0.01

 def setup(self):
 self.w = self.param('w', nn.initializers.normal(self.sigma),
 (self.num_inputs, 1))
 self.b = self.param('b', nn.initializers.zeros, (1))

Define our model, relating its input and parameters to its output

@d2l.add_to_class(LinearRegressionScratch) #@save
def forward(self, X):
 return jnp.matmul(X, self.w) + self.b

3.4.2 Defining the Loss Funciton

@d2l.add_to_class(LinearRegressionScratch) #@save
def loss(self, params, X, y, state):
 y_hat = state.apply_fn({'params": params}, *X) # X unpacked from a tuple
 l = (y_hat - y.reshape(y_hat.shape)) ** 2 / 2
 return l.mean()

3.4.3 Defining the Optimization Algorithm

class SGD(d2l.HyperParameters): #@save

 def __init__(self, lr):
 self.save_hyperparameters()

 def init(self, params):
 del params
 return optax.EmptyState


 def update(self, updates, state, params=None):
 del params
 # When state.apply_gradients method is called to update flax's train_state object, it internally calls

 updates = jax.tree_util.tree_map(lambda g : -self.lr*g, updates)
 return updates

 def __call__():
 return optax.GradientTransformation(self.init,self.update)


@d2l.add_to_class(LinearRegressionScratch) #@save
def configure_optimizers(self):
 return SGD(self.lr)

3.4.4 Training

Initialize parameters (𝑤, 𝑏)
Repeat until done:
- Compute gradient 𝐠 ← ∂_(𝑤,𝑏) (1 / |𝔅|) ∑_𝑖∈𝔅 𝑙(𝐱^(𝑖), 𝑦^(𝑖), 𝑤, 𝑏)
- Update parameters (𝑤, 𝑏) ← (𝑤, 𝑏) − η 𝐠

@d2l.add_to_class(d2l.Trainer) #@save
def prepare_batch(self, batch):
 return batch


@d2l.add_to_class(d2l.Trainer) #@save
def fit_epoch(self):
 self.model.training = True
 if self.state.batch_stats:
 # Mutable states will be used later (e.g. for batch norm)
 for batch in self.train_dataloader:
 (_, mutated_vars), grads = self.model.training_step(self.state.params, self.prepare_batch(batch), self.state)
 self.state = self.state.apply_gradients(grads=grads)
 # can be ignored for models without dropout layers
 self.state = self.state.replace(dropout_rng=jax.random.split(self.state.dropout_rng[0]))
 self.state = self.state.replace(batch_stats=mutated_vars['batch_stats'])
 self.train_batch_idx += 1
 else:
 for batch in self.train_dataloader:
 _, grads = self.model.training_step(self.state.params, self.prepare_batch(batch), self.state)
 self.state = self.state.apply_gradients(grads=grads)
 # Can be ignored for models without dropout layers
 self.state = self.state.replace(dropout_rng=jax.random.split(self.state.dropot_rng)[0])
 self.train_batch_idx += 1

 if self.val_dataloader is None:
 return

 self.model.training = False
 for batch in self.val_dataloader:
 self.model.validation_step(self.state.params, self.prepare_batch(batch), self.state)
 self.val_batch_idx += 1

Testing with SyntheticRegressionData classs

model = LinearRegressionScratch(2, lr=0.03)
data = d2l.SyntheticRegressionData(w=jnp.array([2, -3.4]), b=4.2)
trainer = d2l.Trainer(max_epochs=3)
trainer.fit(model, data)

params = trainer.state.params
print(f"error in estimating w: {data.w - params['w'].reshape(data.w.shape)}")
print(f"error in estimating b: {data.b - params['b']}")

error in estimating w: [ 0.07147813 -0.19255161]
error in estimating b: [0.24309802]

3.4.5 Summary

We implemented the following:

dataloader
model
loss function
optimization procedure

D2L - Chapter 3.5 Concise Implementation of Linear Regression

Sun, 20 Apr 2025 00:00:00 +0000

3.5 Concise Implementation of Linear Regression

3.5.1 Defining the Model

class LinearRegression(d2l.Module): #@save
 lr: float
 def setup(self):
 self.net = nn.Dense(1, kernel_init=nn.initializers.normal(0.01))

@d2l.add_to_class(LinearRegression) #@save
def forward(self, X):
 return self.net(X)

3.5.2 Defining the Loss Function

@d2l.add_to_class(LinearRegression) #@save
def loss(self, params, X, y, state):
 y_hat = state.apply_fn({'params': params}, *X)
 return optax.l2_loss(y_hat, y).mean()

3.5.3 Defining the Optimization Algorithm

@d2l.add_to_class(LinearRegression) #@save
def configure_optimizers(self):
 return optax.sgd(self.lr)

3.5.4 Training

model = LinearRegression(lr=0.03)
data = d2l.SyntheticRegressionData(w=jnp.array([2, -3.4]), b=4.2)
trainer = d2l.Trainer(max_epochs=3)
trainer.fit(model, data)

@d2l.add_to_class(LinearRegression) #@save
def get_w_b(self, state):
 net = state.params['net']
 return net['kernel'], net['bias']

w, b = model.get_w_b(trainer.state)

D2L - Chapter 3.6 Generalization

Sun, 20 Apr 2025 00:00:00 +0000

3.6 Generalization

Exploring both what is known about the principles underlying generalization in various models
Also heuristic techniques that have been found (empirically) to yield improved generalization on tasks of practical interest.

3.6.1 Training Error and Generalization Error

3.6.1.1 Model Complexity

3.6.2 Underfitting or Overfitting

Underfitting, both error high
Overfitting, training error low, validation error high

3.6.2.1 Polynomial Curve Fitting

3.6.2.2 Dataset Size

3.6.3 Model Selection

3.6.3.1 Cross-validation

split into k
train on k - 1, validate on the remaining
average among all the errors

3.6.4 Summary

D2L - Chapter 3.7 Weight Decay

Sun, 20 Apr 2025 00:00:00 +0000

3.7 Weight Decay

3.7.1 Norms and Weight Decay

The total loss with weight decay is:

$$ \mathcal{L}_{\text{total}} = \mathcal{L}_{\text{original}} + \lambda \|\mathbf{w}\|^2 $$

Where:

$ \mathcal{L}\_{\text{original}} $ is the original loss (e.g., cross-entropy)
$ \lambda $ is the weight decay coefficient
$ \|\mathbf{w}\|^2 $ is the squared L2 norm of the weights

The weight update rule with weight decay for linear regression is:

$$ \mathbf{w} \leftarrow (1 - \eta \lambda)\mathbf{w} - \frac{\eta}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} \mathbf{x}^{(i)} \left( \mathbf{w}^\top \mathbf{x}^{(i)} + b - y^{(i)} \right) $$

3.7.2 High-Dimensional Linear Regression

3.7.3 Implementation From Scratch

3.7.3.1. Defining Norm Penalty

def l2_penalty(w):
 return (w ** 2).sum() / 2

3.7.3.2. Defining the Model


class WeightDecayScratch(d2l.LinearRegressionScratch):
 lambd: int = 0

 def loss(self, params, X, y, state):
 return (super().loss(params, X, y, state) +
 self.lambd * l2_penalty(params['w']))


data = Data(num_train=20, num_val=100, num_inputs=200, batch_size=5)
trainer = d2l.Trainer(max_epochs=10)

def train_scratch(lambd):
 model = WeightDecayScratch(num_inputs=200, lambd=lambd, lr=0.01)
 model.board.yscale='log'
 trainer.fit(model, data)
 print('L2 norm of w:',
 float(l2_penalty(trainer.state.params['w'])))

3.7.3.3 Training without Regularization

train_scratch(0)

train_scratch(3)

3.7.4. Concise Implementation

class WeightDecay(d2l.LinearRegression):
 wd: int = 0

 def configure_optimizers(self):
 # Weight Decay is not available directly within optax.sgd, but
 # optax allows chaining several transformations together
 return optax.chain(optax.additive_weight_decay(self.wd),
 optax.sgd(self.lr))

model = WeightDecay(wd=3, lr=0.01)
model.board.yscale='log'
trainer.fit(model, data)

print('L2 norm of w:', float(l2_penalty(model.get_w_b(trainer.state)[0])))

3.7.5. Summary

D2L - Chapter 3.1 Linear Regression

Sat, 19 Apr 2025 00:00:00 +0000

3.1 Linear Regression

3.1.1 Basics

3.1.1.1 Model

y = wx + b
search for the best w and b
measure of the quality of some given model (todo - 1)
a procedure for updating the model to improve its quality (todo - 2)

3.1.1.2 Loss Function

$$ L(\mathbf{w}, b) = \frac{1}{n} \sum_{i=1}^{n} l^{(i)}(\mathbf{w}, b) = \frac{1}{n} \sum_{i=1}^{n} \frac{1}{2}\!\left(\mathbf{w}^\top \mathbf{x}^{(i)} + b - y^{(i)}\right)^{2}. \tag{3.1.6} $$

3.1.1.3 Analytic Solution

$$ \partial_{\mathbf{w}} \| \mathbf{y} - \mathbf{Xw} \|^2 = 2 \mathbf{X}^\top (\mathbf{Xw} - \mathbf{y}) = 0 \quad \text{and hence} \quad \mathbf{X}^\top \mathbf{y} = \mathbf{X}^\top \mathbf{Xw}. \tag{3.1.8} $$

Solving for $\mathbf{w}$ provides us with the optimal solution for the optimization problem. Note that this solution

$$ \mathbf{w}^* = (\mathbf{X}^\top \mathbf{X})^{-1} \mathbf{X}^\top \mathbf{y} \tag{3.1.9} $$

3.1.1.4 Minibatch Stochastic Gradient Descent

Gradient Descent
Stochastic Gradient Descent (SGD)
MiniBatch SGD $$ \mathbf{w} \leftarrow \mathbf{w} - \frac{\eta}{|\mathcal{B}|} \sum_{i \in \mathcal{B}_t} \partial_{\mathbf{w}} l^{(i)}(\mathbf{w}, b) = \mathbf{w} - \frac{\eta}{|\mathcal{B}|} \sum_{i \in \mathcal{B}_t} \mathbf{x}^{(i)} \left( \mathbf{w}^\top \mathbf{x}^{(i)} + b - y^{(i)} \right) \tag{3.1.11} $$

$$ b \leftarrow b - \frac{\eta}{|\mathcal{B}|} \sum_{i \in \mathcal{B}_t} \partial_b l^{(i)}(\mathbf{w}, b) = b - \frac{\eta}{|\mathcal{B}|} \sum_{i \in \mathcal{B}_t} \left( \mathbf{w}^\top \mathbf{x}^{(i)} + b - y^{(i)} \right). $$

3.1.1.5 Predictions

Plug w and b to predict y

3.1.2 vectorization for Speed


jax
tensorflow
n = 10000
a = jnp.ones(n)
b = jnp.ones(n)
# JAX arrays are immutable, meaning that once created their contents
# cannot be changed. For updating individual elements, JAX provides
# an indexed update syntax that returns an updated copy
c = jnp.zeros(n)
t = time.time()
for i in range(n):
 c = c.at[i].set(a[i] + b[i])
f'{time.time() - t:.5f} sec'
/* 22s */

t = time.time()
d = a + b
f'{time.time() - t:.5f} sec'
/* 0.017s */

3.1.3 The Normal Distribution and Squared Loss

3.1.4 Linear Regression as a Neural Network

3.1.4.1 Biology

3.1.5 Summary

introduced/reviewed linear regression, the cornerstone of deep learning.

D2L - Chapter 3.2 Object-Oriented Design for Implementation

Sat, 19 Apr 2025 00:00:00 +0000

3.2 Object-Oriented Design for Implementation

Module contains models, losses, and optimization methods
DataModule provides data loaders for training and validation
Both classes are combined using Trainer class which allows us to train models on a variety of hardware platforms.

Most code in this book adapts Module and DataModule. We will touch upon the Trainer class only when we discuss GPUs, CPUs, parallel training, and optimization algorithms.

3.2.1 Utilities

def add_to_class(Class): #@save
 """Register functions as methods in created class."""
 def wrapper(obj):
 setattr(Class, obj.__name__, obj)
 return wrapper

class A:
 def __init__(self):
 self.b = 1

a = A()

@add_to_class(A)
def do(self):
 print('Class attribute "b" is', self.b)

a.do()
# Class attribute "b" is 1



class HyperParameters: #@save
 """The base class of hyperparameters."""
 def save_hyperparameters(self, ignore=[]):
 raise NotImplemented
# Call the fully implemented HyperParameters class saved in d2l
class B(d2l.HyperParameters):
 def __init__(self, a, b, c):
 self.save_hyperparameters(ignore=['c'])
 print('self.a =', self.a, 'self.b =', self.b)
 print('There is no self.c =', not hasattr(self, 'c'))

b = B(a=1, b=2, c=3)

class ProgressBoard(d2l.HyperParameters): #@save
 """The board that plots data points in animation."""
 def __init__(self, xlabel=None, ylabel=None, xlim=None,
 ylim=None, xscale='linear', yscale='linear',
 ls=['-', '--', '-.', ':'], colors=['C0', 'C1', 'C2', 'C3'],
 fig=None, axes=None, figsize=(3.5, 2.5), display=True):
 self.save_hyperparameters()

 def draw(self, x, y, label, every_n=1):
 raise NotImplemented

board = d2l.ProgressBoard('x')
for x in np.arange(0, 10, 0.1):
 board.draw(x, np.sin(x), 'sin', every_n=2)
 board.draw(x, np.cos(x), 'cos', every_n=10)

3.2.2 Models

The Module class is the base class of all models that we will implement. At the very least we need three models. The first, init stores the learnable parameters, the training_step method accepts a data batch to return the loss value, and finally, configure_optimizers returns the optimization method, or a list of them, that is used to update the learnable parameters. Optionally, we can define validation_steps to report the evaluation measures. Sometimes we put the code for computing the output into a separate of forward method to make it more resubale.

With the introduce of dataclasses in Python3.7, classes decorated with @dataclass automatically add magic methods such as init and repr. The member variables are defined using type annotations. All Flax modules are Python 3.7 dataclasses.

class Module(nn.Module, d2l.HyperParameters): #@save

 """The base class of models"""
 # No need for save_hyperparam when using Python dataclass.
 plot_train_per_epoch: int = field(default=2, init=False)
 plot_valid_per_epoch: int = field(default=1, init=False)
 # Use default_factory to make sure new plots are generated on each run.
 board: ProgressBoard = field(default_factory=lambda: ProgressBoard(), init=False)

 def loss(self, y_hat, y):
 raise NotImplementedError

 # JAX & Flax do not have a forward-method-like syntax. Flax uses setup
 # and built-in __call__ magic methods for forward pass. Adding here
 # for consistency

 def forward(self, X, *args, **kwargs):
 assert hasattr(self, 'net'), 'Neural network is defined'
 return self.net(X, *args, **kwargs)


 def plot(self, key, value, train):
 """Plot a point in animation."""
 assert hasattr(self, 'trainer'), 'Trainer is not inited'
 self.board.xlabel = 'epoch'
 if train:
 x = self.trainer.train_batch_idx / \
 self.trainer.num_train_batches
 n = self.trainer.num_train_batches / \
 self.plot_train_per_epoch
 else:
 x = self.trainer.epoch + 1
 n = self.trainer.num_val_batches / \
 self.plot_valid_per_epoch
 self.board.draw(x, jax.device_put(value, d2l.cpu()),
 ('train_' if train else 'val_') + key,
 every_n=int(n))

 def training_step(self, params, batch, state):
 l, grads = jax.value_and_grad(self.loss)(params, batch[:-1],
 batch[-1], state)
 self.plot("loss", l, train=True)
 return l, grads

 def validation_step(self, params, batch, state):
 l = self.loss(params, batch[:-1], batch[-1], state)
 self.plot('loss', l, train=False)

 def apply_init(self, dummy_input, key):
 """To be defined later in :numref:`sec_lazy_init`"""
 raise NotImplementedError

 def configure_optimizers(self):
 raise NotImplementedError

3.2.3 Data

The Datamodule class is the base class for data. Quite frequently the init method is used to prepare the data.

class DataModule(d2l.HyperParameters): #@save
 """The base class of data."""
 def __init__(self, root='../data'):
 self.save_hyperparameters()

 def get_dataloader(self, train):
 raise NotImplementedError

 def train_dataloader(self):
 return self.get_dataloader(train=True)

 def val_dataloader(self):
 return self.get_dataloader(train=False)

3.2.4 Training

class Trainer(d2l.HyperParameters): #@save
 """The base class for training models with data."""
 def __init__(self, max_epochs, num_gpus=0, gradient_clip_val=0):
 self.save_hyperparameters()
 assert num_gpus == 0, 'No GPU support yet'

 def prepare_data(self, data):
 self.train_dataloader = data.train_dataloader()
 self.val_dataloader = data.val_dataloader()
 self.num_train_batches = len(self.train_dataloader)
 self.num_val_batches = (len(self.val_dataloader)
 if self.val_dataloader is not None else 0)

 def prepare_model(self, model):
 model.trainer = self
 model.board.xlim = [0, self.max_epochs]
 self.model = model

 def fit(self, model, data, key=None):
 self.prepare_data(data)
 self.prepare_model(model)
 self.optim = model.configure_optimizers()

 if key is None:
 root_key = d2l.get_key()
 else:
 root_key = key
 params_key, dropout_key = jax.random.split(root_key)
 key = {'params': params_key, 'dropout': dropout_key}

 dummy_input = next(iter(self.train_dataloader))[:-1]
 variables = model.apply_init(dummy_input, key=key)
 params = variables['params']

 if 'batch_stats' in variables.keys():
 # Here batch_stats will be used later (e.g., for batch norm)
 batch_stats = variables['batch_stats']
 else:
 batch_stats = {}

 # Flax uses optax under the hood for a single state obj TrainState.
 # More will be discussed later in the dropout and batch
 # normalization section
 class TrainState(train_state.TrainState):
 batch_stats: Any
 dropout_rng: jax.random.PRNGKeyArray

 self.state = TrainState.create(apply_fn=model.apply,
 params=params,
 batch_stats=batch_stats,
 dropout_rng=dropout_key,
 tx=model.configure_optimizers())
 self.epoch = 0
 self.train_batch_idx = 0
 self.val_batch_idx = 0
 for self.epoch in range(self.max_epochs):
 self.fit_epoch()

 def fit_epoch(self):
 raise NotImplementedError

3.2.5 Summary

This object-oriented design provides moduality.

D2L - Chapter 3.3 Synthetic Regression Data

Sat, 19 Apr 2025 00:00:00 +0000

3.3 Synthetic Regression Data

3.3.1 Generating the Dataset

generate the fake data


class SyntheticRegressionData(d2l.DataModule): #@save
 """Synthetic data for linear regression."""
 def __init__(self, w, b, noise=0.01, num_train=1000, num_val=1000,
 batch_size=32):
 super().__init__()
 self.save_hyperparameters()
 n = num_train + num_val
 key = jax.random.PRNGKey(0)
 key1, key2 = jax.random.split(key)
 self.X = jax.random.normal(key1, (n, w.shape[0]))
 noise = jax.random.normal(key2, (n, 1)) * noise
 self.y = jnp.matmul(self.X, w.reshape((-1, 1))) + b + noise

3.3.2 Reading the Dataset

@d2l.add_to_class(SyntheticRegressionData)
def get_dataloader(self, train):
 if train:
 indices = list(range(0, self.num_train))
 # The examples are read in random order
 random.shuffle(indices)
 else:
 indices = list(range(self.num_train, self.num_train+self.num_val))
 for i in range(0, len(indices), self.batch_size):
 batch_indices = jnp.array(indices[i: i+self.batch_size])
 yield self.X[batch_indices], self.y[batch_indices]

3.3.3 Concise Implementation of the Data Loader

@d2l.add_to_class(d2l.DataModule) #@save
def get_tensorloader(self, tensors, train, indices=slice(0, None)):
 tensors = tuple(a[indices] for a in tensors)
 # Use Tensorflow Datasets & Dataloader. JAX or Flax do not provide
 # any dataloading functionality
 shuffle_buffer = tensors[0].shape[0] if train else 1
 return tfds.as_numpy(
 tf.data.Dataset.from_tensor_slices(tensors).shuffle(
 buffer_size=shuffle_buffer).batch(self.batch_size))

@d2l.add_to_class(SyntheticRegressionData) #@save
def get_dataloader(self, train):
 i = slice(0, self.num_train) if train else slice(self.num_train, None)
 return self.get_tensorloader((self.X, self.y), train, i)

3.3.4 Summary

D2L - Chapter 2.6 Probablity and Statistics + 2.7 Documentation

Sat, 12 Apr 2025 00:00:00 +0000

Probablity and Statistics

Covariance Matrix, Diagonal & Off-Diagonal Entries

Covariance Matrix
A covariance matrix $\Sigma$ is a square matrix that shows the covariance between each pair of coordinates in a random vector. It is:
- Symmetric
- Positive semidefinite
Diagonal Entries
- Each diagonal entry $\Sigma\_{ii}$ is the variance of the $i$-th coordinate, denoted $\mathrm{Var}(x_i)$.
- Variance measures how spread out the $i$-th coordinate is around its mean.
Off-Diagonal Entries
- Each off-diagonal entry $\Sigma\_{ij}$ (with $i \neq j$) is the covariance $\mathrm{Cov}(x_i, x_j)$.
- Positive covariance means the two coordinates tend to vary in the same direction (above/below their means together), negative covariance indicates they move in opposite directions, and zero covariance indicates no linear co-movement.
Why Symmetric?
- Because $\mathrm{Cov}(x_i, x_j) = \mathrm{Cov}(x_j, x_i)$, the matrix $\Sigma$ must be symmetric.

Documentation


print(dir(jax.random))
help(jax.numpy.ones)

D2L - Chapter 2.4 Calculus + 2.5 Automatic Differentiation

Wed, 09 Apr 2025 00:00:00 +0000

2.4 Calculus

2.4.1 Derivatives and Differentiation

2.4.2 Visualization Utilities

2.4.3 Partial Derivatives and Gradients

When there is no ambiguity, ∇ₓf(x) is typically replaced by ∇f(x). The following rules come in handy for differentiating multivariate functions:

For all A ∈ ℝᵐˣⁿ we have ∇ₓ(Ax) = Aᵀ and ∇ₓ(xᵀA) = A.
For square matrices A ∈ ℝⁿˣⁿ we have that ∇ₓ(xᵀAx) = (A + Aᵀ)x and in particular
∇ₓ‖x‖² = ∇ₓ(xᵀx) = 2x.

2.4.4 Chain Rule

2.4.5 Discussion

Automatic differentiation allows us to compute gradients systematically using composition rules, freeing up cognitive effort for higher-level tasks.
Gradient computation for vector-valued functions involves matrix multiplication, following the chain of variable dependencies.
Dependency graphs:
- Evaluated forward to compute function outputs.
- Traversed backward to compute gradients (backpropagation).
Backpropagation is a computational application of the chain rule, introduced formally in later chapters.
In optimization, gradients indicate how to adjust model parameters to reduce the loss.
Every step of an optimization algorithm relies on computing the gradient.

2.5 Automatic Differentiation

2.5.1 A simple function

from jax import grad
x = jnp.arange(4.0)
x
y = lambda x: 2 * jnp.dot(x, x)
y(x)
# The `grad` transform returns a Python function that
# computes the gradient of the original function
x_grad = grad(y)(x)
x_grad

2.5.2 Backward for Non-Scalar Variables

2.5.3 Detaching Computation

move some calculations outside the recorded computational graph

2.5.4. Gradients and Python Control Flow

2.5.5 Discussion

For now, try to remember these basics:

(i) attach gradients to those variables with respect to which we desire derivatives;
(ii) record the computation of the target value;
(iii) execute the backpropagation function;
(iv) access the resulting gradient.

D2L - Chapter 2.2 Data Preprocessing + 2.3 Linear Algebra

Tue, 08 Apr 2025 00:00:00 +0000

2.2 Data Preprocessing

2.2.1 Reading the Dataset

2.2.2 Data Preparation

inputs = pd.get_dummies(inputs, dummy_na=True)
inputs = inputs.fillna(inputs.mean())

2.2.3 Conversion To the Tensor Format

from jax import numpy as jnp

X = jnp.array(inputs.to_numpy(dtype=float))
y = jnp.array(targets.to_numpy(dtype=float))
X, y

2.3 Linear Algebra

2.3.1 Scalars

x = jnp.array(3.0)
y = jnp.array(2.0)

2.3.2 Vectors

x = jnp.arange(3)
x
x[2]
len(x)
x.shape

2.3.3 Matrices

A = jnp.arange(6).reshape(3, 2)
A
A.T
A = jnp.array([[1, 2, 3], [2, 0, 4], [3, 4, 5]])
A == A.T

2.3.4 Tensors

jnp.arange(24).reshape(2, 3, 4)

2.3.5 Basic Properties of Tensor Arithmetic

element-wise addition
element-wise multiplication (Hadamard Product)
element * scalar

2.3.6 Reduction

By default, invoking the sum function reduces a tensor along all of its axes, eventually producing a scalar. Our libraries also allow us to [specify the axes along which the tensor should be reduced.] To sum over all elements along the rows (axis 0), we specify axis=0 in sum. Since the input matrix reduces along axis 0 to generate the output vector, this axis is missing from the shape of the output.

A.shape, A.sum(axis=0).shape
C = jnp.arange(24).reshape(2, 3, 4)
C
C.sum(axis=2)

2.3.7 Non-Reduction Sum

print(A)
sum_A = A.sum(axis=1, keepdims=True)
sum_A, sum_A.shape
[[0. 1. 2.]
 [3. 4. 5.]]
(Array([[ 3.],
 [12.]], dtype=float32),
 (2, 1))

A.cumsum(axis=0)

2.3.8 Dot Products

weighted sum/ weighted average…

y = jnp.ones(3, dtype = jnp.float32)
x, y, jnp.dot(x, y)

2.3.9 Matrix-Vector Products

2.3.10 Matrix-Matrix Products

nk X km -> n * m dot products

2.3.11 Norms

u = jnp.array([3.0, -4.0])
jnp.linalg.norm(u)
jnp.linalg.norm(u, ord=1) # same as jnp.abs(u).sum()
jnp.linalg.norm(jnp.ones((4, 9)))

2.3.12 Discussions

Scalars, vectors, matrices, and tensors are the basic mathematical objects used in linear algebra and have zero, one, two, and an arbitrary number of axes, respectively.
Tensors can be sliced or reduced along specified axes via indexing, or operations such as sum and mean, respectively.
Elementwise products are called Hadamard products. By contrast, dot products, matrix–vector products, and matrix–matrix products are not elementwise operations and in general return objects having shapes that are different from the the operands.
Compared to Hadamard products, matrix–matrix products take considerably longer to compute (cubic rather than quadratic time).
Norms capture various notions of the magnitude of a vector (or matrix), and are commonly applied to the difference of two vectors to measure their distance apart.
Common vector norms include the l1 and l2 norms, and common matrix norms include the spectral and Frobenius norms.

D2L - Chapter 2.1 Data Manipulation

Mon, 07 Apr 2025 00:00:00 +0000

colab

2.1.1 Get Started

import jax
from jax import numpy as jnp
x = jnp.arange(12)
x -> Array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], dtype=int32)
x.size -> 12
x.shape -> (12, )
X = x.reshape(3, 4)
X
jnp.zeros((2, 3, 4))
jnp.ones((2, 3, 4))
# Any call of a random function in JAX requires a key to be
# specified, feeding the same key to a random function will
# always result in the same sample being generated
jax.random.normal(jax.random.PRNGKey(0), (3, 4))
jnp.array([[2, 1, 4, 3], [1, 2, 3, 4], [4, 3, 2, 1]])

Personal Note

shape -> (axis-0, axis-1, …)
reshape must be multiple

2.1.2 Indexing and Slicing

X[-1], X[1:3]
# JAX arrays are immutable. jax.numpy.ndarray.at index
# update operators create a new array with the corresponding
# modifications made
X_new_1 = X.at[1, 2].set(17)
X_new_1
X_new_2 = X_new_1.at[:2, :].set(12)
X_new_2

2.1.3 Operations

jnp.exp(x)
x = jnp.array([1.0, 2, 4, 8])
y = jnp.array([2, 2, 2, 2])
x + y, x - y, x * y, x / y, x ** y
X = jnp.arange(12, dtype=jnp.float32).reshape((3, 4))
Y = jnp.array([[2.0, 1, 4, 3], [1, 2, 3, 4], [4, 3, 2, 1]])
# axis sets which direction to stack, axis = 0, stacks on the row, meaning will have 6 in axis-0
jnp.concatenate((X, Y), axis=0), jnp.concatenate((X, Y), axis=1)

X == Y
X.sum()

Personal Note

axis sets which direction to stack. For example, axis = 0, stacks on the row, meaning will have 6(3 + 3) items in axis-0.

2.1.4 Broadcasting

By now, you know how to perform elementwise binary operations on two tensors of the same shape. Under certain conditions, even when shapes differ, we can still [perform elementwise binary operations by invoking the broadcasting mechanism.] Broadcasting works according to the following two-step procedure: (i) expand one or both arrays by copying elements along axes with length 1 so that after this transformation, the two tensors have the same shape; (ii) perform an elementwise operation on the resulting arrays.

a = jnp.arange(3).reshape((3, 1))
b = jnp.arange(2).reshape((1, 2))
a,
a + b

Personal Note

broadcasting copies data to reach lcm and do computation.

2.1.5 Conversion to Other Python Objects

a = jnp.array([3.5])
a
a.item()
float(a) 🙅
int(a) 🙅

D2L - Introduction - 2/2

Mon, 07 Apr 2025 00:00:00 +0000

1.3.2 UnSupervised and Self-Supervised Learning

The author lists the difference between traditional software engineering and machine learning. While software engineering focuses on solving problems with rule-based solutions, machine learning(deep learning) tries to solve problems that are ambiguous and the solution can evlove by itself after seeing more examples.

Clustering
PCA
Relationship, Rome - Italy = France - Paris
causality and probabilistic graphical models

1.3.3 Interacting with an Environment

1.3.4 Reinforcement Learning

1.4 Roots

1.5 The road to deep learning

1.6 Success Stories

1.7 The essence of deep learning

Thus, one key advantage of deep learning is that it replaces not only the shallow models at the end of traditional learning pipelines, but also the labor-intensive process of feature engineering.

1.8 Summary

My takeaway

Chapter1. Introduction mainly discussed machine learning from data, definition, examples, history, current success, provided foundation to future studies.

D2L - Introduction

Wed, 02 Apr 2025 00:00:00 +0000

1.0 Introduction

1.1 A Motivating Example

The author lists an example of a wake word system where we feed model with data and let the model learn.

1.2 Key Components

The data that we can learn from.
A model of how to transform the data.
An objective function that quantifies how well or badly the model is doing.
An algorithm to adjust the model’s parameters to optimize the objective function.

1.2.1 Data

Machine learning relies on data, which consists of examples represented numerically through features (inputs) and, in supervised learning, labels (targets). Examples include images or medical records, which can be structured as fixed-length vectors—but not all data fits this format, especially text or variable-sized inputs. Deep learning models handle such complexity more effectively than traditional methods. However, more data generally leads to better models, especially in deep learning, where large datasets are often essential. Crucially, the quality and representativeness of the data matter just as much as quantity—biased or flawed data can lead to harmful and unfair outcomes, even without malicious intent.

1.2.2 Models

Most machine learning involves transforming input data to make predictions. A model refers to the system that performs this transformation—taking data of one type and outputting another, such as predicting emotions from photos or detecting anomalies in sensor readings. While simple models can solve basic problems, more complex challenges often require more powerful tools. Deep learning stands out by using layered models that apply many transformations in sequence, hence the term deep. This approach enables tackling problems that push beyond the limits of classical methods, though traditional techniques still play a role along the way.

1.2.3 Objective function

In machine learning, learning means improving performance at a task over time, which requires a clear way to measure improvement. These measurements are called objective functions or loss functions, which we usually define so that lower values are better. Common examples include squared error for regression and error rate for classification. Some losses are hard to optimize directly, so we often use surrogate objectives that are easier to work with.

We train models by minimizing loss on a training dataset, treating the data as fixed and adjusting model parameters. However, strong performance on training data doesn’t guarantee success on new, unseen data. To evaluate generalization, we use a separate test dataset. If a model performs well on training data but poorly on test data, it’s said to be overfitting—essentially memorizing rather than learning.

1.2.4 Optimization Algorithms

ChatGPT said: Once we have data, a model, and an objective function, the next step is to optimize the model’s parameters to minimize the loss. In deep learning, this is typically done using gradient descent. This algorithm works by computing how a small change in each parameter would affect the loss, then adjusting the parameters in the direction that reduces the loss. Repeating this process iteratively helps the model gradually improve its performance.

1.3 Kinds of Machine Learning Problems

1.3.1 Supervised Learning

1.3.1.1 Regression

Question: How many?
Loss: squared error

1.3.1.2 Classification

Question: Which one?
Loss: cross-entropy

1.3.1.3 Tagging

Question: multi-label classification

1.3.1.4 Search

Question: releance scores

1.3.1.5 Recommender System

1.3.1.6 Sequence Learning

They require a model either to ingest sequences of inputs or to emit sequences of outputs (or both).
Cases

Tagging and Parsing: tag named entity
Automatic Speech Recognition: challenge is that input is way longer than output
Text to Speech: This is the inverse of asr.
Machine Translation: input/output might not align in the same order.

CS224n-NLP-lecture 1, 2

Fri, 29 Nov 2024 00:00:00 +0000

Word2Vector

How

Represent words by their context (Distributional semantics): A word’s meaning is given by words frequently appear close-by, within a fixed-size.
Use the many contexts of w to build up a representation of w.
We build a dense vector for each word, chosen so that it is similar to vectors of words that appear in similar context.
Note that word vectors are also called word embeddings, or neural word representations. They are a distributed representation.

Word2Vector idea

We have a large body of text (corpus).
Every word in a fixed vocabulary is represented by vector.
Go through each position t in the text, which has a center word (c) and a context word (o).
Use similarity of word vectors for c and o, calculate the probablity of o given c.
Keep adjusting the word vector to maximize this probablity.

Math

$L(\theta) = \prod_{t=1}^T \prod_{\substack{-m \leq j \leq m \\ j \neq 0}} P(w_{t+j} \mid w_t; \theta)$
$J(\theta) = -\frac{1}{T} \log L(\theta) = -\frac{1}{T} \sum_{t=1}^T \sum_{\substack{-m \leq j \leq m \\ j \neq 0}} \log P(w_{t+j} \mid w_t; \theta)$
$P(o \mid c) = \frac{\exp(u_o^T v_c)}{\sum_{w \in V} \exp(u_w^T v_c)}$
- Exponentiation makes anything positive
- Dot product compares similarity of o and c.
- Normalize over entire vocabulary to give probability distribution

Gradient Descent and Stochastic Gradient Descent

Problem:

J is a function of all windows in the corpus, gradient is expensive to compute.

Solution:

SGD Stochastic Gradient Descent
SGD in each window: we only have at most 2m + 1 words
Either we need sparse matrix updates operations to only update certain rows of full embedding metrics U and V, or we need to keep around a hash of word vector. Millions of word vectors allow us to do distributed computing.

Word2Vector Algo Family

Two models:

Skip-gram(SG), predict context (outside) words given center words.
Continuous bag of words (CBOW), predict context words from context words.

Negative Sampling

Problem

The normalization demonitator is computionally expensive.

Solution

Train a binary logistic regression for a true pair vs noise pair.
$J(\theta) = \frac{1}{T} \sum_{t=1}^{T} J_t(\theta)$
$J_t(\theta) = \log \sigma \left( u_o^\top v_c \right) + \sum_{i=1}^{k} \mathbb{E}_{j \sim P(w)} \left[ \log \sigma \left( -u_j^\top v_c \right) \right]$
$\sigma(x) = \frac{1}{1 + e^{-x}}$

Take k negative samples
Maxmize probablity that real words appear, minimize probablity that random words appear around the center words.
Sample with P(w) = u(w)^(3/4)

Co-occurence counting

Problems:

vector increase in sizes
Subsequent classification models have sparsity issues.

Singular Value Decomposition

X = USV

GloVe

Crucial Insight: Ratios of co-occurence probablities can encode meaning components (meaning component means going from male to female etc).

$J = \sum_{i,j=1}^{V} f(X_{ij}) \left( w_i^\top \tilde{w}_j + b_i + \tilde{b}_j - \log X_{ij} \right)^2$

$X\_{ij}$: The co-occurrence count between words $ i $ and $ j $.
$ w_i $: The word vector for the word $ i $.
$ \tilde{w}\_j $: The context word vector for the word $ j $.
$ b_i $: The bias term for the word $ i $.
$ \tilde{b}\_j $: The bias term for the context word $ j $.
$ \log X*{ij} $: The logarithm of the co-occurrence count $ X*{ij} $.
$ f(X\_{ij}) $: A weighting function to limit the influence of very frequent or very infrequent word pairs.
$ x\_{\text{max}} $: A maximum co-occurrence count threshold.

How to evaluate

Intrinsics

Definition

Evaluation on a specific subtask.
Fast to compute.
Helps to understand the system.
Not clear if it really helps unless correlation to real task is established.

Example

Word Vector Analogy

man:woman :: king:?
Evaluate word vectors by how will their cosine distance after addition captures intuitive semantics and syntatic analogy questions.
Discard the input words from search

WordSim human judgement

Extrinsics

Definition

Evaluation on a real task.
Can take a long time to compute.
Unclear if the subsystem is the probelm or its interaction or other subsystems.
If replacing exactly one subsystem is making progress, that’s good.

Example

Named Entity recognition.

Word sense

A word can have multiple meanings.

Recommender System Study Notes - Nomination(6/6)

Fri, 06 Sep 2024 00:00:00 +0000

GeoHash Nomination
Author Nomination
- Subscribed Author
- Interacted Author
- Similar Author
Cache Nomination

GeoHash Nomination

Indexing
- GeoHash -> Item with good quality(reverse chronological order)
No personalized

Subscribed Author Nomination

Indexing:
- User -> subscribed authors
- author -> uploaded items
Nomination:
- User -> Subscribed Author -> New Item

Interacted Author Nomination

If someone likes/love/forward an item
Indexing:
- User -> Interacted Author (update it)
Nomination:
- User -> Interacted Author -> New Item

Similar Author Nomination

Indexing:
- Author -> Similar Author
Nomination:
- User ID -> Liked Author -> Similar Author -> New Item

Cache Nomination

Intuition: Use the previous ranking winning candidates discarded by re-ranking.
Top 50 items, but get discarded by re-ranking, go to nomination.
Cache is fixed. How to clear the cache
- Got impressed
- At most 10 times
- At most 3 days

Bloom Filter

Intuition

If user has seen an item, then we do not recommend this item to user
For each user, record item that has been recommended(maybe for a month)
For each item, check if it’s been recommended before

Basics

The Bloom filter determines whether an item ID is in the set of exposed items.
If the judgment is “no”, then the item is definitely not in the set.
If the judgment is “yes”, then the item is likely in the set. (There may be false positives, where an unexposed item is incorrectly judged as exposed and thus filtered out.)

The size of the exposed item set is $ n $, the dimensionality of the binary vector is $ m $, and $ k $ hash functions are used.
The false positive probability of the Bloom filter $ \delta $ is approximately:
\[ \delta \approx \left( 1 - \exp\left( \frac{-kn}{m} \right) \right)^k \]
As $ n $ increases, the more 1’s there are in the vector, and the higher the false positive probability. (The probability that all $ k $ positions corresponding to unexposed items are set to 1 increases.)
As $ m $ increases, the vector becomes longer, making hash collisions less likely.
If $ k $ is too large or too small, it is not ideal; $ k $ has an optimal value.

Exposed Filtering Path:

 +-------------+ +---------+ +-------------+
 | Recall 1 |------------> | Sorting |------------> | Item 1 |
 +-------------+ +---------+ +-------------+
 | Recall 2 | |
 +-------------+ |
 | Recall 3 | |
 +-------------+ v
 +-------------+
 | Item 2 |
 +-------------+
 | ... |
 +-------------+
 | Item q |
 +-------------+
 ^ |
 | |
+------------------+ | |
| Binary Vector |<-+ |
+------------------+ v
 +-------------------+ +-----------------------+
 | Bloom Filter |<----| Real-time Processing |
 | (Exposure Filter) | | (Kafka + Flink) |
 +-------------------+ +-----------------------+

Recommender System Study Notes - Nomination(5/6)

Thu, 05 Sep 2024 00:00:00 +0000

Recommender System Study Notes - Nomination(4/6)

Wed, 04 Sep 2024 00:00:00 +0000

Two tower model full update/incremental update.
- Online Nomination: Online compute user ID, ANN in stored vector database with item.
- Full Update: 1 epoch, 1 day.
- Incremental Update: Only upadte ID embedding, frozen the fully connected layer.
Two tower model Self-supervised learning
- Problem: Two tower cannot learn item with limited amount of clicks (data biased).
- Self-supervised learning: random feature transformation
- bi’, bi’’ high similarity
- bi’, bj’’ low similarity
- Training:
  - pick n (user - item) as a batch for training.
  - m items as a batch(uniform sampling) for self-supervised learning.
  - $\frac{1}{n} \sum_{i=1}^{n} L_{\text{main}}[i] + \alpha \cdot \frac{1}{m} \sum_{j=1}^{m} L_{\text{self}}[j]$

Online Nomination

Implementation

Item
- Use neural network to compute item b’s embedding.
- Store them in vectorized databse (Milvus, Faiss, HnswLib).
- Indexing, for ANN search.
User
- Use user ID, feature, etc to compute user a’s embedding.
- ANN search
  - Use a as query.
  - Return the top k as return.
- Why store item but compute user online?
  - User is only 1, item is O(b).
  - User often changes quickly.

Model Update

Full Update VS Incremental Update

Full Update
- Use yesterday’s tfrecord, train 1 epoch.
- Update the new user neural network, store item data into database.
- Easy to implement, easy requirement for the system.
Incremental Update(online learning)
- Why? User’s interest can change quickly.
- Data streaming, TFRecord.
- Online learning, gradient descent, only update ID Embedding, not other parameters in the neural network.
- Update the new ID embedding.
- To be discarded after full update.
Why only incremental update is not having the best result?
- Hourly data is biased.
- Full update: random shuffle, 1 epoch.
- Incremental update: train from morning to night, this is not having the best result.
- random shuffle makes a positive impact on the learning.

Two tower model Self-supervised learning

Problem

The head effect in the recommendation system is severe:
- A small number of items account for most of the clicks.
- Most items have a low number of clicks.
The features of high-click items are well-learned, but the features of long-tail items are not well-learned.
Self-supervised learning: data augumentation, to better learn long-tail items’ embedding.

Intuition

A simple feature transformation will make minimal impact on their feature embeddings, and pairwise, the similarity still remains large.
i -> i’ -> bi’, i -> i’’ -> bi’’, j -> j’ -> bj’, j -> j’’->bj''
i, bi’ and bi’’ has high similarity
i, j, bi’ and bj’’ has low similarity
cos(bi, bi’’) big, cos(bi’, bj’) small

Methods

Random Mask

Randomly pick labels, and set them to null.

Dropout

Randomly dropout 50% label.

Complementary

ID, Label, Keywords, City.
Randomly divide them into two {ID, keyword} and {Label, City}.
{ID, default, Keyword, default} -> Item embedding.
{default, Label, default, City} -> Item embedding.

Mask a set of correlated features

U = {Male, Female, Unisex}.
V = {Photography, Football, IT, Makeup}.
u = female, v = Makeup, p(u, v) is high.
u = female, v = IT, p(u, v) is low.
$MI(U, V) = \sum_{u \in U} \sum_{v \in V} p(u, v) \cdot \log \frac{p(u, v)}{p(u) \cdot p(v)}$

Learning:

Assume there are a total of $ k $ features. Calculate the MI (Mutual Information) between any two features offline, resulting in a $ k \times k $ matrix.
Randomly select one feature as a seed and find the $ k/2 $ features most related to the seed.
Mask the seed and its related $ k/2 $ features, retaining the remaining $ k/2 $ features.

Training:

Sampling m items as a batch
cos(bi’, bi’’) -> 1, other cos(bi’, bj’’) -> 0

Recommender System Study Notes - Nomination(3/6)

Tue, 03 Sep 2024 00:00:00 +0000

Two Tower Model - Architecture + Training
- Pointwise Training
  - View nomination as binary classification
  - Sample: cos(a,b) -> +1, negative sample: cos(a, b) -> -1
  - Positive : negative = 1:2 / 1:3
- Pairwise Training
  - Intuntion: cos(a, b+) > cos(a, b-)
  - Triplet hinge loss = max(0, cos(a, b-) + m - cos(a, b+))
  - Triplet logistic loss = log(1 + exp(σ) * cos(a, b-) - cos(a, b+))
- Listwise Training
  - CrossEntropyLoss
Two Tower Model - Positive Samples + Negative Samples
- Positive Sample:
  - Impressed and clicked
- Negative Sample:
- Easy negative sample:
  - All items
  - In-batch items
- Hard negative sample:
  - Nominated, but discarded by ranking
- Mix easy and hard for training dataset
- Wrong sampling:
  - Impressed, but not clicked (can be used for training ranking but not nomination)

Two Tower Model(DSSM) - Architecture and Training

Two Tower Model Architecture

Neural Network
 └── Concatenate
 ├── Embedding Layer (User ID)
 │ └── Embedding (Vector Representation)
 ├── Embedding Layers (User Discrete Feature)
 │ └── Embedding (Vector Representation)
 └── Normalization, Binning, etc. (normalize to 1, put into buckets)
 └── User Continuous Features (Vector Representation)

$\cos(a, b) = \frac{a \cdot b}{\|a\|_2 \cdot \|b\|_2}$

 Cosine Similarity:
 / \
 / \
 a b
 / \
 Neural Network Neural Network
 / \
 Feature Transformation Feature Transformation
 / \
User ID, Discrete Features, Product ID, Discrete Features,
Continuous Features Continuous Features

Two Tower Model Training

Pointwise (binary classification)
Pairwise
Listwise

Pointwise Training

Intuition: View nomination as binary classification
positive sample: cos(a,b) -> +1, negative sample: cos(a, b) -> -1
positive : negative = 1:2 / 1:3

Pairwise Training

 cos(a, b⁺) cos(a, b⁻)
 / \ / \
 / \ / \
 b⁺ a b⁻
 (Positive Sample) (User) (Negative Sample)
 | | |
 Neural Network Neural Network Neural Network
 | | |
Feature Transformation Feature Transformation Feature Transformation
 | | |
 Positive Item Feature User Features Negative Item Feature

Intuntion: cos(a, b+) > cos(a, b-)
triplet hinge loss = max(0, cos(a, b-) + m - cos(a, b+))
triplet logistic loss = log(1 + exp(σ) * cos(a, b-) - cos(a, b+))

Listwise Training

CrossEntropyLoss(y, s) = -log(s⁺)

 +-----------------------+
 | CrossEntropyLoss |
 +-----------------------+
 |
 |
 +-------------------------------+
 | Softmax Activation Function |
 +-------------------------------+
 |
 +------------+------------+-------------+ ... +------------+
 | | | | | |
 s⁺ s⁻_1 s⁻_2 ... s⁻_n
 (Positive) (Negative) (Negative) (Negative)

 | | | |
 | | | |
 cos(a, b⁺) cos(a, b⁻_1) cos(a, b⁻_2) ... cos(a, b⁻_n)

 | | | |
(Positive) (Negative) (Negative) (Negative)
 Sample Sample Sample Sample

Two Tower Model(DSSM) - Positive Samples + Negative Samples

Positive Samples

impressed and clicked
problem: 20% item takes 80% click
solution:
- up-sampling: one sample appear multiple times
- down-sampling: discard samples

Negative Samples

Simple Negative Sampling: All Items

Uniform Sampling: Unfair to unpopular items

Most positive samples are popular items.
If uniform sampling produces negative samples, most of the negative samples are unpopular items.
positive sample: popular items, negative sample: unpopular items, this is unfair.

Non-uniform Sampling: The goal is to suppress popular items

The probability of negative sampling is positively correlated with popularity (clicks).
Sampling probability ∝ (clicks) ^ 0.75

Simple Negative Sampling: Negative Sample within a batch

user1 - item1
user2 - item2
user3 - item3

...

user_n - item_n

one batch has n positive samples
one batch has n(n - 1) negative samples
The probablity that an item appear in a batch is propotional to the number of times it clicked.
We aim to get it propotional to clicks^0.75.
offline training cos(a, bi) - log(pi), pi ∝ clicks
online inference cos(a, bi)

Hard Negative Sampling

Hard negative samples:

Items that are ranked and close to the target (relatively hard).
Items that are ranked far behind the target (extremely hard).

Binary classification for positive and negative samples:

All items (simple): High classification accuracy.
Items that are ranked and close to the target (relatively hard): Easy to misclassify.
Items that are ranked far behind the target (extremely hard): Even easier to misclassify.

Datasets

Mix negative samples
50% easy negative sample
50% hard negative sample

Recommender System Study Notes - Nomination(2/6)

Mon, 02 Sep 2024 00:00:00 +0000

Discrete Feature
Matrix Completion

Discrete Feature

Example

sex
nationality
english words
item id
user id

How to encode discrete feature -> vectorization

one-hot encoding
embedding (*)
- size: vector dimension * number of items
- How to implement: TensorFlow, PyTorch -> embedding
- embedding = Parameter matrix * one-hot encoding

Matrix Completion

Intuition

item id -> embedding
 \
 - cos(a, b)
 /
item id -> embedding

Formula

dataset = {u, i, y}
training, u -> au, i -> bi
optimization: $\min_{A, B} \sum_{\substack{(u, i, y) \in \Omega}} \left( y - \langle a_u, b_i \rangle \right)^2$

Pros/Cons

Cons
- It only uses ID embedding without item, user characteristics
  - item characteristics: label, key words
  - user characteristics: gender, age, geo, liked label
  - Two tower Model
- Negative sample is wrong
  - Positive: User interacted, which is good
  - Negative: User did not interact, which is not correct
- Training
  - dot product is worse than cosine similarity
  - Loss(regression) is worse than cross entropy

Implementation

Offline:

Learning A: user
Learning B: item
storing A into {user id: embedding} key-value

Online:

user Id -> find a’s vector
nearest neighbour search, return the top 100 b whose <a, b> is max
Approximate Nearest Neighbor (ANN)
- pre process the circile into sectors
- pick a representative vector(indexing) represents the sector
- find representative vector, then find vector in that sector instead of the whole circle

Recommender System Study Notes - Nomination(1/6)

Sun, 01 Sep 2024 00:00:00 +0000

Item CF
Swing
User CF

Item CF

Intuition

If user likes i1 and i1 is similar with i2 -> Then user is likely to like i2

Formula

$score = \sum_j \text{like}(\text{user}, \text{item}\_j) \times \text{sim}(\text{item}\_j, \text{item})$, to compute Item CF score
$\text{sim}(i_1, i_2) = \frac{|\mathbf{v}|}{\sqrt{|\mathbf{w}_1| \cdot |\mathbf{w}_2|}}$, to compute similarity, w1 as the user who likes i1, w2 as the set, v as the intersection between i1 and i2.
$\text{sim}(i*1, i_2) = \frac{\sum*{v \in v} \text{like}(v, i*1) \cdot \text{like}(v, i_2)}{\sqrt{\sum*{u*1 \in w_1} \text{like}^2(u_1, i_1)} \cdot \sqrt{\sum*{u_2 \in w_2} \text{like}^2(u_2, i_2)}}$, to take like score into consideration).

Implementation

Offline:

User -> Item Indexing
- return the last-n items with interaction that this user interacts with.
Item -> Item Indexing
- compute sim (i, j) maybe maintaining User->Item
- for each item, find the cloest k items

Online Inference: (user id -> 100 items)

user id -> last-n items
for each item in last-n, use item-item to find k items
Use score formula to compute n * k items’ score
pick top 100, and output top 100 items as the nomination result in this Item CF Channel

Swing

Intuition

If two items are not that relavant, but it got forwarded into a same small group, we want to lower the similarity score.

Formula

u1 -> j1, u2 -> j2
- $\text{overlap}(u_1, u_2) = |\mathcal{J}_1 \cap \mathcal{J}_2|$
i1 -> w1, i2 -> w2, v = w1 ∩ w2
- $\text{sim}(i_1, i_2) = \sum_{u_1 \in V} \sum_{u_2 \in V} \frac{1}{\alpha + \text{overlap}(u_1, u_2)}$, to compute similarity, a is a hyper-parameter
if overlap is big, then the contribution to similarity becomes low

User CF

Intuition

I will like the notes which liked by someone who is similar to me.

Formula

Method1: like, collect, click notes are similar
Method2: Subscribed authors are similar
u1 -> j1, u2 -> j2, I = j1 ∩ j2, nl: the number of users who likes l
- $sim = \text{sim}(u_1, u_2) = \frac{|I|}{\sqrt{|\mathcal{J}_1| \cdot |\mathcal{J}_2|}}$
- $sim = \text{sim}(u_1, u_2) = \frac{\sum_{I \in I} \frac{1}{\log(1 + n_I)}}{\sqrt{|\mathcal{J}_1| \cdot |\mathcal{J}_2|}}$
$score = \sum_j \text{sim(user, user}_j\text{)} \times \text{like(user}_j, \text{item)}$

Implementation

Offline:

User -> Item Indexing
- return the last-n items with interaction that this user interacts with.
User -> User Indexing
- compute sim (i, j)
- for each user, find the cloest k users

Online:

user-user indexing: id -> k cloest users
user-item indexing: for each user, find his/her last-n, use user-item to find n items
compute score: Use the formula to compute n * k items’ score
rank: pick top 100, and output top 100 items as the nomination result in this User CF Channel

Recommender System Study Notes Basics (1/1)

Sun, 01 Sep 2024 00:00:00 +0000

Basics

- Impression
- Click
- ScrollToEnd
- Comment
- Like
- Collect
- Share

基础指标

Click-through rate = Number of clicks / Number of impressions
Like rate = Number of likes / Number of impressions
Collection rate = Number of collections / Number of clicks
Forwarding rate = Number of forwards / Number of clicks
Completion rate = Number of times scrolled to the end / Number of clicks * f(Note length)

北极星指标

用户规模 DAU daily active user MAU monthly active user
消费人均使用时长，人均阅读笔记数量
发布发布渗透率，人均发布量

实验流程

- Offline Test
- A/B Test
- experiment rollout

Chain of Recommender System

- Nomination
- Pre-Ranking
- Ranking
- Re-Ranking

Nomination O(billion) -> O(k)
- CF
- Two Tower
- Subscribed Content Creator
Pre-Ranking + Ranking O(k) -> O(100)
- Use a neural network as follow:
- Input:
  - User Feature Embedding
  - Item Feature Embedding
  - Stats Embedding
- Output:
  - Click-through rate
  - Like Rate
  - Collection Rate
  - Forwarding Rate
  - Completion Rate
- Combine them and output a score
Re-Ranking O(100) -> O(80?) | for diversity
- Common methods:
- MMR
- DPP
- use pre-set rules to diversify notes
- insert ads, change sequence based on rules

A/B Test

Basic flow:
- new GNN, Offline testing gives promising results
- online A/B test, vaildate
- tune hyper parameters
how to put user into buckets randomly
- hash user id
- compute metrics for each bucket, DAU, etc
- experiment rollout
Overlapping experiment
- overlapping: nomination, pre-ranking, ranking, re-ranking, user interface, ads
- each layer: mutually exclusive
- between layers: orthogonality
Holdout mechanism
- 10% as holdout bucket + 90% as overlapping experiment
- clean holdout and rollout experiment
Experiment Rollout
Experiment Reverse