Loss Functions

Machine learning from scratch · part 2Loss functions

A loss function turns one prediction and the truth into a single number. Training makes the average of that number as small as it can, so the choice of loss decides what the model learns.

Each card plots one loss against the thing the model controls: a probability, a margin, an error or a distance. Drag across a plot, or use its slider, to move the dot along the curve, and read the loss and its gradient off the readout. Sliders under a card change the loss's own settings. Dashed and dotted curves are other losses, drawn for comparison.

01

What a loss function is

A loss scores one prediction against the truth. Training adjusts the model to make the average loss as small as possible.

training minimises (1/n) · Σi L(yi, ŷi)

A loss does two jobs. Its value says how wrong a prediction is, and its slope, the gradient, says which way to change the model to make it less wrong. A loss with no useful slope cannot be trained on, however well it describes the goal.

That is why a loss and a metric are not always the same. A metric, as in the evaluation series, is what you report. Accuracy is the natural metric for classification, but it is flat almost everywhere, so training uses a smooth stand-in such as cross-entropy. For regression the two often coincide: mean squared error (MSE) is both.

One result runs through the classification losses. With a sigmoid or a softmax in front, the gradient of cross-entropy with respect to the logit z is simply p − y: the predicted probability minus the truth. Part 3, on logistic regression, derives it.

02

Binary classification

Two classes, labelled y = 0 and y = 1, or y = −1 and y = +1 for the margin losses. The model outputs a score z; cross-entropy turns it into a probability p = σ(z) first, while the hinge family works on z directly through the margin y·z, which is positive when the prediction is on the right side.

Binary cross-entropy (BCE, log loss)

−log of the probability given to the truth

    L = −[y log p + (1 − y) log(1 − p)]
    y = 1 leaves −log p; y = 0 leaves −log(1 − p)
    ∂L/∂z = p − y

    Takes
    a label y ∈ {0, 1} and a probability p = σ(z)
    Gradient
    p − y with respect to the logit z: small when right, largest when confidently wrong
    Zero
    only when p equals y exactly
    Used for
    any binary classifier, and logistic regression (part 3)
    Watch out
    −log p is infinite at p = 0, so one confident wrong answer can dominate. Pass the logit to BCEWithLogitsLoss rather than a sigmoid output to BCELoss: the same loss, computed stably
    Library
    PyTorch BCEWithLogitsLoss, BCELoss; scikit-learn log_loss

    Weighted binary cross-entropy

    w₊ × the positive term

      L = −[w₊ y log p + (1 − y) log(1 − p)]

      Takes
      the same as cross-entropy, plus a weight w₊ for the positive class
      Effect
      a missed positive costs w₊ times as much
      Typical w₊
      the number of negatives divided by the number of positives
      Used for
      imbalanced data, where positives are rare: fraud, a rare disease
      Watch out
      it pushes predicted probabilities up, so they stop matching real frequencies. Move the threshold or recalibrate if you need true probabilities
      Library
      PyTorch BCEWithLogitsLoss(pos_weight=…)

      Focal loss

      −(1 − pₜ)γ log pₜ

        L = −α (1 − pₜ)γ log pₜ
        pₜ = p if y = 1, else 1 − p

        Takes
        pₜ, the probability given to the true class, a focus γ ≥ 0 and a class weight α
        Effect
        multiplies cross-entropy by (1 − pₜ)γ, which is tiny for easy examples
        γ = 0
        plain cross-entropy (with α = 1)
        Used for
        heavy imbalance, where thousands of easy negatives would swamp the loss: one-stage object detectors. RetinaNet uses γ = 2 and α = 0.25
        Watch out
        two settings to tune, and they interact
        Library
        torchvision sigmoid_focal_loss

        Hinge

        max(0, 1 − y·z)

          L = max(0, 1 − y·z), with y ∈ {−1, +1}
          ∂L/∂z = −y while y·z < 1, then 0

          Takes
          a label y ∈ {−1, +1} and a raw score z, not a probability
          Margin
          y·z is positive when the sign of z is right; the loss is zero once it reaches 1
          Gradient
          a constant push until the margin is reached, then none
          Used for
          support vector machines (SVMs). The flat zero region is why only the points near the boundary, the support vectors, shape the model
          Watch out
          no probabilities come out, and it has a corner at y·z = 1
          Library
          scikit-learn SGDClassifier(loss="hinge"), hinge_loss

          Squared hinge

          max(0, 1 − y·z)²

            L = max(0, 1 − y·z)²
            ∂L/∂z = −2y · max(0, 1 − y·z)

            Takes
            the same as hinge
            Shape
            smooth at the margin, quadratic for wrong answers
            Zero
            once y·z ≥ 1
            Used for
            linear SVMs: it is the default loss of scikit-learn's LinearSVC
            Watch out
            squaring lets badly wrong points, often mislabelled ones, dominate
            Library
            scikit-learn LinearSVC(loss="squared_hinge")

            Exponential

            e−y·z

              L = e−y·z
              ∂L/∂z = −y · e−y·z

              Takes
              a label y ∈ {−1, +1} and a raw score z
              Never zero
              it only approaches 0 as the margin grows
              Grows
              exponentially for wrong answers: about 20 at y·z = −3
              Used for
              AdaBoost (adaptive boosting), which reweights the training examples by exactly this loss each round
              Watch out
              a single mislabelled point can dominate training
              Library
              scikit-learn AdaBoostClassifier, GradientBoostingClassifier(loss="exponential")

              Dice loss

              1 − the overlap

                L = 1 − 2 Σ piyi / (Σ pi + Σ yi)

                Takes
                a predicted probability for every pixel, and the true mask
                Measures
                the overlap between prediction and mask, like an F1 score
                Here
                p on the object's pixels and q on the background, with the object a share f of the image
                Used for
                segmentation where the object is a small part of the image, such as a lesion in a scan. Drag f down: Dice keeps its full range while the mean cross-entropy, dominated by background pixels, barely changes
                Watch out
                unstable when the mask is empty, so it is often added to cross-entropy rather than used alone
                Library
                MONAI DiceLoss, segmentation_models_pytorch

                The 0–1 loss, the fraction misclassified and drawn dotted on the hinge card, is what classification really cares about. Its gradient is zero everywhere it exists, so every loss above is a smooth stand-in for it. Log loss written with ±1 labels, log(1 + e−y·z), is binary cross-entropy under another name: PyTorch calls it SoftMarginLoss.

                03

                Multiclass classification

                One of K classes is true. The model outputs a logit zk for each, and the softmax turns them into probabilities pk that sum to 1. The label is a one-hot vector y: 1 for the true class and 0 for every other.

                Cross-entropy adds up −yk log pk over all K classes.

                L = −Σk=1…K yk log pk

                Because y is one-hot, every term but one is multiplied by 0. With three classes, the second one true, so y = (0, 1, 0), and the model predicting p = (0.2, 0.7, 0.1):

                L = −(0 · log 0.2 + 1 · log 0.7 + 0 · log 0.1) = −log 0.7 = 0.357

                So for a one-hot label the sum is just L = −log py, minus the log of the probability given to the true class, and that is what the plots below show. With K = 2 the sum has two terms, y log p and (1 − y) log(1 − p), which is binary cross-entropy from section 02. Label smoothing is the exception: its y is not one-hot, so every term counts.

                Cross-entropy (CE)

                −Σ yk log pk

                  L = −Σk yk log pk, over all K classes
                  y is one-hot, so only the true class is left:
                  L = −log py
                  ∂L/∂zk = pk − yk

                  Takes
                  a logit for each of K classes, and the true class as a one-hot vector y
                  Gradient
                  softmax − one-hot: p − y again, class by class
                  A uniform guess
                  1/K for every class scores log K
                  Used for
                  every multiclass classifier
                  Watch out
                  PyTorch's CrossEntropyLoss applies the softmax itself: pass logits, not probabilities, or it is applied twice
                  Library
                  PyTorch CrossEntropyLoss (logits), NLLLoss (log-probabilities); Keras categorical_crossentropy (one-hot labels) and sparse_categorical_crossentropy (integer labels); scikit-learn log_loss

                  Label-smoothed cross-entropy

                  aim for 1 − ε, not 1

                    target = 1 − ε + ε/K on the true class, ε/K on each other
                    L = −Σk targetk log pk

                    Takes
                    logits, the true class, and a smoothing ε
                    Lowest at
                    py = 1 − ε + ε/K: the model is never pushed to certainty
                    Here
                    the other K − 1 classes share whatever py leaves
                    Used for
                    image classifiers and transformers, where ε = 0.1 is common. Predictions come out less over-confident and better calibrated
                    Watch out
                    it caps the confidence the model will reach, which hurts if you need probabilities near 1, and it can weaken knowledge distillation
                    Library
                    PyTorch CrossEntropyLoss(label_smoothing=0.1)

                    Weighted cross-entropy

                    wy × the cross-entropy

                      L = −wy log py

                      Takes
                      logits, the true class, and one weight per class
                      Effect
                      mistakes on a class with weight w cost w times as much
                      Typical weights
                      inversely proportional to how common each class is
                      Used for
                      imbalanced multiclass data, such as rare diagnoses among common ones
                      Watch out
                      PyTorch's mean divides by the sum of the weights in the batch, not by the number of examples, so the loss scale shifts from batch to batch
                      Library
                      PyTorch CrossEntropyLoss(weight=…)

                      Focal loss, multiclass

                      −(1 − py)γ log py

                        L = −αy (1 − py)γ log py, p from the softmax

                        Takes
                        logits, the true class, γ, and optionally a weight α per class
                        Same as
                        the binary focal loss, with pₜ the softmax probability of the true class
                        γ = 0
                        plain cross-entropy
                        Used for
                        multiclass detection and segmentation with many easy examples
                        Watch out
                        there is no PyTorch builtin; it is a few lines on top of log_softmax
                        Library
                        written by hand from F.log_softmax

                        Multiclass hinge

                        (1/K) Σ max(0, 1 − zy + zj)

                          L = (1/K) Σj≠y max(0, 1 − zy + zj)

                          Takes
                          a raw score for every class, not probabilities
                          Zero
                          once the true class's score beats every other by at least 1
                          Here
                          three classes; the other two scores are 0 and s
                          Used for
                          multiclass SVMs
                          Watch out
                          Crammer and Singer's version, in LinearSVC(multi_class="crammer_singer"), takes the largest term instead of the sum. Cross-entropy never reaches 0, so it keeps pushing the scores apart
                          Library
                          PyTorch MultiMarginLoss

                          Kullback–Leibler (KL) divergence

                          Σ q log(q / p)

                            KL(q ‖ p) = Σk qk log(qk / pk)
                            cross-entropy = KL(q ‖ p) + H(q)

                            Takes
                            a predicted distribution p and a target distribution q
                            Zero
                            only when p = q
                            vs cross-entropy
                            they differ by H(q), the target's entropy, which does not depend on the model, so the gradients are identical
                            Used for
                            knowledge distillation, matching a teacher model's soft outputs, and the regulariser in variational autoencoders
                            Watch out
                            PyTorch's KLDivLoss wants log-probabilities as input and probabilities as target, and reduction="batchmean" for the true KL
                            Library
                            PyTorch KLDivLoss

                            Multi-label is different. When several classes can be true at once, such as the tags on a photo, there are K separate yes-or-no questions. Use binary cross-entropy on each class, BCEWithLogitsLoss, not softmax with cross-entropy, which forces the classes to compete.

                            04

                            Regression

                            The model predicts a number ŷ, and most regression losses are a function of the error e = y − ŷ. A positive e means the prediction was too low.

                            Mean squared error (MSE, L2)

                            e²

                              L = mean (y − ŷ)²
                              ∂L/∂ŷ = 2(ŷ − y)

                              Predicts
                              the mean of y
                              Gradient
                              proportional to the error, so big misses dominate
                              Units
                              squared units of y
                              Used for
                              the default for regression, and the least squares behind linear regression
                              Watch out
                              outliers dominate: one error of 10 costs as much as a hundred errors of 1
                              Library
                              PyTorch MSELoss; scikit-learn mean_squared_error

                              Mean absolute error (MAE, L1)

                              |e|

                                L = mean |y − ŷ|
                                ∂L/∂ŷ = sign(ŷ − y)

                                Predicts
                                the median of y
                                Gradient
                                always ±1, whatever the size of the error
                                Units
                                the units of y
                                Used for
                                data with outliers, or when every unit of error costs the same
                                Watch out
                                the gradient stays full size right up to the answer, so training can bounce around the minimum; lower the learning rate as it goes
                                Library
                                PyTorch L1Loss; scikit-learn mean_absolute_error

                                Huber

                                e² near 0, |e| beyond δ

                                  L = ½e² if |e| ≤ δ, else δ(|e| − ½δ)
                                  ∂L/∂ŷ = −e inside δ, −δ · sign(e) outside

                                  Predicts
                                  something between the mean and the median, set by δ
                                  Gradient
                                  like MSE near 0, capped at δ for large errors
                                  δ
                                  the error size where it switches from squared to linear
                                  Used for
                                  regression with some outliers, and bounding-box regression in object detection as Smooth L1
                                  Watch out
                                  δ is in the units of y, so it has to be chosen for each target
                                  Library
                                  PyTorch HuberLoss(delta); SmoothL1Loss(beta) is Huber divided by β; scikit-learn HuberRegressor

                                  Log-cosh

                                  log cosh e

                                    L = log cosh(y − ŷ)
                                    ∂L/∂ŷ = −tanh(y − ŷ)

                                    Near 0
                                    about ½e², like MSE
                                    Far out
                                    about |e| − log 2, like MAE
                                    Gradient
                                    tanh of the error: smooth, and never above 1
                                    Used for
                                    a Huber-like loss with no δ to choose. It has a second derivative everywhere, which gradient boosting libraries use for custom objectives
                                    Watch out
                                    cosh overflows for errors beyond about 710; compute it as |e| + log(1 + e−2|e|) − log 2
                                    Library
                                    no builtin in PyTorch or scikit-learn: one line by hand

                                    Quantile (pinball)

                                    max(τe, (τ − 1)e)

                                      L = max(τ·e, (τ − 1)·e), with e = y − ŷ
                                      ∂L/∂ŷ = −τ if e > 0, 1 − τ if e < 0

                                      Predicts
                                      the τ-th quantile of y: at τ = 0.9, a value that 90% of outcomes fall below
                                      Shape
                                      a tilted V: under-predicting costs τ per unit, over-predicting 1 − τ
                                      τ = 0.5
                                      half the MAE, so it predicts the median
                                      Used for
                                      prediction intervals, from one model at τ = 0.05 and one at τ = 0.95, and demand forecasting
                                      Watch out
                                      models trained separately for different τ can cross, giving a lower bound above the upper one
                                      Library
                                      scikit-learn mean_pinball_loss, GradientBoostingRegressor(loss="quantile", alpha=τ), QuantileRegressor

                                      Mean squared log error (MSLE)

                                      (log(1 + ŷ) − log(1 + y))²

                                        L = (log(1 + ŷ) − log(1 + y))²

                                        Takes
                                        targets and predictions above −1, usually positive
                                        Measures
                                        relative error: predicting 20 for 10 costs about as much as 200 for 100
                                        Here
                                        the true value is y = 10
                                        Used for
                                        targets that span orders of magnitude, such as prices or counts
                                        Watch out
                                        an under-prediction costs more than an over-prediction of the same size, and it predicts the geometric mean of 1 + y, minus 1, so predictions sit below the average
                                        Library
                                        scikit-learn mean_squared_log_error, root_mean_squared_log_error

                                        Poisson

                                        ŷ − y log ŷ

                                          NLL = ŷ − y log ŷ, plus a term without ŷ
                                          deviance = 2[y log(y/ŷ) − (y − ŷ)], plotted

                                          Takes
                                          a count y ≥ 0 and a predicted rate ŷ > 0
                                          Lowest at
                                          ŷ = y, where the deviance is 0
                                          Shape
                                          steep towards 0, gentle above: predicting too low a rate is costly
                                          Used for
                                          counts: hospital visits, insurance claims, clicks
                                          Watch out
                                          ŷ must stay positive, so models usually predict log ŷ and exponentiate (log_input=True in PyTorch)
                                          Library
                                          PyTorch PoissonNLLLoss; scikit-learn mean_poisson_deviance, PoissonRegressor, HistGradientBoostingRegressor(loss="poisson")

                                          Gaussian negative log-likelihood (NLL)

                                          ½(log σ² + e²/σ²)

                                            L = ½(log σ² + (y − μ)² / σ²)
                                            lowest at σ = |y − μ|

                                            Takes
                                            a predicted mean μ and a predicted variance σ² > 0
                                            Lowest at
                                            σ = |y − μ|: the model learns to predict how wrong it will be
                                            Effect
                                            a large error costs less when σ is large, so the model can admit uncertainty
                                            Used for
                                            regression that reports its own uncertainty, such as a measurement with error bars. σ² usually comes from a softplus output (part 1)
                                            Watch out
                                            on points it fits exactly, the model can shrink σ towards 0 and drive the loss to −∞; PyTorch clamps the variance at a small eps
                                            Library
                                            PyTorch GaussianNLLLoss(input=μ, target=y, var=σ²)

                                            Also seen: Gamma and Tweedie deviance, for positive, skewed targets such as insurance claim amounts, and mean absolute percentage error (MAPE) as a loss, which is rarely a good idea for the reasons in part 13 of the evaluation series.

                                            05

                                            Contrastive losses

                                            These train an embedding: a model that maps each input to a vector. There is no class to predict. Training only needs to know which inputs belong together.

                                            A positive pair is two inputs that should end up close: two photos of the same person, two random crops of the same image, or a photo and its own caption.

                                            A negative pair is two inputs that should end up far apart: photos of two different people, crops of two different images, or a photo and another photo's caption.

                                            Pairs come from labels, such as who is in each photo, or are built from the data itself: crop one image twice and you have a positive pair without labelling anything, which is how self-supervised learning works. In a batch, each input's positive is its partner, and the rest of the batch usually serves as its negatives. Closeness is measured as a distance d between the two vectors (0 when they are identical) or as a cosine similarity s (1 when they point the same way, −1 when they point opposite ways).

                                            The pair loss takes one pair at a time and a label y: 1 for a positive pair, 0 for a negative pair. As with binary cross-entropy, one of its two terms is always 0, so a positive pair costs d² and a negative pair costs max(0, m − d)².

                                            L = y · d² + (1 − y) · max(0, m − d)²

                                            Contrastive loss (pairs)

                                            pull positive pairs in, push negative pairs past m

                                              L = y · d² + (1 − y) · max(0, m − d)²
                                              y = 1, a positive pair: L = d²
                                              y = 0, a negative pair: L = max(0, m − d)²

                                              Takes
                                              two embeddings, the distance d between them, and y: 1 for a positive pair, 0 for a negative pair
                                              Positive pairs
                                              cost d², so they are pulled together all the way to d = 0
                                              Negative pairs
                                              cost (m − d)² while closer than the margin m, and nothing once they are at least m apart
                                              Used for
                                              learning embeddings from pairs: face and signature verification, with Siamese networks (two copies of one network, sharing weights)
                                              Watch out
                                              conventions vary: some papers use y = 1 for a negative pair, put ½ in front, or use d instead of d². Most negative pairs are already far apart and contribute nothing, so training looks for hard negatives: negative pairs that are still close
                                              Library
                                              no core PyTorch class; pytorch-metric-learning ContrastiveLoss, or a few lines by hand

                                              InfoNCE takes one anchor at a time and compares it with N candidates: its positive and N − 1 negatives. With s+ the anchor's similarity to its positive, sj its similarity to negative j, and T a temperature:

                                              L = −log [ es+/T / ( es+/T + Σj=1…N−1 esj/T ) ]

                                              The numerator, es+/T, comes from the positive pair alone. It grows as the positive moves closer to the anchor.

                                              The denominator is that same term plus one term for each negative: a total over all N candidates. Any negative that sits close to the anchor makes it bigger.

                                              The fraction is a softmax over the N candidates: the probability the anchor gives to its own positive. It is near 1 when the positive is far more similar than every negative, and 1/N when the model cannot tell them apart. Its −log is cross-entropy with the positive as the true class, so the loss is 0 for a perfect match and log N for a blind guess. To lower it, training raises the numerator by pulling the positive in, and shrinks the negatives' share of the denominator by pushing them away. A small T exaggerates differences in similarity, so the closest negatives dominate the denominator.

                                              For example, take an anchor whose positive has similarity 0.8, three negatives at 0.1, 0.3 and −0.3, and T = 0.5:

                                              numerator = e0.8/0.5 = 4.953
                                              denominator = 4.953 + e0.1/0.5 + e0.3/0.5 + e−0.3/0.5
                                                = 4.953 + 1.221 + 1.822 + 0.549 = 8.545
                                              fraction = 4.953 / 8.545 = 0.580;  L = −log 0.580 = 0.545

                                              InfoNCE (info noise-contrastive estimation)

                                              cross-entropy over similarities

                                                L = −log ( numerator / denominator )
                                                numerator = es₊/T, the positive pair
                                                denominator = es₊/T + Σj esj/T, the positive and all N − 1 negatives

                                                Takes
                                                an anchor, its positive (s₊ is their cosine similarity) and N − 1 negatives (the rest of the batch)
                                                It is
                                                cross-entropy with N classes, where the right class is the positive. A blind guess scores log N, the dashed line
                                                Here
                                                every negative has similarity s₋ to the anchor
                                                Used for
                                                self-supervised and multimodal learning: SimCLR (a simple framework for contrastive learning) as NT-Xent, the normalised temperature-scaled cross-entropy; MoCo (momentum contrast); and CLIP (contrastive language–image pre-training), which matches each image to its own caption among the batch's
                                                Watch out
                                                the temperature matters a lot: small T sharpens the softmax and focuses on the hardest negatives. Bigger batches give more negatives, which is why these methods train with large ones
                                                Library
                                                F.cross_entropy(similarities / T, targets), as in the code below

                                                Triplet loss sits between the two: max(0, d(a, p) − d(a, n) + m) takes an anchor a, a positive p and a negative n, and only asks that the negative be at least m further away than the positive. It is PyTorch's TripletMarginLoss, and it was used to train FaceNet.

                                                06

                                                What each loss predicts

                                                For regression, the loss decides which summary of y the model learns to predict.

                                                MSE → the mean  ·  MAE → the median  ·  pinball at τ → the τ-th quantile
                                                MSLE → the geometric mean of 1 + y, minus 1

                                                Take a skewed sample of 4,001 values and find the single number that makes each loss smallest. For MSE it is 1.83, the sample's mean. For MAE it is 1.31, the median; for pinball at τ = 0.9 it is 3.68, the 90th percentile; and for MSLE it is 1.51. On symmetric data the first two agree; on skewed data, such as incomes or hospital stays, they can differ by a lot, and the loss you train with decides which one you get.

                                                07

                                                Which to use

                                                problemlosswhy
                                                Binary classificationbinary cross-entropyprobabilities, and a p − y gradient
                                                … with rare positivesweighted BCE or focalstop the common class swamping the loss
                                                Maximum-margin classifierhingeonly points near the boundary matter
                                                Segmenting small objectsDice, often plus BCEscores overlap, not pixel counts
                                                Multiclass classificationcross-entropythe standard; softmax probabilities
                                                … over-confident modelslabel smoothingcaps the target below 1
                                                … imbalanced classesweighted CE or focalrare classes count for more
                                                Matching soft targetsKL divergencecompares whole distributions
                                                Several labels at onceBCE per classindependent yes-or-no answers
                                                RegressionMSEpredicts the mean
                                                … with outliersMAE or Huberlarge errors do not dominate
                                                … values spanning magnitudesMSLErelative error
                                                … countsPoissonpredicts a rate
                                                … a prediction intervalquantilepredicts a chosen quantile
                                                … with its own uncertaintyGaussian NLLpredicts a mean and a variance
                                                Learning embeddings from pairscontrastivedistances, with a margin
                                                Learning embeddings from batchesInfoNCEone positive against many negatives
                                                08

                                                In code

                                                Each loss is a line or two of NumPy. Below the definitions they are run on small examples, and the numbers in the comments match PyTorch and scikit-learn on the same inputs.

                                                by_hand.py

                                                import numpy as np
                                                
                                                def sigmoid(z):     return 1 / (1 + np.exp(-z))
                                                def log_softmax(Z): Z = Z - Z.max(axis=1, keepdims=True); return Z - np.log(np.exp(Z).sum(axis=1, keepdims=True))
                                                
                                                # ---- binary: y in {0, 1}, p = sigmoid(z) ----
                                                def bce(y, p):              return -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
                                                def weighted_bce(y, p, w):  return -np.mean(w * y * np.log(p) + (1 - y) * np.log(1 - p))
                                                def focal(y, p, gamma=2):   pt = np.where(y == 1, p, 1 - p); return -np.mean((1 - pt) ** gamma * np.log(pt))
                                                def hinge(y, z):            return np.mean(np.maximum(0, 1 - (2 * y - 1) * z))    # labels mapped to -1, +1
                                                
                                                # ---- multiclass: Z = logits, one row per example; y = the true class index ----
                                                def cross_entropy(Z, y):            # y = index of the true class
                                                    Y = np.eye(Z.shape[1])[y]       # one-hot: 1 for the true class, 0 for the rest
                                                    return -np.mean(np.sum(Y * log_softmax(Z), axis=1))   # only the true class's term survives
                                                def smoothed_ce(Z, y, eps=0.1):
                                                    K = Z.shape[1]; T = np.full(Z.shape, eps / K); T[np.arange(len(y)), y] += 1 - eps
                                                    return -np.mean(np.sum(T * log_softmax(Z), axis=1))
                                                def kl(q, p):               return np.sum(q * np.log(q / p))
                                                
                                                # ---- regression: e = y - y_hat ----
                                                def mse(y, yh):             return np.mean((y - yh) ** 2)
                                                def mae(y, yh):             return np.mean(np.abs(y - yh))
                                                def huber(y, yh, d=1.0):    e = np.abs(y - yh); return np.mean(np.where(e <= d, 0.5 * e ** 2, d * (e - 0.5 * d)))
                                                def log_cosh(y, yh):        e = np.abs(y - yh); return np.mean(e + np.log1p(np.exp(-2 * e)) - np.log(2))   # no overflow
                                                def pinball(y, yh, tau):    e = y - yh; return np.mean(np.maximum(tau * e, (tau - 1) * e))
                                                def gaussian_nll(y, mu, var): return np.mean(0.5 * (np.log(var) + (y - mu) ** 2 / var))
                                                
                                                # ---- contrastive: embeddings, compared by distance or similarity ----
                                                def contrastive(d, y, m=1.0):      return np.mean(y * d ** 2 + (1 - y) * np.maximum(0, m - d) ** 2)   # y = 1: positive pair
                                                def info_nce(S, t=0.1):            return cross_entropy(S / t, np.arange(len(S)))   # row i: column i is the positive, the rest negatives
                                                
                                                y  = np.array([1, 0, 1, 0]);  p = np.array([0.9, 0.2, 0.6, 0.4]);  z = np.log(p / (1 - p))
                                                bce(y, p)                     # 0.338
                                                weighted_bce(y, p, w=4.0)     # 0.800
                                                focal(y, p, gamma=2)          # 0.043
                                                hinge(y, z)                   # 0.297
                                                
                                                Z = np.array([[2.0, 0.5, -1.0], [0.1, 1.2, 0.3]]);  c = np.array([0, 1])
                                                cross_entropy(Z, c)           # 0.397
                                                smoothed_ce(Z, c, eps=0.1)    # 0.506
                                                kl(np.array([0.7, 0.3]), np.array([0.6, 0.4]))   # 0.022
                                                
                                                y_true = np.array([3.0, 5.0, 2.5, 7.0]);  y_hat = np.array([2.5, 5.5, 2.0, 9.0])
                                                mse(y_true, y_hat)            # 1.188
                                                mae(y_true, y_hat)            # 0.875
                                                huber(y_true, y_hat, d=1.0)   # 0.469
                                                log_cosh(y_true, y_hat)       # 0.421
                                                pinball(y_true, y_hat, 0.9)   # 0.287
                                                gaussian_nll(y_true, y_hat, var=np.full(4, 2.0))   # 0.643
                                                
                                                contrastive(np.array([0.2, 0.9, 1.5, 0.4]), y=np.array([1, 1, 0, 0]))      # 0.302, two positive pairs, two negative
                                                S = np.array([[0.9, 0.1, -0.2], [0.0, 0.8, 0.3], [-0.1, 0.2, 0.7]])          # anchor i against candidate j; positives on the diagonal
                                                info_nce(S, t=0.1)            # 0.005

                                                with_pytorch.py

                                                import torch, torch.nn.functional as F
                                                from torchvision.ops import sigmoid_focal_loss
                                                from sklearn.metrics import hinge_loss, mean_pinball_loss
                                                
                                                z = torch.logit(torch.tensor([0.9, 0.2, 0.6, 0.4]));  y = torch.tensor([1., 0., 1., 0.])
                                                F.binary_cross_entropy_with_logits(z, y)                               # 0.338, takes logits
                                                F.binary_cross_entropy_with_logits(z, y, pos_weight=torch.tensor(4.))  # 0.800
                                                sigmoid_focal_loss(z, y, alpha=-1, gamma=2, reduction="mean")          # 0.043, alpha=-1: no class weight
                                                hinge_loss(2 * y.numpy() - 1, z.numpy())                               # 0.297, labels as -1, +1
                                                
                                                Z = torch.tensor([[2.0, 0.5, -1.0], [0.1, 1.2, 0.3]]);  c = torch.tensor([0, 1])
                                                F.cross_entropy(Z, c)                                  # 0.397, takes logits
                                                F.cross_entropy(Z, c, label_smoothing=0.1)             # 0.506
                                                F.kl_div(torch.tensor([0.6, 0.4]).log(), torch.tensor([0.7, 0.3]), reduction="sum")   # 0.022
                                                
                                                y_true = torch.tensor([3.0, 5.0, 2.5, 7.0]);  y_hat = torch.tensor([2.5, 5.5, 2.0, 9.0])
                                                F.mse_loss(y_hat, y_true)                              # 1.188
                                                F.l1_loss(y_hat, y_true)                               # 0.875
                                                F.huber_loss(y_hat, y_true, delta=1.0)                 # 0.469
                                                mean_pinball_loss(y_true, y_hat, alpha=0.9)            # 0.287
                                                F.gaussian_nll_loss(y_hat, y_true, var=torch.full((4,), 2.0))   # 0.643
                                                
                                                S = torch.tensor([[0.9, 0.1, -0.2], [0.0, 0.8, 0.3], [-0.1, 0.2, 0.7]])
                                                F.cross_entropy(S / 0.1, torch.arange(3))              # 0.005, InfoNCE is cross-entropy
                                                torch.nn.TripletMarginLoss(margin=1.0)                 # anchor, positive, negative embeddings

                                                Next, part 3: logistic regression, which puts the sigmoid from part 1 and binary cross-entropy together into a complete classifier, built from scratch.