Edit on GitHub

MC Layers

The goal of MC Layers is to facilitate converting any layer defined in Flux into its Bayesian counterpart. MC Dropout is a principled tecnique to estimate predictive uncertainty in neural networs. The paper Dropout as a Bayesian Approximation proves that a neural network with dropout applied before every layer is equivalent to a Deep Gaussain Process. We can apply dropout at test time to simulate functions draws in a Gaussian Process for approximate Variational Inference.

In simple terms, a network with dropout applied before every layer has a different weight configuration in every forward pass (Note that we also apply dropout at test time, usually dropout is turned off during test time to ensemble the various weight samples implicitly). Each weight configuration can be considered an independent sample from the underlying weight distribution. The mean and variance of these sample predictions are the mean and variance of the predictive distribution. Higher the variance, more the uncertianty.

DeepUncertainty.MCLayer — Type
MCLayer(layer, dropout)

A generic Monte Carlo dropout layer. Takes in any "traditional" flux layer and a function that implements dropout. Performs the usual layer forward pass and then passes the acitvations through the given dropout function. Let x be the input array, the forward pass then becomes – dropout(layer(x))

Fields

  • layer: Traditional Flux layer
  • dropout: A function that implements dropout
source

We also implement MC versions of Conv and Dense layers as a drop-in replacement.

DeepUncertainty.MCDense — Function
MCDense(in::Int, out::Int, dropout_rate, σ=identity; bias=true, init=glorot_uniform)
MCDense(layer, dropout_rate)

Creates a traditional dense layer with MC dropout functionality. MC Dropout simply means that dropout is activated in both train and test times Reference - Dropout as a bayesian approximation - https://arxiv.org/abs/1506.02142

The traditional dense layer is a field in the struct MCDense, so all the arguments required for the dense layer can be provided, or the layer can be provided too. The forward pass is the affine transformation of the dense layer followed by dropout applied on the resulting activations. 

y = dropout(σ.(W * x .+ bias), dropout_rate)

Fields

  • layer: A traditional dense layer
  • dropout: A function that implements dropout

Arguments

  • in::Integer: Input dimension of features
  • out::Integer: Output dimension of features
  • dropout_rate::AbstractFloat: Dropout rate
  • σ::F=identity: Activation function, defaults to identity
  • init=glorot_normal: Initialization function, defaults to glorot_normal
source
DeepUncertainty.MCConv — Function
MCConv(filter, in => out, σ = identity;
        stride = 1, pad = 0, dilation = 1, groups = 1, [bias, weight, init])
MCConv(layer, dropout_rate)

Creates a traditional Conv layer with MC dropout functionality. MC Dropout simply means that dropout is activated in both train and test times

Reference - Dropout as a bayesian approximation - https://arxiv.org/abs/1506.02142

The traditional conv layer is a field in the struct MCConv, so all the arguments required for the conv layer can be provided, or the layer can be provided too. The forward pass is the conv operation of the conv layer followed by dropout applied on the resulting activations. 

y = dropout(Conv(x), dropout_rate)

Fields

  • layer: A traditional conv layer
  • dropout_rate::AbstractFloat: Dropout rate

Arguments

  • filter::NTuple{N,Integer}: Kernel dimensions, eg, (5, 5)
  • ch::Pair{<:Integer,<:Integer}: Input channels => output channels
  • dropout_rate::AbstractFloat: Dropout rate
  • σ::F=identity: Activation function, defaults to identity
  • init=glorot_normal: Initialization function, defaults to glorot_normal
source