Among pre-existing RNN units, RUM is most similar to GORU in spirit because both models transform GRU. Note that GORU parametrizes an orthogonal operation while RUM extracts an orthogonal operation in the form of a rotation.

H !0h h#$% !#$% , ! x# ,# -̃# + ↑2 6 u# h# h + # !# !# + * 1− h5# Figure 2: Model: RUM's operation R, which projects and rotates h, the information pipeline in RUM. We let the RNN generate the special vectors at time step t by linearly embedding the RNN in˜t RNh , put xt RNx to an embedded input and by obtaining a target memory t as a linear combination of the current input xt and the previous history ht−1.

RUM consists of an update gate u RNh that has the same function as in GRU. However, instead of a reset gate, the model learns the memory target RNh. RUM also learns to embed the input vector x RNx RNh. Hence, Rotation eninto RNh to yield codes the rotation between the embedded input and the target, which is accumulated in the associative memory unit Rt RNh ×Nh. Here, is a nonnegative integer that is a hyper-parameter of the model.

Kernel for target RUM ! = 1 % = N/A target 4 and gate 5 input 6 and hidden state −5.0 −1 ( 40 70 3.0 0 RUM ! = 0 % = N/A 0.2 !"# ) !## !"# di !## a portion of the diagonal, visualized in a horizontal position, has the function to generate a target memory ) ( 40 80 target memory rota hidden s which is effecti RUM ! = 1 % = N/A 10k 0.0 !## 10 training steps Associative recall task is another testbed for long-term memory.

An interpr RUM, = 1, with Nh = 50 and without time the diagonal 0.5 activations 1.0of RUM's Whh kernel on NLP tas 0.4 0.0 and off-diagonal Level Penn Treebank and the model is = 0 RUM, Nh = 2000, = additional examples.

Namely, encRUM is a LEAD with a bi-directional RUM as an encoder, decRUM is LEAD with a bi-directional RUM as a decoder, and allRUM is LEAD with all LSTM units replaced by RUM ones.

5 Discussion RUM vs. GORU. Here, we study the energy landscape of the loss function in order to give some intuition about why RUM's choice of rotation is more appealing than what was used in previous phase-coded models.