RNN GRU LSTM Time Series

Overview

                Vanilla RNN, GRU, and LSTM compared on the International Airline Passengers dataset
                (144 monthly counts, 1949–1960). A baseline (Test 1) vs. an improved training regime
                (Test 2) shows that training methodology — not just architecture — determines
                whether a theoretically stronger model delivers better results.
                Best result: LSTM RMSE = 33.17 (Test 2, lr=0.005).
            

33.17

Best RMSE (LSTM, Test 2)

52%

LSTM Improvement over Baseline

64

Optimal Hidden Units

lr = 0.005

Optimal Learning Rate

Method

Figure 1: Shared recurrent network architecture — 64 hidden units, dropout, FC output

All three models share the same structure: a recurrent layer with 64 hidden units, a dropout layer (rate=0.0), and a fully connected output layer producing a single scalar. The vanilla RNN uses tanh activation; GRU and LSTM use gating mechanisms.

Dataset: 144 monthly passenger counts normalized to [0,1] with min-max scaling. Sliding window of length 12 (matching the seasonal period) creates input-output pairs. 80/20 temporal split: 105 training sequences, 27 validation sequences.

Training Configurations

Hyperparameter	Test 1 (Baseline)	Test 2 (Improved)
Epochs	150	300
Learning Rate	0.001	0.005
Batch Size	16	16
Optimizer	Adam	Adam
LR Scheduler	None	Plateau-based adaptive
Best-Model Checkpointing	No	Yes
Gradient Clipping	Max norm 1.0	Max norm 1.0

Results

Loss Curves — Test 1 (Baseline)

Figure 2A: Training and Validation Loss Curves — RNN, GRU, LSTM (Test 1, seq_len=12, hidden=64, lr=0.001)

Loss Curves — Test 2 (Improved)

Figure 2B: Training and Validation Loss Curves — RNN, GRU, LSTM (Test 2, seq_len=12, hidden=64, lr=0.001)

Predictions vs. Actual

Figure 3: Predictions vs. Actual Passengers — Validation Set (Test 2, lr=0.001)

Hyperparameter Analysis — Learning Rate

Learning Rate	Test 1 RMSE	Test 1 MAE	Test 2 RMSE	Test 2 MAE
0.0001	76.71	67.14	74.61	59.87
0.001	69.64	57.61	49.50	36.46
0.005 ★	46.83	39.08	33.17	27.67
0.01	61.25	51.40	37.15	28.83

lr=0.005 + adaptive scheduler achieves the best RMSE of 33.17. The scheduler amplifies the benefit of a correct learning rate while providing robustness against suboptimal choices.

Hyperparameter Analysis — Hidden Size

Hidden Units	Parameters	Test 1 RMSE	Test 1 MAE	Test 2 RMSE	Test 2 MAE
16	1,233	66.11	48.58	46.92	37.72
32	4,513	61.53	44.81	37.63	31.71
64 ★	17,217	69.64	57.61	33.17	27.67
128	67,201	44.72	38.63	37.62	32.11

In Test 2, RMSE improves steadily from 16 to 64 hidden units, then worsens at 128 due to overfitting on just 105 training sequences. 64 units provides the right capacity for this small dataset.

Conclusion

In Test 1 (baseline), the LSTM performed worst (RMSE=69.64) despite its architectural advantage — a fixed learning rate fails to navigate its complex loss landscape.
In Test 2 (adaptive scheduler + extended training + checkpointing), the LSTM recovered to RMSE=33.17 — the best result overall and a 52% improvement over its own baseline.
Architecture alone does not determine performance; a more powerful model requires a training regime capable of exploiting its capacity.
Optimal configuration: LSTM, lr=0.005, 64 hidden units, 300 epochs, plateau-based LR scheduler.

Comparative Analysis of RNN, GRU, and LSTM for Airline Passenger Time Series Forecasting