OpenTS - A Comprehensive and Fair Benchmark for Time Series Analytics

Leaderboards

Leaderboard for univariate time series forecasting

Learning Paradigm  [all]

Statistics Learning

Machine Learning

Deep Learning

---

Metrics  [all]

MAE

MSE

MSMAPE

---

Datasets  [all|off]

Frequency

Hourly

Daily

Weekly

Monthly

Quarterly

Yearly

Other

Characteristics

Trend

Transition

Shifting

Seasonality

Stationarity

Irregular

---

Score

🥇

🥇 + 🥈 + 🥉

× 🥇 + × 🥈 + × 🥉

Rank	Model	Score	🥇	🥈	🥉	Paper	Publication	Year
1	Linear Regression	5911	2570	1902	1439	paper	Wiley [bib]	2005
2	TimesNet	5683	1945	1908	1830	paper	ICLR [bib]	2023
3	N-HiTS	4761	1997	1452	1312	paper	AAAI [bib]	2023
4	Random Forest	4403	1522	1503	1378	paper	ML [bib]	2021
5	N-BEATS	4168	1510	1411	1247	paper	ICLR [bib]	2020
6	Arima	3935	1403	1282	1250	paper	JASA [bib]	1970
7	Xgboost	3776	1320	1209	1247	paper	KDD [bib]	2016
8	PatchTST	3477	969	1148	1360	paper	ICLR [bib]	2023
9	Naive Drift	3383	1032	995	1356	paper	Darts [bib]	——
10	Naive Moving Average	3149	1000	1143	1006	paper	Darts [bib]	——
11	TiDE	3046	963	1031	1052	paper	arXiv [bib]	2023
12	Informer	3003	968	982	1053	paper	AAAI [bib]	2021
13	FEDformer	2414	715	771	928	paper	ICML [bib]	2022
14	FiLM	2300	753	764	783	paper	NeurIPS [bib]	2022
15	Naive Seasonal	2143	0	1000	1143	paper	Darts [bib]	——
16	Non-stationary Transformers	2091	607	733	751	paper	NeurIPS [bib]	2022
17	AutoCES	2056	679	669	708	paper	Darts [bib]	——
18	Kalman Filter	1907	809	569	529	paper	CUP [bib]	1990
19	AutoTheta	1846	454	682	710	paper	Darts [bib]	——
20	AutoETS	1791	467	567	757	paper	Darts [bib]	——
21	LightGBM	1564	553	560	451	paper	NIPS [bib]	2017
22	Naive Mean	1506	531	524	451	paper	Darts [bib]	——
23	DLinear	1155	232	383	540	paper	AAAI [bib]	2023
24	NLinear	1089	430	336	323	paper	AAAI [bib]	2023
25	Triformer	654	230	249	175	paper	IJCAI [bib]	2022
26	RNN	603	238	178	187	paper	arXiv [bib]	2019
27	Crossformer	396	127	129	140	paper	ICLR [bib]	2023
28	TCN	387	175	119	93	paper	arXiv [bib]	2018

Rules:

• For univariate forecasting algorithms, we consider 8,068 series, and the MAE、MSE、MSMAPE metrics, thus having 24,204 (8,068 * 3) unique evaluation settings, click here to see the detailed results, click here to download detailed evaluation results for each of the 8,068 univariate time series.
• Datasets are classified by sampling frequency into Yearly, Quarterly, Monthly, Weekly, Daily, Hourly, and Other. Their corresponding forecasting horizons are 6, 8, 18, 13, 14, 48, and 8, respectively. The look-back window is set to 1.25 times the respective forecasting horizon.
• For each forecasting algorithm, we count the number of times that the algorithm receives the gold, silver, and bronze medals, i.e., having the lowest, 2nd lowest, and 3rd lowest errors, shown as 🥇, 🥈, and 🥉, respectively.
• We provide three different types of scores for ranking the forecasting algorithms. First, the scores equal to the numbers of gold medals. Second, the scores are the sum of the numbers of gold, silver, and bronze medals. Third, the scores are the weighted sum of the gold, silver, and bronze medals, where the weights can be customized. The larger the score, the higher the ranking.

Univariate forecasting results

Results classified according to characteristics

Datasets		Metric	Crossformer	N-HiTS	TiDE	DLinear	NLinear	PatchTST	TimesNet	FEDformer	FiLM	Stationary	Triformer	RF	Informer	N-BEATS	RNN	TCN	LightGBM	Xgboost	LR	KF	Arima	ND	NM	NMA	NS	AutoCES	AutoETS	AutoTheta
Seasonality	√	mase	29.704	2.189	2.074	2.409	2.850	1.660	1.446	2.100	1.882	2.384	19.378	1.649	2.390	2.081	29.456	24.159	3.426	1.715	7.9e+9	9.002	2.830	4.111	8.855	4.164	4.164	2.845	4.091	3.901
		msmape	161.074	13.557	14.566	19.628	20.938	12.263	10.927	19.041	13.537	15.190	107.957	12.790	14.301	15.794	155.612	121.335	18.263	13.404	19.183	57.409	19.868	23.686	45.386	21.606	21.606	19.227	26.386	27.179
		mase_rank	17	201	175	22	31	129	329	51	141	33	12	235	60	147	19	17	59	212	588	144	203	72	25	47	0	63	30	51
		msmape_rank	17	196	156	19	32	147	303	51	134	29	11	259	61	154	18	15	57	226	597	144	206	76	28	44	0	58	33	46
	×	mase	23.704	1.678	2.526	1.949	1.733	1.639	1.478	1.879	1.769	1.638	16.496	1.731	1.601	1.677	23.257	15.441	3.267	1.814	2.6e+10	3.318	1.496	1.594	9.191	1.650	1.650	1.645	1.544	1.503
		msmape	166.859	24.127	30.968	26.966	27.154	21.671	20.497	26.766	22.331	22.050	138.945	25.316	21.413	25.705	160.553	140.174	35.614	26.838	33.413	44.551	24.491	25.961	45.340	22.753	22.753	21.805	25.190	25.697
		mase_rank	23	474	139	51	107	189	321	192	109	166	60	272	261	367	71	42	126	222	244	147	231	279	153	317	0	166	123	98
		msmape_rank	24	477	150	53	108	192	318	188	110	164	56	275	252	345	69	44	123	222	250	149	232	283	148	329	0	160	125	105
Trend	√	mase	41.287	2.553	3.316	3.016	2.914	2.220	1.911	2.758	2.492	2.651	28.091	2.271	2.658	2.512	40.788	28.716	4.874	2.355	7.9e+9	8.486	2.822	3.611	15.686	3.789	3.789	2.817	3.615	3.486
		msmape	184.125	11.090	13.747	14.435	13.463	10.679	9.247	13.917	11.442	11.709	133.424	10.832	11.216	11.583	181.222	136.243	24.348	11.221	16.920	50.062	11.686	14.275	43.440	13.191	13.191	11.957	12.986	14.322
		mase_rank	2	337	180	49	72	158	393	103	159	96	22	217	150	255	6	8	85	189	688	80	295	158	77	104	0	154	106	115
		msmape_rank	2	341	180	48	78	177	376	103	153	95	19	219	145	248	4	6	85	194	703	80	297	161	75	96	0	146	112	116
	×	mase	8.941	1.116	1.272	1.131	1.326	1.007	0.968	1.077	1.052	1.127	5.878	1.059	1.064	1.073	8.711	7.718	1.127	1.127	3.1e+10	2.185	1.104	1.397	1.648	1.314	1.314	1.197	1.311	1.246
		msmape	142.824	30.053	36.802	34.972	37.374	26.261	25.243	34.808	27.307	27.894	119.760	31.262	26.993	33.385	133.399	129.153	34.693	33.307	40.210	48.911	35.019	37.166	47.503	32.507	32.507	31.528	39.814	39.629
		mase_rank	38	338	134	24	66	160	257	140	91	103	50	290	171	259	84	51	100	245	144	211	139	193	101	260	0	75	47	34
		msmape_rank	39	332	126	24	62	162	245	136	91	98	48	315	168	251	83	53	95	254	144	213	141	198	101	277	0	72	46	35
Stationarity	√	mase	9.380	1.212	1.343	1.139	1.290	1.004	0.961	1.057	1.066	1.132	6.309	1.043	1.133	1.162	9.216	7.870	1.187	1.100	15.848	3.172	1.257	1.677	1.971	1.594	1.594	1.189	1.618	1.544
		msmape	135.888	31.080	37.594	35.434	37.306	27.024	26.120	35.122	28.172	28.539	114.323	32.232	27.546	33.519	128.366	122.194	31.229	34.281	38.320	52.234	36.212	38.615	48.139	33.876	33.876	31.216	41.513	40.645
		mase_rank	34	253	106	19	58	130	220	119	79	83	56	224	143	202	80	50	97	196	146	197	125	127	77	220	0	58	37	38
		msmape_rank	37	240	101	19	55	134	206	114	79	82	51	258	141	194	82	50	93	198	150	197	122	132	78	231	0	53	40	37
	×	mase	36.826	2.305	3.007	2.768	2.732	2.065	1.793	2.553	2.297	2.451	24.945	2.125	2.408	2.269	36.320	25.907	4.431	2.214	3.1e+10	7.032	2.500	3.143	13.664	3.288	3.288	2.585	3.118	3.003
		msmape	183.264	12.885	16.225	16.800	16.610	12.206	10.754	16.425	12.940	13.397	135.177	12.853	12.904	14.325	178.281	139.836	27.269	13.455	21.168	47.757	13.942	16.306	43.554	14.809	14.809	14.981	15.365	16.947
		mase_rank	6	422	208	54	80	188	430	124	171	116	16	283	178	312	10	9	88	238	686	94	309	224	101	144	0	171	116	111
		msmape_rank	4	433	205	53	85	205	415	125	165	111	16	276	172	305	5	9	87	250	697	96	316	227	98	142	0	165	118	114
Transition	√	mase	19.759	1.601	1.827	1.820	1.985	1.397	1.282	1.571	1.505	1.744	11.972	1.380	1.779	1.543	19.414	13.744	2.098	1.474	5.7e+4	3.998	1.930	2.739	5.891	2.684	2.684	1.949	2.723	2.603
		msmape	155.700	22.672	27.664	25.741	29.013	21.932	20.869	24.803	22.973	23.707	117.240	23.002	22.978	23.031	151.088	125.136	26.486	24.545	29.179	45.521	25.020	30.833	36.610	28.376	28.376	25.504	28.725	29.234
		mase_rank	35	457	210	37	83	214	440	168	164	132	65	420	224	348	87	56	137	334	481	85	230	176	154	240	0	97	71	52
		msmape_rank	36	451	198	36	80	222	422	165	159	130	60	448	220	341	84	57	135	346	489	87	226	181	152	251	0	94	74	53
	×	mase	37.372	2.371	3.303	2.682	2.489	2.102	1.799	2.676	2.369	2.277	27.831	2.278	2.136	2.358	36.961	27.987	4.984	2.323	5.3e+10	8.259	2.157	2.253	14.810	2.506	2.506	2.284	2.172	2.113
		msmape	180.775	15.284	19.143	21.216	17.039	10.984	9.435	21.932	11.622	11.399	144.591	15.905	10.860	19.785	172.323	146.942	31.454	16.404	25.629	56.754	18.520	14.673	61.194	11.329	11.329	13.554	20.089	20.905
		mase_rank	5	218	104	36	55	104	210	75	86	67	7	87	97	166	3	3	48	100	351	206	204	175	24	124	0	132	82	97
		msmape_rank	5	222	108	36	60	117	199	74	85	63	7	86	93	158	3	2	45	102	358	206	212	178	24	122	0	124	84	98
Shifting	√	mase	36.092	2.345	2.975	2.823	2.747	2.138	1.857	2.646	2.352	2.507	25.570	2.224	2.314	2.289	35.612	24.925	4.835	2.306	3.7e+10	6.862	2.331	2.836	14.610	3.002	3.002	2.695	2.799	2.676
		msmape	173.924	14.248	17.860	19.930	19.013	13.453	11.873	19.874	14.074	14.454	133.554	14.877	13.509	16.573	168.690	137.113	28.252	16.159	21.775	50.844	17.550	18.608	52.310	16.209	16.209	15.329	20.517	21.668
		mase_rank	10	343	153	43	80	150	373	105	136	79	23	176	169	248	21	13	58	168	530	214	284	168	49	136	0	148	110	106
		msmape_rank	10	349	153	43	84	168	354	99	133	77	25	185	164	240	20	10	57	171	536	213	294	173	46	132	0	140	114	107
	×	mase	15.639	1.390	1.709	1.410	1.563	1.142	1.062	1.262	1.257	1.355	9.401	1.159	1.485	1.363	15.381	12.499	1.394	1.229	7.4e+4	4.122	1.681	2.288	3.341	2.228	2.228	1.411	2.248	2.174
		msmape	155.031	26.022	31.618	28.465	30.674	22.763	21.881	27.811	23.945	24.258	120.183	26.254	23.983	27.347	148.280	128.544	29.073	27.310	34.251	48.150	28.022	31.762	38.186	28.605	28.605	26.882	30.950	31.018
		mase_rank	30	332	161	30	58	168	277	138	114	120	49	331	152	266	69	46	127	266	302	77	150	183	129	228	0	81	43	43
		msmape_rank	31	324	153	29	56	171	267	140	111	116	42	349	149	259	67	49	123	277	311	80	144	186	130	241	0	78	44	44

Results classified according to frequency

Datasets	Metric	Crossformer	N-HiTS	TiDE	DLinear	NLinear	PatchTST	TimesNet	FEDformer	FiLM	Stationary	Triformer	RF	Informer	N-BEATS	RNN	TCN	LightGBM	Xgboost	LR	KF	Arima	ND	NM	NMA	NS	AutoCES	AutoETS	AutoTheta
Hourly	mase	9.670	1.494	1.219	1.457	1.865	1.239	0.952	1.165	1.355	1.884	6.850	0.884	3.087	1.115	9.384	9.631	NaN	0.986	0.868	7.954	4.316	7.158	7.322	7.244	7.244	NaN	7.492	7.160
	msmape	73.124	24.469	31.181	27.319	29.860	25.530	23.859	28.480	28.337	30.702	36.421	24.274	29.396	24.061	70.492	41.590	NaN	28.921	28.473	70.382	36.713	50.216	43.165	49.580	49.580	NaN	48.714	49.235
	mase_rank	6	55	21	5	14	19	101	23	28	4	18	79	19	27	13	14	0	60	145	13	14	2	3	20	0	0	3	0
	msmape_rank	8	53	13	5	11	23	91	17	27	3	16	102	17	26	15	9	0	65	149	15	14	1	2	21	0	0	3	0
Daily	mase	19.945	0.996	1.173	1.275	1.247	1.079	1.024	1.174	1.110	1.143	10.520	1.159	1.098	1.065	19.444	7.917	NaN	1.237	1.343	1.206	1.109	1.254	5.918	1.225	1.225	1.262	1.210	1.167
	msmape	120.256	23.464	26.197	26.792	24.895	20.662	19.547	25.422	20.507	22.205	71.532	25.388	21.107	23.516	112.613	77.606	NaN	25.923	26.463	22.004	25.984	24.513	42.843	23.787	23.787	24.457	25.826	24.652
	mase_rank	28	181	61	12	17	16	81	69	24	49	38	131	73	144	56	40	0	116	75	32	37	37	15	111	0	21	8	11
	msmape_rank	30	175	63	13	17	14	78	68	21	48	35	132	75	143	57	46	0	128	76	30	31	36	15	118	0	18	8	9
Weekly	mase	44.139	1.300	1.758	1.803	1.547	1.296	1.182	1.345	1.430	1.285	22.404	1.588	1.207	1.426	43.859	7.649	2.670	1.721	32.454	1.370	1.498	1.710	23.588	1.555	1.555	1.428	1.515	1.615
	msmape	144.661	32.196	39.461	50.164	40.255	17.957	17.137	53.293	17.811	18.270	75.392	35.276	17.443	53.560	106.534	71.959	51.593	36.482	56.279	19.288	43.371	37.112	113.559	14.490	14.490	30.525	49.869	63.426
	mase_rank	3	37	16	12	4	19	43	11	15	20	5	30	23	30	17	0	15	21	33	153	51	111	5	99	0	10	13	5
	msmape_rank	1	37	18	11	3	16	43	10	18	19	5	29	23	27	13	0	13	22	31	153	55	119	5	107	0	10	11	6
Monthly	mase	29.547	1.525	1.388	1.764	2.280	1.117	1.053	1.392	1.310	1.867	13.725	1.148	1.541	1.486	29.427	20.278	1.670	1.236	5.5e+9	2.120	1.725	2.616	5.352	2.545	2.545	2.339	2.577	2.342
	msmape	193.177	14.904	17.172	16.285	21.481	13.718	13.402	14.678	15.148	16.378	118.475	13.955	15.250	14.698	192.250	137.981	16.491	14.903	17.214	18.355	16.226	22.252	28.820	21.001	21.001	18.750	20.191	18.223
	mase_rank	0	95	82	11	11	121	222	47	91	30	3	140	65	74	1	0	54	100	278	32	72	23	29	30	0	30	20	13
	msmape_rank	0	97	79	10	11	132	212	53	90	29	4	136	64	71	1	0	54	99	284	31	69	23	27	35	0	29	20	14
Quarterly	mase	19.201	1.416	1.438	1.902	2.217	1.187	1.127	1.741	1.418	1.729	18.362	1.401	1.395	1.441	19.194	20.263	2.662	1.433	1.211	1.903	1.164	1.940	5.722	1.984	1.984	1.777	1.726	1.706
	msmape	198.238	15.648	18.295	18.070	23.067	14.425	13.991	16.813	15.994	16.753	191.489	15.482	15.103	15.031	198.359	192.939	21.733	16.348	16.000	19.524	14.235	18.583	35.988	18.241	18.241	17.440	17.251	16.572
	mase_rank	0	104	72	10	25	87	127	37	52	47	2	67	67	86	0	0	57	67	193	24	140	52	70	33	0	37	38	20
	msmape_rank	0	103	77	10	24	96	120	37	50	48	1	65	62	83	0	0	60	68	194	27	140	50	69	31	0	36	41	22
Yearly	mase	24.387	3.870	6.277	3.811	3.114	3.476	2.921	3.891	3.470	3.297	24.293	3.208	3.344	3.745	24.363	29.224	6.283	3.251	9.5e+10	19.012	3.242	2.783	7.276	3.039	3.039	3.168	2.864	2.729
	msmape	198.211	20.776	30.446	23.540	22.018	22.237	18.990	23.864	22.118	20.281	199.307	20.602	21.386	20.247	197.946	199.599	36.939	21.333	41.919	157.051	20.268	20.112	42.554	19.935	19.935	20.272	19.464	19.314
	mase_rank	0	175	39	11	55	44	59	40	32	36	0	39	60	127	1	0	59	53	72	20	91	111	49	61	0	117	59	90
	msmape_rank	0	181	37	11	61	47	60	38	31	33	0	41	58	121	0	0	53	49	77	20	100	113	52	51	0	111	65	90
Other	mase	59.541	2.709	2.694	3.212	3.123	2.307	2.153	3.617	3.434	2.634	42.544	3.198	2.289	2.557	55.492	48.087	NaN	3.358	2.378	2.596	2.093	2.380	30.345	2.941	2.941	2.370	2.232	2.358
	msmape	157.858	10.102	10.283	13.841	13.916	10.952	9.853	13.804	13.576	12.069	115.990	11.151	10.390	9.770	151.264	122.739	NaN	12.206	12.015	15.013	12.097	13.258	36.152	13.905	13.905	11.657	12.944	12.935
	mase_rank	3	28	23	12	12	12	17	16	8	13	6	21	14	26	2	5	0	17	36	17	29	15	7	10	0	14	12	10
	msmape_rank	2	27	19	12	13	11	17	16	7	13	6	29	14	28	1	4	0	17	36	17	29	17	6	10	0	14	10	10