This paper presents an ensemble of temporal Transformers for financial time series forecasting, aiming to improve the accuracy of price predictions. The study shows that temporal Transformers with similarity embedding outperform traditional models like ARIMA and other deep learning techniques in multi-horizon forecasts due to their ability to account for the conditional heteroscedasticity in financial data. However, further optimization of the forecasting methods is needed to achieve higher precision. The proposed ensemble approach efficiently uses available data over extended timeframes by combining multiple temporal Transformer models learned within sliding windows. Additionally, a stacking meta-learner is introduced, which uses a quantile estimator to determine optimal weights for combining base models of smaller windows. By decomposing the constituent time series of an extended timeframe, the method optimizes the utilization of the series for financial deep learning. This approach simplifies the training process of a temporal Transformer model over an extended time series while achieving better performance, especially when accounting for the non-constant variance of financial time series. Experiments on 20 companies from the Dow Jones Industrial Average show more than 40% and 60% improvement in predictive performance compared to the baseline temporal Transformer. The study highlights the potential of temporal Transformers in financial forecasting, particularly when combined with ensemble techniques. The paper also discusses the limitations of the attention mechanism in Transformer models, which assumes homoscedasticity in financial time series data. To address this, the authors introduce a class of deep learning architecture called Similarity embedded temporal Transformer (SeTT) and r-SeTT, which incorporate the characteristic of non-constant volatility into the Transformer's attention mechanism, demonstrating improved performance across various timeframes and intervals.This paper presents an ensemble of temporal Transformers for financial time series forecasting, aiming to improve the accuracy of price predictions. The study shows that temporal Transformers with similarity embedding outperform traditional models like ARIMA and other deep learning techniques in multi-horizon forecasts due to their ability to account for the conditional heteroscedasticity in financial data. However, further optimization of the forecasting methods is needed to achieve higher precision. The proposed ensemble approach efficiently uses available data over extended timeframes by combining multiple temporal Transformer models learned within sliding windows. Additionally, a stacking meta-learner is introduced, which uses a quantile estimator to determine optimal weights for combining base models of smaller windows. By decomposing the constituent time series of an extended timeframe, the method optimizes the utilization of the series for financial deep learning. This approach simplifies the training process of a temporal Transformer model over an extended time series while achieving better performance, especially when accounting for the non-constant variance of financial time series. Experiments on 20 companies from the Dow Jones Industrial Average show more than 40% and 60% improvement in predictive performance compared to the baseline temporal Transformer. The study highlights the potential of temporal Transformers in financial forecasting, particularly when combined with ensemble techniques. The paper also discusses the limitations of the attention mechanism in Transformer models, which assumes homoscedasticity in financial time series data. To address this, the authors introduce a class of deep learning architecture called Similarity embedded temporal Transformer (SeTT) and r-SeTT, which incorporate the characteristic of non-constant volatility into the Transformer's attention mechanism, demonstrating improved performance across various timeframes and intervals.