The article introduces PirateNets, a novel physics-informed deep learning architecture designed to address the training instability and poor performance of traditional physics-informed neural networks (PINNs) when using deeper networks. The main issue identified is that the use of multi-layer perceptron (MLP) architectures with unsuitable initialization schemes leads to poor trainability of network derivatives, resulting in unstable minimization of the PDE residual loss. PirateNets introduce an adaptive residual connection that allows networks to be initialized as shallow networks that progressively deepen during training. This approach enables the encoding of appropriate inductive biases corresponding to a given PDE system into the network architecture. The article provides empirical evidence showing that PirateNets are easier to optimize and can gain accuracy from increased depth, achieving state-of-the-art results across various benchmarks. The architecture is evaluated on the Allen-Cahn equation benchmark, demonstrating superior performance compared to traditional PINNs and modified MLPs. The results show that PirateNets achieve a relative $L^2$ error of $2.24 \times 10^{-5}$, indicating excellent agreement with the reference solution. The study also highlights the effectiveness of physics-informed initialization, which improves the performance of both PirateNets and other architectures. The code and data for this study will be made publicly available at https://github.com/PredictiveIntelligenceLab/jaxpi.The article introduces PirateNets, a novel physics-informed deep learning architecture designed to address the training instability and poor performance of traditional physics-informed neural networks (PINNs) when using deeper networks. The main issue identified is that the use of multi-layer perceptron (MLP) architectures with unsuitable initialization schemes leads to poor trainability of network derivatives, resulting in unstable minimization of the PDE residual loss. PirateNets introduce an adaptive residual connection that allows networks to be initialized as shallow networks that progressively deepen during training. This approach enables the encoding of appropriate inductive biases corresponding to a given PDE system into the network architecture. The article provides empirical evidence showing that PirateNets are easier to optimize and can gain accuracy from increased depth, achieving state-of-the-art results across various benchmarks. The architecture is evaluated on the Allen-Cahn equation benchmark, demonstrating superior performance compared to traditional PINNs and modified MLPs. The results show that PirateNets achieve a relative $L^2$ error of $2.24 \times 10^{-5}$, indicating excellent agreement with the reference solution. The study also highlights the effectiveness of physics-informed initialization, which improves the performance of both PirateNets and other architectures. The code and data for this study will be made publicly available at https://github.com/PredictiveIntelligenceLab/jaxpi.