Nano-strengthening by employing nanoparticles is necessary for high-efficiency strengthening of elastomers, which has already been validated by numerous researches and industrial applications, but the underlying mechanism is still an open challenge. In this work, we mainly focus our attention on studying the variation of the tensile strength of nanofilled elastomers by gradually increasing the filler content, within a low loading range. Interestingly, the percolation phenomenon is observed in the relationship between the tensile strength and the filler loading, which shares some similarities with the percolation phenomenon occurring in rubber toughened plastics. That is, as the loading of nanofillers (carbon black, zinc oxide) increases, the tensile strength of rubber nanocomposites (SBR, EPDM) increases slowly at first, then increases abruptly and finally levels off. Meanwhile, the bigger the particle size, the higher the filler content at the percolation point, and the lower the corresponding tensile strength of rubber nanocomposites. The concept of a critical particle–particle distance (CPD) is proposed to explain the observed percolation phenomenon. It is suggested that rubber strengthening through nanoparticles is attributed to the formation of stretched straight polymer chains between neighbor particles, induced by the slippage of adsorbed polymer chains on the filler surface during tension. Meanwhile, the factors to govern this CPD and the critical minimum particle size (CMPS) figured out in this work are both discussed and analyzed in detail. Within the framework of this percolation phenomenon, this paper also clearly answers two important and intriguing issues: (1) why is it necessary and essential to strengthen elastomers through nanofillers; (2) why does it need enough loading of nanofillers to effectively strengthen elastomers. Moreover, on the basis of the percolation phenomenon, we give out some guidance for reinforcement design of rubbery materials: the interfacial interactions between rubber and fillers cannot be complete chemical bonding, and partial physical absorption of macromolecular chains on the filler surface is necessary, otherwise the formation of stretched straight chains would be seriously hindered. There should exist such an optimum crosslinking density for a certain filler reinforced rubber system, and as well an optimum filler loading for rubber strengthening. Additionally, the different percolation behaviors of Young’s modulus, the tensile strength and the electrical conductivity are compared and analyzed in our work. Lastly, molecular simulation indicates that it is not possible to strengthen glassy or hard polymer matrices by incorporating spherical nanoparticles. In general, by providing substantial experimental data and detailed analyses, this work is believed to promote the fundamental understanding of rubber reinforcement, as well provide better guidance for the design of high-performance and multi-functional rubber nanocomposites.