The originality of this study is the introduction of numerical investigations on the bioconvection flow of nano-encapsulated phase change materials (NEPCMs) with oxytactic microorganisms in a new configuration of a circular annulus with a rotating wavy inner cylinder. The incompressible smoothed particle hydrodynamics (ISPH) method was applied to solve the governing partial differential equations for the velocity, temperature, concentration, and density of motile microorganisms. Compared with the conventional mesh-based method, this mesh-free, particle-based approach offers strong advantages in the simulation of complex problems with free surfaces and moving boundaries with large displacements. The pertinent parameters are the undulation number (<i>N<sub>und</sub></i> = 2-36), bioconvection Rayleigh number (<i>Ra<sub>b</sub></i> = 1-1000), Darcy parameter (Da = 10<sup>-5</sup>-10<sup>-2</sup>), length of the inner fin (<i>L<sub>Fin</sub></i> = 0.05-0.15), radius of the inner wavy cylinder (<i>R<sub>c</sub></i> = 0.05-0.25), Rayleigh number (Ra = 10<sup>3</sup>-10<sup>5</sup>), undulation amplitude of the inner wavy cylinder surface (<i>A</i> = 0.1-0.4), and frequency parameter (<i>ω </i>= 1-5). The undulation number of the inner wavy cylinder enhanced the flow of the oxytactic microorganisms and isotherms, whereas it had the reverse effect on the velocity, decreasing the maximum velocity by 26.56%. In addition, the comparatively high undulation amplitude and frequency increased the average Nusselt and Sherwood numbers. It was found that the embedded wavy cylinder interacting with fins plays an important role in enhancing heat transfer and the bioconvection flow within a closed domain.