We present a general patchy ellipsoidal particle model suitable for conducting dynamics simulations of the aggregation behaviors of various shape- and/or surface-anisotropic colloids, especially patchy ellipsoids with continuously variable shape and tunable patchiness. To achieve higher computational efficiency in dynamics simulations, we employ a multi-GPU acceleration technique based on a domain decomposition algorithm. The validation and performance evaluation of this GPU-assisted model are performed by simulating several typical benchmark systems of non-patchy and patchy ellipsoids. Given the generality and efficiency of our GPU-assisted patchy ellipsoidal particle model, it will provide a highly feasible dynamics simulation framework to investigate the aggregation behaviors of anisotropic soft matter systems comprised of shape- and/or surface-anisotropic building blocks.