Abstract: A method of forming a three-dimensional memory device includes forming a stack of alternating first and second material layers over a substrate, forming a memory opening through the stack, forming a memory film and a semiconductor channel in the memory opening, and forming backside recesses by removing the second material layers selective to the first material layers and the memory film, where an outer sidewall of the memory film is physically exposed within each backside recess. The method also includes forming at least one set of surfaces selected from silicon deposition inhibiting surfaces on the first material layers and silicon deposition promoting surfaces over the memory film in the back side recesses, selectively growing a silicon-containing semiconductor portion laterally within each backside recess, forming at least one blocking dielectric within the backside recesses, and forming conductive material layers by depositing a conductive material within the backside recesses.