robots.txt controls nothing.
It’s like leaving a sugar cookie and a chocolate chip cookie on a park bench with a sign that says “Please eat the sugar cookie”.
Some people will only eat the sugar cookie, some will eat the chocolate chip, some will eat both, some will eat none.
Same thing with search engines. You can express your desire, but they are free to ignore you and do what they feel like.
Also crawling is different than indexing. Just because something is crawled doesn’t mean it will make it into the search results.