It's difficult to think about grammar when you're trying to speak. I think it's much easier to improve your vocabulary score than your grammar score.
- Grammar is only 25% of your speaking score.
- If you are thinking too much about grammar, you will lose fluency.
- If you try to use memorised grammatical structures, your speech will not sound 'natural'.
- You will not avoid small grammar mistakes unless you have lived in an English speaking country for many years.
I've taught many students who have achieved a score of 7 or higher, and they all made small grammar mistakes. They got their high scores because we focused on vocabulary, ideas, opinions and fluency.
Have a look through my speaking lessons, and you'll notice that the focus is usually on vocabulary.