This is an interesting topic to discuss. Testers always find it difficult to find a solution to this problem. Most of the times they go ahead without proper planning about the procedure to be followed in automation, and then end up being in a mess. Proper idea of automation should be put into use for deciding the process that needs to be followed in test automation. Else, one may face a road-block, where it would be difficult to change the procedure, and also cannot proceed forward with the same procedure.
So what are the pre-requisites that one need to have before deciding on test automation? The answer is quite simple. Complete understanding of the application or software that you are proceeding to test. Let me explain it a bit more deeply. The engineer should have confidence regarding the application that he is going to do automation for. He should have the in-and-out idea about the various functionalities that exist in the application. This plays an important role in deciding the type of automation that needs to be performed.
Idea about the development platform is another important aspect that a person needs to have before deciding on automation. If one has idea about that is the language that the application is built in, then he can choose a tool, that is most suited for automating applications made in that language. For example: There are issues that we normally face with automation when dealing with .NET applications, since they may include many third party controls. This makes many a number of tools which are unable to identify the objects embedded in the application, insufficient for automation. So the complexity of the code should be properly understood prior to select an automation tool or framework.
Another important aspect that needs to be taken care before planning automation is the amount of automation that is expected. Although it might not seem to be affecting automation, it certainly is important because it is a deciding factor in choosing the tool that we can utilize for automation. Why should we choose a tool like QTP, when there are very few test cases which need to be automated? So these factors need to be defined well earlier during the planning process of automation itself.
There are other important factors also which decide test automation like tool support, cost of tool, effort required in automation etc. So the proper way should be beginning with a discussion including the various departments involved in the development process taking into consideration all the above mentioned facts.