Episode 11 - The Dashboard, part 3 - removing duplicates

We get duplicates in our search results for two reasons:

1. Running the same search again and again will return some of the results already fetched in the previous runs.
2. Search terms can be defined more than once, and we get results for each of the search terms.

In this episode we deal with removing the first source for duplicates. For this we will add a column to our Search model and use it to query Twitter so it won’t send us results we already received.

Before we add the extra column, we refactor our search code in order to support the changes in logic we want to introduce.

Unfortunately, due to circumstances beyond our control, we don’t get around to finishing the algorithm, but we do get to the point where we only have a couple of more steps which we explain. To check out the finished code look at the Brandizzle project on GitHub for the Search model and the Search model spec.