I need a program that will run under Delphi 7, with source code provided, that will remove all duplicate lines from an ASCII text file.
The utility must be able to handle files that contain over 2 million lines of text and that will not exhaust the memory of a PC with 4 gigabytes of RAM.
IOW, assuming the file contains 12 lines of data, like this:
aa ccde aa cxy y ccde huisk aaa ccde huisk huisk xiops rtwpo y
...I would want the output to be in one of two optional formats:
aa aaa ccde cxy huisk rtwpo xiops y
aa ccde cxy y huisk aaa xiops rtwpo
The program should not be able to perform either of the above tasks with a 2 million line file in less than 10 minutes. Some sort of visual progress indicator should be included so that the user can have an idea of how the task is progressing.