JSI Tip 4730. How can I remove duplicate entries from a file, without disturbing the record sequence?


If you have a requirement to remove duplicate entries from a file, without disturbing the record sequence, run:

remdups filename.

RemDups.bat contains:

@echo off
setlocal
if \{%1\}==\{\} goto syntax
if not exist %1 goto syntax
set folder=%~dp1
set file=%1
set filename=%~nx1
pushd %folder%
copy /Y %file% "%TEMP%\%filename%" >nul 2>&1
del /f /q %file%
for /f "tokens=*" %%a in ('type "%TEMP%\%filename%"') do set line=%%a&call :dups
popd
endlocal
goto :EOF
:syntax
@echo Syntax: RemDups filename
endlocal
goto :EOF
:dups
if not exist %file% @echo %line%>%file%&goto :EOF
find "%line%" %file%>nul 2>&1
if %errorlevel% EQU 1 @echo %line%>>%file%


Hide comments

Comments

  • Allowed HTML tags: <em> <strong> <blockquote> <br> <p>

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
Publish