remove words from mutiple files

On Mon, 29 Dec 2008 18:57:25 -0800, Jesse Dorland wrote:

> I have downloaded some 100 astronomical files. All of the files have
> words "Mr. Big". Is there a way to remove these words?

man sed


--
"Ubuntu" -- an African word, meaning "Slackware is too hard for me".
The Usenet Improvement Project: http://improve-usenet.org
Ahhhhhhh!: http://brandybuck.site40.net/pics/relieve.jpg

On Dec 30, 12:36 am, Mysterious Traveler <mysterious_trave....RemoveThis@dot.net>
wrote:
> Jesse Dorland wrote:
> > I have downloaded some 100 astronomical files. All of the files have
> > words "Mr. Big". Is there a way to remove these words?
>
> If your word processor has a find and replace option, you can have it
> find "Mr. Big" and in the replace field try using a space for the word.
> In Windows I was able to do this by holding down the alt key and
> pressing 255 on the keypad, this makes an invisible character that takes
> the place of the words you want to get rid of, but in Linux those
> characters can't be made. Maybe someone else can suggest a way, or you
> could edit those text files running Windows.
>
> Mysterious Traveler

Well, my situation is bit more complicated. You see, in this folder I
have some 100 .doc files, and about 200 .jpg files. Inside all the doc
files, I have a word "Mr. Big", and then all the files names
(Saturn_Mr.Big.doc), (Venus_Mr.Big.doc), and same goes for .jpgs
files.

So I want to remove not only from documents, but from files names too.

On Dec 30, 4:06 am, "J.O. Aho" <u....DeleteThis@example.net> wrote:
> Mysterious Traveler wrote:
> > Jesse Dorland wrote:
> >> I have downloaded some 100 astronomical files. All of the files have
> >> words "Mr. Big". Is there a way to remove these words?
> > If your word processor has a find and replace option, you can have it
> > find "Mr. Big" and in the replace field try using a space for the word.
> > In Windows I was able to do this by holding down the alt key and
> > pressing 255 on the keypad, this makes an invisible character that takes
> > the place of the words you want to get rid of, but in Linux those
> > characters can't be made. Maybe someone else can suggest a way, or you
> > could edit those text files running Windows.
>
> The absolute fastest way is to use sed to remove the text.
>
> If using the slow and boring way, loading everything into a word processor
> like openoffice or using emacs, you use the replace function, only filling the
> search box, but not entering anything in the replace box (for you microsoft
> users, this works even in microsoft products).
>
> In the end is the question, do you want to do 100 load in, use replace and
> then save or type a short command line that will take care of all the files
> and remove the text, sed will be finished before you have loaded the word
> processor.


Do you know if it will remove the words from inside documents, or
files names as well?

Mysterious Traveler wrote:
> Jesse Dorland wrote:
>> I have downloaded some 100 astronomical files. All of the files have
>> words "Mr. Big". Is there a way to remove these words?
> If your word processor has a find and replace option, you can have it
> find "Mr. Big" and in the replace field try using a space for the word.
> In Windows I was able to do this by holding down the alt key and
> pressing 255 on the keypad, this makes an invisible character that takes
> the place of the words you want to get rid of, but in Linux those
> characters can't be made. Maybe someone else can suggest a way, or you
> could edit those text files running Windows.

The absolute fastest way is to use sed to remove the text.

If using the slow and boring way, loading everything into a word processor
like openoffice or using emacs, you use the replace function, only filling the
search box, but not entering anything in the replace box (for you microsoft
users, this works even in microsoft products).

In the end is the question, do you want to do 100 load in, use replace and
then save or type a short command line that will take care of all the files
and remove the text, sed will be finished before you have loaded the word
processor.

--

//Aho

Jesse Dorland wrote:

> Do you know if it will remove the words from inside documents, or
> files names as well?

sed will remove inside the file, if you want to replace the string in a
filename, you need to make a small shell script that reads the whole directory
and uses mv and sed together to rename the files that has the offending string
it's name.

--- remove string from filename ---
#!/bin/sh
for file in *
do
change_name_to=`echo $file | sed 's/Mr\. Big\//'`
mv "./$file" "./$change_name_to"
done
--- eof ---

This should fix that later problem, and it's space safe (it works with file
names with spaces too), with a bit of modification you can it to make both
changing the file content and the file name and allow you to use option which
allows you to use the script to replace what ever you want instead of the hard
coded "Mr. Big" thing.

--

//Aho

On Tue, 30 Dec 2008 05:41:17 -0800, Jesse Dorland wrote:

> On Dec 30, 12:36 am, Mysterious Traveler <mysterious_trave....TakeThisOut@dot.net>
> wrote:
>> Jesse Dorland wrote:
>> > I have downloaded some 100 astronomical files. All of the files have
>> > words "Mr. Big". Is there a way to remove these words?
>>
>> If your word processor has a find and replace option, you can have it
>> find "Mr. Big" and in the replace field try using a space for the word.
>> In Windows I was able to do this by holding down the alt key and
>> pressing 255 on the keypad, this makes an invisible character that
>> takes the place of the words you want to get rid of, but in Linux those
>> characters can't be made. Maybe someone else can suggest a way, or you
>> could edit those text files running Windows.
>>
>> Mysterious Traveler
>
> Well, my situation is bit more complicated. You see, in this folder I
> have some 100 .doc files, and about 200 .jpg files. Inside all the doc
> files, I have a word "Mr. Big", and then all the files names
> (Saturn_Mr.Big.doc), (Venus_Mr.Big.doc), and same goes for .jpgs files.
>
> So I want to remove not only from documents, but from files names too.

You said you had text files, now you tell us it is .doc files. Big
difference.

On Tue, 30 Dec 2008 15:13:33 +0100, J.O. Aho wrote:

> Jesse Dorland wrote:
>
>> Do you know if it will remove the words from inside documents, or files
>> names as well?
>
> sed will remove inside the file, if you want to replace the string in a
> filename, you need to make a small shell script that reads the whole
> directory and uses mv and sed together to rename the files that has the
> offending string it's name.
>
> --- remove string from filename ---
> #!/bin/sh
> for file in *
> do
> change_name_to=`echo $file | sed 's/Mr\. Big\//'` mv "./$file"
> "./$change_name_to"
> done
> --- eof ---
>
> This should fix that later problem, and it's space safe (it works with
> file names with spaces too), with a bit of modification you can it to
> make both changing the file content and the file name and allow you to
> use option which allows you to use the script to replace what ever you
> want instead of the hard coded "Mr. Big" thing.

Easier would be to do two separate operations using wildcards to affect
all the files: First, sed to find and remove the text; then, rename
(instead of mv) to rename the files. Very fast. Very simple. No
scripting required.

Stef

jellybean stonerfish wrote:
> On Tue, 30 Dec 2008 05:41:17 -0800, Jesse Dorland wrote:

>> Well, my situation is bit more complicated. You see, in this folder I
>> have some 100 .doc files, and about 200 .jpg files. Inside all the doc
>> files, I have a word "Mr. Big", and then all the files names
>> (Saturn_Mr.Big.doc), (Venus_Mr.Big.doc), and same goes for .jpgs files.
>>
>> So I want to remove not only from documents, but from files names too.
>
> You said you had text files, now you tell us it is .doc files. Big
> difference.

That depends on the version of the doc standard, most ms-doc files has the
text in plain text in the file, including font information in the begining of
each line, sed do work still well here, with latest revisions this may not be
true.

--

//Aho

On Tue, 30 Dec 2008 17:04:26 +0000, Stefan Patric wrote for every to
trash:

> On Tue, 30 Dec 2008 15:13:33 +0100, J.O. Aho wrote:
>
>> Jesse Dorland wrote:
>>
>>> Do you know if it will remove the words from inside documents, or
>>> files names as well?
>>
>> sed will remove inside the file, if you want to replace the string in a
>> filename, you need to make a small shell script that reads the whole
>> directory and uses mv and sed together to rename the files that has the
>> offending string it's name.
>>
>> --- remove string from filename ---
>> #!/bin/sh
>> for file in *
>> do
>> change_name_to=`echo $file | sed 's/Mr\. Big\//'` mv "./$file"
>> "./$change_name_to"
>> done
>> --- eof ---
>>
>> This should fix that later problem, and it's space safe (it works with
>> file names with spaces too), with a bit of modification you can it to
>> make both changing the file content and the file name and allow you to
>> use option which allows you to use the script to replace what ever you
>> want instead of the hard coded "Mr. Big" thing.
>
> Easier would be to do two separate operations using wildcards to affect
> all the files: First, sed to find and remove the text; then, rename
> (instead of mv) to rename the files. Very fast. Very simple. No
> scripting required.
>
> Stef

No mv is more efficient.

On Tue, 30 Dec 2008 05:41:17 -0800, Jesse Dorland wrote for every to
trash:

> On Dec 30, 12:36 am, Mysterious Traveler <mysterious_trave....TakeThisOut@dot.net>
> wrote:
>> Jesse Dorland wrote:
>> > I have downloaded some 100 astronomical files. All of the files have
>> > words "Mr. Big". Is there a way to remove these words?
>>
>> If your word processor has a find and replace option, you can have it
>> find "Mr. Big" and in the replace field try using a space for the word.
>> In Windows I was able to do this by holding down the alt key and
>> pressing 255 on the keypad, this makes an invisible character that
>> takes the place of the words you want to get rid of, but in Linux those
>> characters can't be made. Maybe someone else can suggest a way, or you
>> could edit those text files running Windows.
>>
>> Mysterious Traveler
>
> Well, my situation is bit more complicated. You see, in this folder I
> have some 100 .doc files, and about 200 .jpg files. Inside all the doc
> files, I have a word "Mr. Big", and then all the files names
> (Saturn_Mr.Big.doc), (Venus_Mr.Big.doc), and same goes for .jpgs files.
>
> So I want to remove not only from documents, but from files names too.

That would be a good use for the rename feature if its in Ubuntu.

If not use Aho's example to rename the file during the mv step.

or here's an old example

#!/bin/sh
# renames files
if [ $# -eq 0 ] # if there is input
then
echo $1 "requires dir filenam/n";
exit
fi

for filename in $1 # Not necessary to use basename,
# since "*" won't return any file containing "/".

do n=`echo "$filename" | tr '[A-Z]' '{a-z}'`
# Variable substitution:

n=${n%/} # Removes trailing slash, added above, from filename.
[[ $filename == $n ]] || cp "$filename" "$n"
# Checks if filename already lowercase.
[[ $filename == $n ]] || echo "$filename" " moved to " "$n" >> /tmp/
fmtfile.ans
# Prints changed filenames
done

On Tue, 30 Dec 2008 17:55:25 +0100, J.O. Aho wrote:

> jellybean stonerfish wrote:
>> On Tue, 30 Dec 2008 05:41:17 -0800, Jesse Dorland wrote:
>
>>> Well, my situation is bit more complicated. You see, in this folder I
>>> have some 100 .doc files, and about 200 .jpg files. Inside all the doc
>>> files, I have a word "Mr. Big", and then all the files names
>>> (Saturn_Mr.Big.doc), (Venus_Mr.Big.doc), and same goes for .jpgs
>>> files.
>>>
>>> So I want to remove not only from documents, but from files names too.
>>
>> You said you had text files, now you tell us it is .doc files. Big
>> difference.
>
> That depends on the version of the doc standard, most ms-doc files has
> the text in plain text in the file, including font information in the
> begining of each line, sed do work still well here, with latest
> revisions this may not be true.

When I did my test, using sed, on .doc files I created with openoffice, I
found it works if the strings are the same length, but fails if the
strings are different lengths.

sed 's/Big/Abc/' file.doc ::: WORKS
sed 's/Big/Ab/' file.doc ::: DOES NOT WORK

stonefish

jellybean stonerfish wrote:
> On Tue, 30 Dec 2008 17:55:25 +0100, J.O. Aho wrote:
>
>> jellybean stonerfish wrote:
>>> On Tue, 30 Dec 2008 05:41:17 -0800, Jesse Dorland wrote:
>>>> Well, my situation is bit more complicated. You see, in this folder I
>>>> have some 100 .doc files, and about 200 .jpg files. Inside all the doc
>>>> files, I have a word "Mr. Big", and then all the files names
>>>> (Saturn_Mr.Big.doc), (Venus_Mr.Big.doc), and same goes for .jpgs
>>>> files.
>>>>
>>>> So I want to remove not only from documents, but from files names too.
>>> You said you had text files, now you tell us it is .doc files. Big
>>> difference.
>> That depends on the version of the doc standard, most ms-doc files has
>> the text in plain text in the file, including font information in the
>> begining of each line, sed do work still well here, with latest
>> revisions this may not be true.
>
> When I did my test, using sed, on .doc files I created with openoffice, I
> found it works if the strings are the same length, but fails if the
> strings are different lengths.
>
> sed 's/Big/Abc/' file.doc ::: WORKS
> sed 's/Big/Ab/' file.doc ::: DOES NOT WORK

OpenOffice and StarOffice differs from ms-office in the way they create the
doc files, just take one saved with ms-office, open it in OO and then save it
with another file, and then do a 'ls -l' on both files, the original made with
ms-office and the one saved with OO, you will see atleast 25% difference on
the file size.

--

//Aho

On Tue, 30 Dec 2008 23:10:54 +0100, J.O. Aho wrote:

> jellybean stonerfish wrote:
>> On Tue, 30 Dec 2008 17:55:25 +0100, J.O. Aho wrote:
>>
>>> jellybean stonerfish wrote:
>>>> On Tue, 30 Dec 2008 05:41:17 -0800, Jesse Dorland wrote:
>>>>> Well, my situation is bit more complicated. You see, in this folder
>>>>> I have some 100 .doc files, and about 200 .jpg files. Inside all the
>>>>> doc files, I have a word "Mr. Big", and then all the files names
>>>>> (Saturn_Mr.Big.doc), (Venus_Mr.Big.doc), and same goes for .jpgs
>>>>> files.
>>>>>
>>>>> So I want to remove not only from documents, but from files names
>>>>> too.
>>>> You said you had text files, now you tell us it is .doc files. Big
>>>> difference.
>>> That depends on the version of the doc standard, most ms-doc files has
>>> the text in plain text in the file, including font information in the
>>> begining of each line, sed do work still well here, with latest
>>> revisions this may not be true.
>>
>> When I did my test, using sed, on .doc files I created with openoffice,
>> I found it works if the strings are the same length, but fails if the
>> strings are different lengths.
>>
>> sed 's/Big/Abc/' file.doc ::: WORKS sed 's/Big/Ab/' file.doc
>> ::: DOES NOT WORK
>
> OpenOffice and StarOffice differs from ms-office in the way they create
> the doc files, just take one saved with ms-office, open it in OO and
> then save it with another file, and then do a 'ls -l' on both files, the
> original made with ms-office and the one saved with OO, you will see
> atleast 25% difference on the file size.

That's ok. I rarely create a .doc file.
Editing the .doc files with sed, as the op asked sounds like the wrong
solution.
I would eliminate the .doc files from the equation.

wvText Saturn_Mr.Big.doc Saturn.txt
sed -ie 's/Mr. Big//' Saturn.txt